Prior to his career in acting, Brimley dropped out of high school to serve in the United States Marine Corps, where he served in the Aleutian Islands for three years. He also worked as a bodyguard for Howard Hughes, and as a ranch hand, a wrangler, and a blacksmith. He then began shoeing horses for film and television. He began acting in the 1960s as a riding extra in Westerns and a stunt man at the urging of his friend, actor Robert Duvall.