Milton Graham was born on July 18, 1934 in Chatham, MA, and attended Rensselaer High School in Defreetsville, New York where he starred in basketball. After high school he was recruited to play basketball for Colgate University and decided to play football also. He captained he basketball team and also went on to earn All-East honors in 1954 and honorable mention for All-American in football.

He was selected in the 14th round of the 1956 NFL Draft by the Chicago Bears. He was also selected by the Syracuse Nationals in the NBA Draft, but decided to play football in Canada with the Ottawa Rough Riders from 1959-1961. He made the All Star team in 1958 and helped win the Grey Cup with the Rough Riders in 1960.

He returned and played tackle for the Boston Patriots for three seasons (1961-1963). After suffering many concussions, he decided to retire.

After leaving football, he became an agent in the FBI, where he would remain for 20 years. He worked in the south enforcing the Civil Rights Act and later in New York City, working to capture fugitives.

In 1984 Milt Graham was inducted into Colgate University's Hall of Honor.