James Henry Rivoir was born in Philadelphia, Pennsylvania, on July 3, 1913. He attended Northeast High School, graduating in 1930, where he was a member of the Sketch Club and the Record Book Staff as a staff artist. He graduated from Temple University in 1942 with a Bachelor of Fine Arts in 1940 and worked as an art instructor before being inducted into the US Army in June 1942.

After the war ended, he was temporarily assigned as enlisted personnel to the MFAA Section of the Office of Military Government for Bavaria (OMGB). Following this assignment, Rivoir handled various administrative duties at the Munich Central Collecting Point. There, he worked alongside Monuments Men Edwin C. Rae, Captain Jonathan T. Morey, Sergeant Harry Huberman, and Corporal Ole C. Risom, among others. Rivoir remained involved in the effort to inspect, identify, and sort thousands of looted objects until at least March 1946.

Rivoir spent much of his postwar time in Germany, where he married a German national. He also served as an instructor in the Graphic and Plastic Arts Branch of the Army University in Biarritz, France.

James H. Rivoir died in 1984 and is buried with his family in Pennsylvania.