James Middleton is also listed in the following Federal censuses:
Fayette Co., AL, 1850, p. 31b (Division 16, Beat 3): listed as James Midelton;
Fayette, 1860, p. 450 (Middle Division, p. 12): listed as James Middleton;
Fayette, 1870, p. 410b (T15R13, Fayette CH, p. 8): listed as James Middle;
as well as the census you mentioned,
Lamar, 1880, p. 604b (T15R15, p.16): listed as James Middleton.
I understand that his first wife was Parilee Elizabeth Moore (1814-1846), and that she is buried at the Fayette City Cemetery.
James Middleton (April 6, 1812 - June 19, 1888) and his second wife, S. A. (Sarah A. Stewart, December 1, 1825 - September 12,1887), are buried at the Vernon City Cemetery. Also buried there is their son, William Greene Middleton (June 22, 1856 - Nov. 18, 1898), whose grave is next to that of "Anna wife of W. G. Middleton," (Anna P. Sizemore, Nov. 13, 1861 - May 12, 1894).
James and Sarah's eldest daughter, Eugenia Ellen, is buried next to her husband, Thomas Henry Sizemore, at the Sizemore Cemetery on Seedtick Road about 5 miles south of Beaverton.
The second daughter, Jerusha Ann, married James Smith Hankins. He is buried at Fairview Baptist Church Cemetery, but I don't know the location of her grave.
The third daughter was Rebecca J., who married George W. Young on Nov. 20, 1878 in Lamar County.
I had no information on the youngest child, Nannie R., until reading Allison Denman Holland's response to one of your related queries.