I can't thank you enough for all your assistance and direction. There is one thing-- I couldn't find the Freedman's Bank records on James Henry Sparrow on ancestry. I found it on familysearch.org and I don't see a Daniel Moore listed as his brother, I see a David Moore; this could be a reference to my GGpa. If James Sparrow was GGpa's brother, he would have been the oldest son from what I can tell by the 1850 census. His father Cannady must have died sometime after the 1850 census unless he left Beaufort County. I have scoured the Beaufort County census for 1850 and 1860 and cannot find Washington Keys. I am wondering how much free colored people who did not farm moved around to find work. I also wonder about all the free women of color listed as heads of household with no occupation listed. Is it possible that these men worked away from home a fair share of the time?
I am going to begin searching Martin County next. If you have any additional ideas/comments, please let me know.