I certainly know what you mean concerning the place of birth changing on the census.
Thomas and Agnes told the census taker my Mary Johnson was born in TN however when she got in her own home, someone who provided the information always said she was born in GA.
Her last son is always listed as being born in GA on the census but his daughter told me he was born in Ducktown TN. I questioned her about that because we have a Ducktown GA, too. So, I am going by what her father told her although his household always put GA as his place of birth. It gets really confusing!
In Dahlonega the gold mining was going big there in the 1800s, too.