My family Name that lived in that area was Cassady/Cassidy. John Cassidy was the father, his wife Mary and at least 10 children. They lived in that area in 1850 possibly a litte before that. John died in 1871 4 Jan when a tree fell on him. The 1850 census lists John Cassady with his wife Mary and daughter Catherine am not sure if they are mine or not. Mine show up in 1860 with several children that are my family. That is why I suspect they had more children and were in the area at a few years before 1850.... I have the census cites at home and will not be able to access it until at least tomorrow evening. Incidently my grandmother and some of her family worked at the Danbury Hat Factory at one time.
Other names in that area may have been Garvey
Hope this helps.