The 1851/52 census you mentioned has the family as
Stephen P. age 60 born United States
Mary age 56 born United States
Mary age 28 born United States
Horace age 16, born Canada
William age 25, born Canada residing Cowansville
Stephen age 17 born Canada residing Alburgh
Isaac age 23, born Canada residing Stockholm
Joseph age 4 born United States
so it certainly looks like he would be son of Stephen
I did not find the family in 1861 Canada census so looked to US to see if they moved there as Stephen, Mary and Mary were born there.
There is a William Wadsworth of right age, born Canada, living in St. Lawrence Co. NY with a wife Catherine
In 1880 census William shown born Canada with both parents born VT
the earliest US census for William and Catherine is 1860 - so if this is your Wm. it explains why he was not in 1861 Canada census.
In 1860 there is a Phoebe Brooks with them, age 66, born VT so possible mother to Catherine?
later census including 1880 have them in Stockholm, NY
also in 1860 Stockholm, St. Lawrence, NY are Stephen 68, Mary 65, Horace 24, Lucy 22 and Joseph Kelly 11 (Horace and Lucy sh own as married)
also at familysearch.org there is a death for a Stephen Pearl Wadsworth age 89 died 1923 VT, son of Stephen Pearl Wadsworth and Mary Sordiff
it says father born South Hero, VT, mother born Canada and says Stephen born what looks like Ealdwales Manor, Canada (not totally sure of that spelling) Jan. 15, 1834
this age matches with the age of Stephen in 1851 census
and death of a Daniel Wadsworth died 1899 South Hero VT, aged 83, born Canada, father Stephen P. Wadsworth, mother Mary
Daniel was not with the family in 1851 census but could have been on his own by then.
I went back to Quebec records and found a bapt. for son Daniel
EPISCOPAL CHURCH, CALDWELL AND CHRISTIE, IBERVILLE
Jan. 30, 1826 baptism Daniel son of Stephen P. Wadsworth and Mary his wife, born the 21st day of Sept. 1816 - Abraham Gottiel and Jane Deas being sponsors.
and on same page
Jan. 30, 1826 baptism Jane Amelia daughter of Stephen P. Wadsworth and Mary his wife, born on the 14th day of Oct. 1818 - Henry Deas , Janes Deas and Jane St. Louis being sponsors.
also on same day these children were baptized (on same page)
Elizabeth born June 27, 1820 - John Griggs, Letitia Griggs and Abigail Griggs being sponsors.
Mary, born June 25, 1822 - John Griggs, Letitia Griggs, Abigail Griggs being sponsors.
Henry, born May 8, 1824 - Henry Deas, Abraham Stiel, Jane Deas being sponsors
METHODIST CHURCH, ST-ARMAND
Millicent, daughter of Stephen P. Wadsworth of Oldwells Manor (hard to read) was born April 9, 1821, also Stephen, son of the same Stephen P. and Mary was born 27th of January 1834 - both were baptized Oct. 9, 1841.
** this entry was a bit harder to read so hope dates are right
so somehow William is missed from all these entries so the only thing I can think of to prove for sure that he is Stephen and Mary's son - even though he is with family in 1851 census
would be to find his marriage record and hope it names family - or to find the will of Stephen P. Wadsworth or wife Mary and hope it names sons and where they live.
Stephen Pearl Wadswoth born about 1793 South Hero, died Sept. 5, 1882 Vermont, age 89 years 11 months 8 days parents Daniel Wadsworth and Zuba
also familyserch.org has William dying 1917 NY so perhaps his will would mention brothers, sisters, ?? Could be one of his brother might be an executor and that may help to narrow down parents.