I was searching for William Elliott's origins and found at least some material relevant to a Prime Minister. I fear the story is apocryphal, but does have a basis. I'll paste in what I have so far (most of which you likely have) and if I find something on William get back to you.
WILLIAM ELLIOTT b. NB 12NOV1837, m. 1st Amherst, Cumberland Co., NS 11DEC1863 CATHERINE (CASSIE) AMELIA CARTER (b. NS ca. 1844-5, d. Boston, Suffolk Co., MA 7JUL1881), d/o Capt. Isaac and Mary Metcalf (Atkinson) Carter; m. 2nd 7APR1886 MARGARET (MAGGIE) J. (FERGUSON) HOLSTEAD (b. Moncton, Westmorland Co., NB 24OCT1845, d. 30 Botsford St., Moncton 1MAY1922, buried Elmwood Cemetery, Moncton; m. 1st Charles Albert Holstead (ca. 1840-22FEB1883)), d/o Capt. Thomas (John in her death registration) and Eliza (Chapman) Ferguson.
William does not appear to have been in Westmorland Co., NB in 1871. Instead, he was in Head of Amherst, Cumberland Co., NS. He was age 34 and a farmer. Catherine was age 26 and Ontario was age 5. Another daughter (Emma?) was age 16 suggesting that William was married before. He was b. in NB and Catherine in NS. Ontario and Emma are also given as b. in NS, but in the circumstances this appears to be unlikely.
Catherine's father was likely Isaac Carter (53) in Amherst in 1871. Isaac was a hotel keeper and was b. NB, while his wife, Mary A. (45) was b. NS.
It is likely through Charles A. Holstead that the family was connected to R. A. Borden. Charles and Borden were law partners in Moncton in 1871. Presumably this Borden was connected to Sir Robert Laird Borden (1854-1937), Prime Minister of Canada 1911-1920. Also a lawyer, Sir Robert was from Annapolis Co., NS. R. A. appears have also to have been Robert Borden. Specifically Robert Allison Borden. When the funeral of Mrs. C. D. Thompson took place from Botsford St. in 1881 (probably from William Elliot's home) R. A. Borden was a pall bearer.
Robert Allison Borden was b. Avonport, Kings Co., NS 3FEB1845. He was a judge for Kent and Westmorland counties. He was a s/o George Newton Borden and Mariam Crane.
Now, here, we have a problem. If someone remembered Robert Borden visiting the family they may have assumed that it was Robert Laird Borden, the Prime Minister. They may have been related or they may not have been related. If a relationship can be shown then the visitations by the Prime Minister are possible. If not, then they were unlikely.
1. MARY ONTARIO (ONTARIO MARY) ELLIOTT b. JUL1865, d. Riverside, Cook Co., Illinois 20(or 27)JUN1907 age 42 years, m. Botsford St., Moncton, Westmorland Co., NB 13SEP1887 WILLIAM FRANCIS EVANS (b. Worsley, Lancashire, England 21JUL1863, d. Waukegan, Illinois 10SEP1956), s/o William Frederick and Mary (Wells) Evans.