You always have to worry a little about family stories, especially from the memory of a 6 year old. Do any of these have more evidence than what your grandfather remembered from childhood?
Do you know if the brothers were older? Their names would be a big help. If the ship did sink they couldn't be younger....unless maybe it sank on the way back here.
I wonder who they left behind. The brothers? I have encountered situations where a father came to the US first and then went back for the rest of the family. I have not encountered both parents coming and leaving children behind, but it certainly could happen.
If the ship did sink I think it would be a well known event. It may be how he dealt with their absence.
It doesn't help that there is no sign of him in records before 1914 and in no census before 1930.
The father's name is likely different in Sweden, Some form of Petter/Peter/Per rather than Patter in the first and last name.
I will do some looking but there isn't much to go on