I got my family tree file(s) fixed. The way I did it was to use FTM 2012 for PC to import the problematic file from FTM 2005 SE for PC. That gave me the new format, and all the media files made it through. Then I used the file migration utility (PC app) to export a version for "FTM for Mac 2". And that worked as well and now I have my family tree on the new Mac version like I wanted.
Also of note is that customer support got back to me and suggested another method as outlined by this article:http://ancestry.custhelp.com/app/answers/detail/a_id/1396
They suggested the problem might have been corrupted media. There may be some credence to that because one of the media files I ended up with didn't have a proper file extension and I'm not sure what it's supposed to be. (But I can live with one media file lost.)
Thank you to everyone who proffered suggestions and support.