For reference, yes, I do have gmail.google.com as a site set to allow popups via Firefox's default popup blocker. After looking for info online about the problem, I also added it to TBE's "whitelist", but still no luck.
Well, I figured it out. It was indeed the new version of Tabbrowser Extensions. :/ At least I can simply disable it, send an invite, then reenable it. Closing and reopening the browser each time, of course. What a pain. :P Oh well, maybe this will help others.