With git, you can specify a non-standard ssh port using a host alias in ~/.ssh/config, then use git as normal, with the host alias instead of the normal domain. hg-git seems to be ignoring the entry in ~/.ssh/config file.
Update: Looks like although it's ignoring the host alias setting in ~/.ssh/config, it works if I add an entry with the full address into ~/.ssh/config