We could play around with optimization using different compiler options. Since it's really easy to deploy new code with update packages, let's see what options work best.
http://www.raspberrypi.org/phpBB3/viewtopic.php?t=11629&p=127928 suggests using:
-mcpu=arm1176jzf-s
-mtune=arm1176jzf-s
-mfpu=vfp
-mfloat-abi=hard
We can also play around with -O, -O2, -O3 to see what gives the best results.