After several hundred PTT presses, the level of the transmit audio drops off substantially. I've traced the issue down to the microphone samples: the samples being read from the microphone drop in amplitude. The amplitudes recover if you speak very loudly / turn up the signal level and then drop if back to its original level. This implies some strange state in the DSP on the Teensy audio hat's SGTL5000 chip.
Next step: examine the I2S samples with a logic analyzer to confirm that the data issues are on the Teensy audio hat.