In pipewire.conf you can set the allowed sample rates that your DAC may use. This allows Pipewire to set the desired sample rate based upon the sample rate of the active audio stream.
If multiple streams are active, subsequent audio streams will be re-sampled to that of the first active stream. Once streams are idle and/or removed, it allows for new active streams to switch the sample rate again.
The current behavior of the pttkey stream is that it hijacks this functionality by always being considered active. And likely the first active stream upon logging into a new desktop session.
I'm not sure if only adding the stream when it needs to play would be the desired approach, or if there is a way for it to idle while remaining open.