Sound is very delayed in safari after leaving tab and returning

[amanda-phet]

I typically notice this when I've left a tab and come back a lot later, like hours or days later. Today I noticed it after leaving the tab and coming back about 5 minutes later.

Screen.Recording.2024-05-08.at.4.08.45.PM.mov

2018 macbook pro
OS 13.6.6
Safari 16.6

[jbphet]

@Nancy-Salpepi let me know today that this has been reported by QA before, see #156 (comment) and
#156 (comment).

[jbphet]

This issue may be related, and possibly investigated at the same time: #157.

[KatieWoe]

I saw this in phetsims/qa#1141. Refreshing the page didn't fix it, I had to use an entirely new tab.

[Nancy-Salpepi]

Mentioning this issue again, because I see it with phetsims/qa#1159.

After moving to a different window/tab sound will be stop completely or be very delayed in safari. As previously mentioned, refreshing doesn't work. An entirely new tab or window needs to be used.

[pixelzoom]

Looks like this is a chronic problem in Safari, but not getting attention. I'll add it to the Developer Priorities board for discussion, and mention it in Slack#planning.

[jbphet]

I've been given clearance to spend some time on this is during the current iteration, so I did some investigation today. Here are a few notes:

• I'm working to duplicate it on a Mac that we generally refer to as "Denzell's old Mac". Is running MacOS 13.7 Ventura, and Safari 16.6. This is a little behind the times - Safari 18 was released in September of 2024, so about 2 months ago, but this problem has been around for a while, so having the latest version probably isn't absolutely necessary.
• I tried a number of different tests where I'd load it, set a timer, then check it later. So far it's always been fine after 5-10 minutes, but I was able to get a small amount of delay after setting the Mac to go to sleep after 5 minutes of inactivity, then waking it up and testing. This was maybe a few milliseconds, but nowhere near the ~2 seconds of delay seen in the video above.
• I've create a special version with some logging and will work with @Nancy-Salpepi to see if she can reproduce with this logging in place. That will help to give me some additional direction.
• I also tested this out on an iPad, and could fairly easily get into situations when switching tabs where sound would stop being produced altogether. Whatever solutions I try for this delay problem, I should see if it changes (hopefully fixes) the iPad behavior too.

Patch with logging code

Subject: [PATCH] add logging code to investigate delayed sound, see https://github.com/phetsims/tambo/issues/194
---
Index: js/sound-generators/SoundClip.ts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/sound-generators/SoundClip.ts b/js/sound-generators/SoundClip.ts
--- a/js/sound-generators/SoundClip.ts	(revision d1eb357492a487d88e6afcd2ab7c25b273420b21)
+++ b/js/sound-generators/SoundClip.ts	(date 1730930876428)
@@ -138,6 +138,8 @@
         // initiate deferred play if this is a loop or if it hasn't been too long since the request was made
         if ( this.loop || ( Date.now() - this.timeOfDeferredPlayRequest ) / 1000 < MAX_PLAY_DEFER_TIME ) {
 
+          console.log( 'deferred play' );
+
           // Play the sound, but with a little bit of delay.  The delay was found to be needed because otherwise on
           // some browsers the sound would be somewhat muted, probably due to some sort of fade in of the audio levels
           // that the browser does automatically to avoid having the web page's sound start too abruptly.  The amount of
@@ -176,6 +178,8 @@
    */
   public play( delay = 0 ): void {
 
+    console.log( 'play called' );
+
     if ( this.audioContext.state === 'running' && this.wrappedAudioBuffer.audioBufferProperty.value ) {
 
       const now = this.audioContext.currentTime;
Index: js/soundManager.ts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/soundManager.ts b/js/soundManager.ts
--- a/js/soundManager.ts	(revision d1eb357492a487d88e6afcd2ab7c25b273420b21)
+++ b/js/soundManager.ts	(date 1730929179308)
@@ -338,6 +338,8 @@
     let previousAudioContextState: AudioContextState = phetAudioContext.state;
     audioContextStateChangeMonitor.addStateChangeListener( phetAudioContext, ( state: AudioContextState ) => {
 
+      console.log( `audio context state = ${state}` );
+
       phet.log && phet.log(
         `audio context state changed, old state = ${
           previousAudioContextState

[jbphet]

After some research, here are a few things I'd like to try once we can reproduce the problem consistently:

1. Calling resume on the audio context on a change to the visibility of the app when on Safari. While this should technically have no effect if the audio context isn't suspended, there are some hints online that this might refresh the audio context and reduce any latency.
2. Explicitly set the latencyHint value to interactive when creating the audio contexts. According to the MDN doc at https://developer.mozilla.org/en-US/docs/Web/API/AudioContext/AudioContext, this is the default value, so I'm not super optimistic about it, but it's worth a try.
3. Add browser-specific code that periodically calls resume on the audio context when the sim is not being actively used.
4. Play a very quite, basically silent, loop all the time, or perhaps only when the sim isn't visible, to prevent the audio rendering thread from going to sleep.

[Nancy-Salpepi]

With macOS 15.0.1 and safari 18.0.1:
I cleared my cache before each test.
I wasn't able to reproduce the long (~2 sec) delay that was seen in the original comment in this issue. I most often experienced no sound after several minutes. I also found that the outcomes weren't consistent -- there were times I lost all sound after switching away from a tab for ~1 min and then other times it took 10 minutes.

Observations using https://phet-dev.colorado.edu/jb-tests/mean-share-and-balance_en_phet.html:

1. Test 1: Open 3 tabs, 2 with the sim and 1 with https://webaudioapi.com/samples/rapid-sounds/

   • played with each for a second or 2 to produce sound
   • waited 5 minutes -- all sims had sound. No delay.

2. Test 2: Open 3 tabs, 2 with the sim and 1 with https://webaudioapi.com/samples/rapid-sounds/

   • played with each for a second or 2 to produce sound
   • waited 10 minutes -- The sim tab that still had focus, still had sound and no delay initially (see 4). The other sim
     tab and the webaudio tab didn't have sound

3. Further testing with the 2 tabs that didn't have sound:

   • Pasted a sim link into a tab where sound wasn't working --produced no sound when the sim was opened
   • Sometimes pressing the back button and then the forward button in safari fixed the sound, but not always.

4. Further testing with the sim tab that still had focus for 10 minutes of non-interaction:

   • I started to see a delay as I switched to the other tabs and then back to it (1-3 minute switch each time).
   • In the console I saw that the "play called" message appeared when I pressed the button. Here is the video:

SoundDelay.mp4

I never saw anything in the console other then:

Sorry I couldn't give you anything more concrete at this time, but hopefully this helped a little.

[Nancy-Salpepi]

About three hours later I noticed I still had the tabs open.
The audio context state = suspended message was there. When I started to interact with the sim again, it said audio context state = running, but there was no sound.

EDIT:
Ran a few more quick tests this morning to try and get the "audio context state = suspended message....

1. When the screen saver turned on, I left it for ~1 minute and I then went back to the sim. There was no message in the console about the audio. Interacting with the controls produced sound without delay.
2. When the display was allowed to turn off (black screen) and then on again, there was no message in the console about the audio. Interacting with the controls produced sound without delay.
3. Closing the MacBook and reopening it right away gave this message: audio context state = interrupted. Once opened, the sound worked fine without delay.

[jbphet]

Thanks @Nancy-Salpepi. The two things that I am going to key in on from your tests are:

1. The "play" message appeared on the console at the correct time when the sound was delayed. This tells me that our code is behaving correctly and that the delay is in the Web Audio code (which I suspected but wanted to confirm). It's probably in the audio rendering thread.
2. The audio context did move to the interrupted state, but the delay was next experienced after that. This state is unique to Safari, so I thought perhaps it was related to the delay, but it appears that this is not the case.

I'm going to try out some of the ideas that I listed in #194 (comment) locally and see if they improve the situation on the iPad and/or Mac that I have, and will publish some versions for testing based on what I learn.

[Nancy-Salpepi]

When the sim had no sound, opening https://phet-dev.colorado.edu/jb-tests/play-sound.html in the same tab also resulted in no sound.

When the sim had a slight delay (just under ~1), opening the above link also showed the same delay.

I didn't see anything new in the console, once no sound/delay happened.

It is difficult to get a longer delay on my mac....let me know if you need me to spend more time trying to get that to happen.

[jbphet]

I've created a temporary version of MS&B with some code to resume the context if it's in a non-running state after the sim is re-shown. @Nancy-Salpepi is going to do some testing on this. Here's the patch:

Details

Subject: [PATCH] resume context after visibility state changes, see https://github.com/phetsims/tambo/issues/194
---
Index: js/phetAudioContext.ts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/phetAudioContext.ts b/js/phetAudioContext.ts
--- a/js/phetAudioContext.ts	(revision 2829a039362aa7c99352fe70cff2bda9d8ec3db4)
+++ b/js/phetAudioContext.ts	(date 1731718225093)
@@ -7,6 +7,7 @@
  * @author John Blanco
  */
 
+import platform from '../../phet-core/js/platform.js';
 import IntentionalAny from '../../phet-core/js/types/IntentionalAny.js';
 import tambo from './tambo.js';
 
@@ -29,4 +30,25 @@
 // register for phet-io
 tambo.register( 'phetAudioContext', phetAudioContext );
 
+// If running on Safari, add a listener to resume the audio context when the page becomes visible.  This is necessary
+// only on Safari, which seems to have issues on both the mobile and desktop versions where audio can get delayed or
+// goes away altogether after the page is hidden for a while.  See https://github.com/phetsims/tambo/issues/194.
+if ( phetAudioContext.state && platform.safari ) {
+  document.addEventListener( 'visibilitychange', async () => {
+
+    // TODO: Remove some of this logging if the code is retained, see https://github.com/phetsims/tambo/issues/194.
+    console.log( `Visibility change detected, phetAudioContext.state = ${phetAudioContext.state}` );
+
+    if ( document.visibilityState === 'visible' && phetAudioContext.state !== 'running' ) {
+      try {
+        await phetAudioContext.resume();
+        console.log( 'AudioContext resumed after visibility change.' );
+      }
+      catch( err ) {
+        console.error( 'Failed to resume AudioContext:', err );
+      }
+    }
+  } );
+}
+
 export default phetAudioContext;
\ No newline at end of file

[jbphet]

@Nancy-Salpepi - I've created three variations on the theme of trying to resume the audio context after the sim regains visibility. Please try each one out on both MacOS and iOS and let me know if any of these seem to resolve the issues. They can be found at https://phet-dev.colorado.edu/jb-tests/, please test in this order:

• msab-always-resume_en.html - always try calling resume on the audio context when the sim regains visibility regardless of the audio context's reported state
• msab-resume-after-visibility-and-touch.html - call resume on the audio context on the first user gesture after the sim regains visibility
• msab-alert-and-resume.html - put up an alert message when returning to visibility, thus forcing the user to click/touch, then call resume on the audio context

For reference, here are patches for each of the three versions posted:

diff for msab-always-resume

Subject: [PATCH] when on Safari, always resume context after visibility state changes, see https://github.com/phetsims/tambo/issues/194
---
Index: js/phetAudioContext.ts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/phetAudioContext.ts b/js/phetAudioContext.ts
--- a/js/phetAudioContext.ts	(revision 46e24e04da54e3a7997682310bb1ec41600b7780)
+++ b/js/phetAudioContext.ts	(date 1732131525556)
@@ -7,6 +7,7 @@
  * @author John Blanco
  */
 
+import platform from '../../phet-core/js/platform.js';
 import IntentionalAny from '../../phet-core/js/types/IntentionalAny.js';
 import tambo from './tambo.js';
 
@@ -29,4 +30,25 @@
 // register for phet-io
 tambo.register( 'phetAudioContext', phetAudioContext );
 
+// If running on Safari, add a listener to resume the audio context when the page becomes visible.  This is necessary
+// only on Safari, which seems to have issues on both the mobile and desktop versions where audio can get delayed or
+// goes away altogether after the page is hidden for a while.  See https://github.com/phetsims/tambo/issues/194.
+if ( phetAudioContext.state && platform.safari ) {
+  document.addEventListener( 'visibilitychange', async () => {
+
+    // TODO: Remove some of this logging if the code is retained, see https://github.com/phetsims/tambo/issues/194.
+    console.log( `Visibility change detected, phetAudioContext.state = ${phetAudioContext.state}` );
+
+    if ( document.visibilityState === 'visible' ) {
+      try {
+        await phetAudioContext.resume();
+        console.log( 'AudioContext resumed after visibility change.' );
+      }
+      catch( err ) {
+        console.error( 'Failed to resume AudioContext:', err );
+      }
+    }
+  } );
+}
+
 export default phetAudioContext;
\ No newline at end of file

diff for msab-resume-after-visibility-and-touch

Subject: [PATCH] when on Safari, try to resume audio context on first user gesture after regaining visibility, see https://github.com/phetsims/tambo/issues/194
---
Index: js/soundManager.ts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/soundManager.ts b/js/soundManager.ts
--- a/js/soundManager.ts	(revision 46e24e04da54e3a7997682310bb1ec41600b7780)
+++ b/js/soundManager.ts	(date 1732147861856)
@@ -22,6 +22,7 @@
 import Utils from '../../dot/js/Utils.js';
 import arrayRemove from '../../phet-core/js/arrayRemove.js';
 import optionize from '../../phet-core/js/optionize.js';
+import platform from '../../phet-core/js/platform.js';
 import { Display } from '../../scenery/js/imports.js';
 import PhetioObject from '../../tandem/js/PhetioObject.js';
 import Tandem from '../../tandem/js/Tandem.js';
@@ -374,6 +375,41 @@
       );
     } );
     this.soundGeneratorsAwaitingAdd.length = 0;
+
+    // TODO: Don't commit this.  See https://github.com/phetsims/tambo/issues/194.
+
+    // If running on Safari, add a listener to resume the audio context when the page becomes visible.  This is
+    // necessary only on Safari, which seems to have issues on both the mobile and desktop versions where audio can get
+    // delayed or goes away altogether after the page is hidden for a while.  See
+    // https://github.com/phetsims/tambo/issues/194.
+    if ( phetAudioContext.state && platform.safari ) {
+
+      // Define a listener that will call `resume` on the audio context regardless of its current state.
+      const resumeRegardlessOfState = () => {
+        console.log( 'calling resume on audio context' );
+        phetAudioContext.resume()
+          .then( () => {
+            console.log( 'AudioContext resumed successfully.' );
+          } )
+          .catch( err => {
+            console.error( 'Failed to resume AudioContext:', err );
+          } );
+        Display.userGestureEmitter.removeListener( resumeRegardlessOfState );
+      };
+
+      document.addEventListener( 'visibilitychange', () => {
+
+        if ( document.visibilityState === 'visible' ) {
+
+          console.log( 'Sim returned to visibility, adding gesture listener to resume context.' );
+
+          // Add a listener that will attempt to make sure the audio context is working after the next user gesture.
+          if ( !Display.userGestureEmitter.hasListener( resumeRegardlessOfState ) ) {
+            Display.userGestureEmitter.addListener( resumeRegardlessOfState );
+          }
+        }
+      } );
+    }
   }
 
   /**

diff for msab-alert-and-resume

Subject: [PATCH] when on Safari, put up a dialog and resume context after visibility changes, see https://github.com/phetsims/tambo/issues/194
---
Index: js/phetAudioContext.ts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/phetAudioContext.ts b/js/phetAudioContext.ts
--- a/js/phetAudioContext.ts	(revision 46e24e04da54e3a7997682310bb1ec41600b7780)
+++ b/js/phetAudioContext.ts	(date 1732143072166)
@@ -7,6 +7,7 @@
  * @author John Blanco
  */
 
+import platform from '../../phet-core/js/platform.js';
 import IntentionalAny from '../../phet-core/js/types/IntentionalAny.js';
 import tambo from './tambo.js';
 
@@ -29,4 +30,26 @@
 // register for phet-io
 tambo.register( 'phetAudioContext', phetAudioContext );
 
+// If running on Safari, add a listener to resume the audio context when the page becomes visible.  This is necessary
+// only on Safari, which seems to have issues on both the mobile and desktop versions where audio can get delayed or
+// goes away altogether after the page is hidden for a while.  See https://github.com/phetsims/tambo/issues/194.
+if ( phetAudioContext.state && platform.safari ) {
+  document.addEventListener( 'visibilitychange', async () => {
+
+    // TODO: Remove some of this logging if the code is retained, see https://github.com/phetsims/tambo/issues/194.
+    console.log( `Visibility change detected, phetAudioContext.state = ${phetAudioContext.state}` );
+
+    if ( document.visibilityState === 'visible' ) {
+      window.alert( 'Continue' );
+      try {
+        await phetAudioContext.resume();
+        console.log( 'AudioContext resumed after visibility change.' );
+      }
+      catch( err ) {
+        console.error( 'Failed to resume AudioContext:', err );
+      }
+    }
+  } );
+}
+
 export default phetAudioContext;
\ No newline at end of file