[BUG] Deadlock when using DefaultAzureCredentials and virtual threads are available #46231
Description
Describe the bug
When using two or more DefaultAzureCredential instances to automatically discover Service Principal credentials from the environment and virtual threads are available and used by the SharedExecutorService, making any request (in our case blob storage listing) can lead to a deadlock in retrieving tokens. This is because acquiring a token is pinning the virtual thread. With only a few carrier threads available, this is quickly leading to a deadlock where the Azure SDK, and virtual threads in general, become unusable.
Exception or Stack Trace
First carrier thread
"ForkJoinPool-1-worker-10" daemon prio=5 tid=416 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.lang.VirtualThread.parkOnCarrierThread()
local variable: java.lang.VirtualThread#2
local variable: jdk.internal.event.VirtualThreadPinnedEvent#1
at java.lang.VirtualThread.park()
local variable: java.lang.VirtualThread#2
at java.lang.System$2.parkVirtualThread()
local variable: java.lang.System$2#1
local variable: java.lang.VirtualThread#2
local variable: java.lang.VirtualThread#2
at jdk.internal.misc.VirtualThreads.park()
at java.util.concurrent.locks.LockSupport.park()
local variable: java.lang.VirtualThread#2
at java.util.concurrent.CompletableFuture$Signaller.block()
local variable: java.util.concurrent.CompletableFuture$Signaller#1
at java.util.concurrent.ForkJoinPool.unmanagedBlock()
local variable: java.util.concurrent.CompletableFuture$Signaller#1
at java.util.concurrent.ForkJoinPool.managedBlock()
at java.util.concurrent.CompletableFuture.waitingGet()
local variable: java.util.concurrent.CompletableFuture#19
local variable: java.util.concurrent.CompletableFuture$Signaller#1
at java.util.concurrent.CompletableFuture.get()
local variable: java.util.concurrent.CompletableFuture#19
at com.azure.identity.implementation.IdentitySyncClient.authenticateWithConfidentialClientCache(IdentitySyncClient.java:173)
local variable: com.azure.identity.implementation.IdentitySyncClient#9
local variable: com.azure.core.credential.TokenRequestContext#3
local variable: com.microsoft.aad.msal4j.ConfidentialClientApplication#1
local variable: com.microsoft.aad.msal4j.SilentParameters$SilentParametersBuilder#1
at com.azure.identity.ClientSecretCredential.getTokenSync(ClientSecretCredential.java:124)
local variable: com.azure.identity.ClientSecretCredential#3
local variable: com.azure.core.credential.TokenRequestContext#3
at com.azure.identity.EnvironmentCredential.getTokenSync(EnvironmentCredential.java:186)
local variable: com.azure.identity.EnvironmentCredential#3
local variable: com.azure.core.credential.TokenRequestContext#3
at com.azure.identity.ChainedTokenCredential.getTokenSync(ChainedTokenCredential.java:130)
local variable: com.azure.identity.DefaultAzureCredential#3
local variable: com.azure.core.credential.TokenRequestContext#3
local variable: java.util.ArrayList#6042
at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
local variable: com.azure.core.implementation.AccessTokenCache#3
local variable: com.azure.identity.DefaultAzureCredential#3
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#3
at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
local variable: com.azure.core.implementation.AccessTokenCache#3
local variable: com.azure.core.credential.TokenRequestContext#1
local variable: com.azure.core.implementation.AccessTokenCacheInfo#3
local variable: com.azure.identity.implementation.MsalToken#2
local variable: java.time.OffsetDateTime#4
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#3
local variable: com.azure.identity.implementation.MsalToken#2
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40#1
at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
local variable: com.azure.core.implementation.AccessTokenCache#3
local variable: com.azure.core.credential.TokenRequestContext#1
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.credential.TokenRequestContext#1
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.credential.TokenRequestContext#1
at com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy.authorizeRequestSync(StorageBearerTokenChallengeAuthorizationPolicy.java:63)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: java.lang.String[]#1260
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.processSync(BearerTokenAuthenticationPolicy.java:174)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#2
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#3
at com.azure.storage.common.policy.MetadataValidationPolicy.processSync(MetadataValidationPolicy.java:42)
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#3
at com.azure.core.http.policy.AddHeadersFromContextPolicy.processSync(AddHeadersFromContextPolicy.java:67)
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#3
at com.azure.core.http.policy.AddDatePolicy.processSync(AddDatePolicy.java:50)
local variable: com.azure.core.http.policy.AddDatePolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#1
local variable: com.azure.core.http.policy.AddDatePolicy#3
at com.azure.storage.common.policy.RequestRetryPolicy.lambda$attemptSync$2(RequestRetryPolicy.java:261)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0.call()
local variable: com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0#1
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:62)
local variable: reactor.core.publisher.MonoCallable#15
local variable: java.time.Duration#229
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:55)
local variable: reactor.core.publisher.MonoCallable#15
at com.azure.storage.common.policy.RequestRetryPolicy.attemptSync(RequestRetryPolicy.java:270)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.HttpRequest#3
local variable: reactor.core.publisher.MonoCallable#15
at com.azure.storage.common.policy.RequestRetryPolicy.processSync(RequestRetryPolicy.java:71)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.HttpRequest#3
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.storage.common.policy.RequestRetryPolicy#3
at com.azure.core.http.policy.RequestIdPolicy.processSync(RequestIdPolicy.java:77)
local variable: com.azure.core.http.policy.RequestIdPolicy#3
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.policy.RequestIdPolicy#3
at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
local variable: com.azure.core.http.policy.UserAgentPolicy$1#4
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
local variable: com.azure.core.http.policy.UserAgentPolicy#4
local variable: com.azure.core.http.HttpPipelineCallContext#1
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
local variable: com.azure.core.http.policy.UserAgentPolicy#4
at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
local variable: com.azure.core.http.HttpPipeline#3
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#3
at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#26
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#26
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
local variable: com.azure.core.util.tracing.Utils$$Lambda+0x000078380135f268#1
at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#26
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: java.lang.Object[]#9601
local variable: com.azure.core.http.HttpRequest#3
local variable: com.azure.core.util.Context#7
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
local variable: com.azure.core.http.rest.RestProxy#26
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.reflect.Method#8034
local variable: java.lang.Object[]#9601
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.util.Context$1#1
at jdk.proxy2.$Proxy213.listBlobFlatSegmentSync()
local variable: jdk.proxy2.$Proxy213#5
local variable: java.lang.String#70505
local variable: java.lang.String#102455
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#39616
local variable: java.lang.String#102453
local variable: java.lang.String#139122
local variable: com.azure.core.util.Context$1#1
at com.azure.storage.blob.implementation.ContainersImpl.listBlobFlatSegmentWithResponse(ContainersImpl.java:5959)
local variable: com.azure.storage.blob.implementation.ContainersImpl#5
local variable: java.lang.String#102455
local variable: java.lang.String#39616
local variable: com.azure.core.util.Context$1#1
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#139122
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$6(BlobContainerClient.java:1046)
local variable: com.azure.storage.blob.BlobContainerClient#5
local variable: com.azure.storage.blob.models.ListBlobsOptions#1
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760.call()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#1
at com.azure.storage.common.implementation.StorageImplUtils.sendRequest(StorageImplUtils.java:494)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#1
local variable: class com.azure.storage.blob.models.BlobStorageException
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$7(BlobContainerClient.java:1050)
local variable: com.azure.storage.blob.BlobContainerClient#5
local variable: com.azure.storage.blob.models.ListBlobsOptions#2
local variable: com.azure.storage.blob.models.ListBlobsOptions#1
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#1
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#1
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$8(BlobContainerClient.java:1065)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#1
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#1
at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#1
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#1
at com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40.getPage()
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#1
at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#1
local variable: java.util.concurrent.atomic.AtomicBoolean#270
at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.(ContinuablePagedByItemIterable.java:83)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#1
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#1
local variable: com.azure.core.http.rest.PagedIterableBase$$Lambda+0x00007838015367b0#1
at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:58)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable#1
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#1
at java.lang.Iterable.spliterator()
at com.azure.core.util.paging.ContinuablePagedIterable.stream(ContinuablePagedIterable.java:85)
local variable: com.azure.core.http.rest.PagedIterable#1
Second carrier thread
"ForkJoinPool-1-worker-9" daemon prio=5 tid=417 WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
at java.lang.VirtualThread.parkOnCarrierThread()
local variable: java.lang.VirtualThread#3
local variable: jdk.internal.event.VirtualThreadPinnedEvent#2
at java.lang.VirtualThread.park()
local variable: java.lang.VirtualThread#3
at java.lang.System$2.parkVirtualThread()
local variable: java.lang.System$2#1
local variable: java.lang.VirtualThread#3
local variable: java.lang.VirtualThread#3
at jdk.internal.misc.VirtualThreads.park()
at java.util.concurrent.locks.LockSupport.park()
local variable: java.lang.VirtualThread#3
at java.util.concurrent.CompletableFuture$Signaller.block()
local variable: java.util.concurrent.CompletableFuture$Signaller#2
at java.util.concurrent.ForkJoinPool.unmanagedBlock()
local variable: java.util.concurrent.CompletableFuture$Signaller#2
at java.util.concurrent.ForkJoinPool.managedBlock()
at java.util.concurrent.CompletableFuture.waitingGet()
local variable: java.util.concurrent.CompletableFuture#21
local variable: java.util.concurrent.CompletableFuture$Signaller#2
at java.util.concurrent.CompletableFuture.get()
local variable: java.util.concurrent.CompletableFuture#21
at com.azure.identity.implementation.IdentitySyncClient.authenticateWithConfidentialClientCache(IdentitySyncClient.java:173)
local variable: com.azure.identity.implementation.IdentitySyncClient#18
local variable: com.azure.core.credential.TokenRequestContext#4
local variable: com.microsoft.aad.msal4j.ConfidentialClientApplication#2
local variable: com.microsoft.aad.msal4j.SilentParameters$SilentParametersBuilder#2
at com.azure.identity.ClientSecretCredential.getTokenSync(ClientSecretCredential.java:124)
local variable: com.azure.identity.ClientSecretCredential#6
local variable: com.azure.core.credential.TokenRequestContext#4
at com.azure.identity.EnvironmentCredential.getTokenSync(EnvironmentCredential.java:186)
local variable: com.azure.identity.EnvironmentCredential#6
local variable: com.azure.core.credential.TokenRequestContext#4
at com.azure.identity.ChainedTokenCredential.getTokenSync(ChainedTokenCredential.java:130)
local variable: com.azure.identity.DefaultAzureCredential#6
local variable: com.azure.core.credential.TokenRequestContext#4
local variable: java.util.ArrayList#6043
at com.azure.core.implementation.AccessTokenCache.lambda$new$2(AccessTokenCache.java:65)
local variable: com.azure.core.implementation.AccessTokenCache#6
local variable: com.azure.identity.DefaultAzureCredential#6
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#6
at com.azure.core.implementation.AccessTokenCache.lambda$retrieveTokenSync$11(AccessTokenCache.java:230)
local variable: com.azure.core.implementation.AccessTokenCache#6
local variable: com.azure.core.credential.TokenRequestContext#2
local variable: com.azure.core.implementation.AccessTokenCacheInfo#4
local variable: com.azure.identity.implementation.MsalToken#1
local variable: java.time.OffsetDateTime#5
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783800f17958#6
local variable: com.azure.identity.implementation.MsalToken#1
at com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40.get()
local variable: com.azure.core.implementation.AccessTokenCache$$Lambda+0x0000783801368d40#2
at com.azure.core.implementation.AccessTokenCache.getTokenSync(AccessTokenCache.java:93)
local variable: com.azure.core.implementation.AccessTokenCache#6
local variable: com.azure.core.credential.TokenRequestContext#2
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderHelperSync(BearerTokenAuthenticationPolicy.java:220)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.credential.TokenRequestContext#2
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.setAuthorizationHeaderSync(BearerTokenAuthenticationPolicy.java:207)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.credential.TokenRequestContext#2
at com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy.authorizeRequestSync(StorageBearerTokenChallengeAuthorizationPolicy.java:63)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: java.lang.String[]#1261
at com.azure.core.http.policy.BearerTokenAuthenticationPolicy.processSync(BearerTokenAuthenticationPolicy.java:174)
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#5
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.storage.common.policy.StorageBearerTokenChallengeAuthorizationPolicy#6
at com.azure.storage.common.policy.MetadataValidationPolicy.processSync(MetadataValidationPolicy.java:42)
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.storage.common.policy.MetadataValidationPolicy#6
at com.azure.core.http.policy.AddHeadersFromContextPolicy.processSync(AddHeadersFromContextPolicy.java:67)
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.core.http.policy.AddHeadersFromContextPolicy#6
at com.azure.core.http.policy.AddDatePolicy.processSync(AddDatePolicy.java:50)
local variable: com.azure.core.http.policy.AddDatePolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#4
local variable: com.azure.core.http.policy.AddDatePolicy#6
at com.azure.storage.common.policy.RequestRetryPolicy.lambda$attemptSync$2(RequestRetryPolicy.java:261)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0.call()
local variable: com.azure.storage.common.policy.RequestRetryPolicy$$Lambda+0x0000783801366eb0#2
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:62)
local variable: reactor.core.publisher.MonoCallable#16
local variable: java.time.Duration#229
at reactor.core.publisher.MonoCallable.block(MonoCallable.java:55)
local variable: reactor.core.publisher.MonoCallable#16
at com.azure.storage.common.policy.RequestRetryPolicy.attemptSync(RequestRetryPolicy.java:270)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.HttpRequest#4
local variable: reactor.core.publisher.MonoCallable#16
at com.azure.storage.common.policy.RequestRetryPolicy.processSync(RequestRetryPolicy.java:71)
local variable: com.azure.storage.common.policy.RequestRetryPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.HttpRequest#4
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.storage.common.policy.RequestRetryPolicy#6
at com.azure.core.http.policy.RequestIdPolicy.processSync(RequestIdPolicy.java:77)
local variable: com.azure.core.http.policy.RequestIdPolicy#6
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.policy.RequestIdPolicy#6
at com.azure.core.http.policy.HttpPipelineSyncPolicy.processSync(HttpPipelineSyncPolicy.java:51)
local variable: com.azure.core.http.policy.UserAgentPolicy$1#8
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.http.policy.UserAgentPolicy.processSync(UserAgentPolicy.java:174)
local variable: com.azure.core.http.policy.UserAgentPolicy#8
local variable: com.azure.core.http.HttpPipelineCallContext#2
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.http.HttpPipelineNextSyncPolicy.processSync(HttpPipelineNextSyncPolicy.java:53)
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
local variable: com.azure.core.http.policy.UserAgentPolicy#8
at com.azure.core.http.HttpPipeline.sendSync(HttpPipeline.java:138)
local variable: com.azure.core.http.HttpPipeline#7
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
local variable: com.azure.core.http.HttpPipelineNextSyncPolicy#6
at com.azure.core.implementation.http.rest.SyncRestProxy.send(SyncRestProxy.java:62)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#32
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:83)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#32
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
local variable: com.azure.core.util.tracing.Utils$$Lambda+0x000078380135f268#1
at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124)
local variable: com.azure.core.implementation.http.rest.SyncRestProxy#32
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.reflect.Method#8034
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: java.lang.Object[]#9605
local variable: com.azure.core.http.HttpRequest#4
local variable: com.azure.core.util.Context#8
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95)
local variable: com.azure.core.http.rest.RestProxy#32
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.reflect.Method#8034
local variable: java.lang.Object[]#9605
local variable: com.azure.core.implementation.http.rest.SwaggerMethodParser#2
local variable: com.azure.core.util.Context$1#1
at jdk.proxy2.$Proxy213.listBlobFlatSegmentSync()
local variable: jdk.proxy2.$Proxy213#6
local variable: java.lang.String#182829
local variable: java.lang.String#102455
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#39623
local variable: java.lang.String#102453
local variable: java.lang.String#139122
local variable: com.azure.core.util.Context$1#1
at com.azure.storage.blob.implementation.ContainersImpl.listBlobFlatSegmentWithResponse(ContainersImpl.java:5959)
local variable: com.azure.storage.blob.implementation.ContainersImpl#6
local variable: java.lang.String#102455
local variable: java.lang.String#39623
local variable: com.azure.core.util.Context$1#1
local variable: java.lang.String#169761
local variable: java.lang.String#84991
local variable: java.lang.String#139122
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$6(BlobContainerClient.java:1046)
local variable: com.azure.storage.blob.BlobContainerClient#6
local variable: com.azure.storage.blob.models.ListBlobsOptions#3
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760.call()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#2
at com.azure.storage.common.implementation.StorageImplUtils.sendRequest(StorageImplUtils.java:494)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#2
local variable: class com.azure.storage.blob.models.BlobStorageException
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$7(BlobContainerClient.java:1050)
local variable: com.azure.storage.blob.BlobContainerClient#6
local variable: com.azure.storage.blob.models.ListBlobsOptions#4
local variable: com.azure.storage.blob.models.ListBlobsOptions#3
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801537760#2
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#2
at com.azure.storage.blob.BlobContainerClient.lambda$listBlobs$8(BlobContainerClient.java:1065)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#2
at com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170.apply()
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#2
at com.azure.core.http.rest.PagedIterable.lambda$new$5(PagedIterable.java:193)
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801536170#2
local variable: com.azure.storage.blob.BlobContainerClient$$Lambda+0x0000783801534938#2
at com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40.getPage()
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#2
at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:104)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#2
local variable: java.util.concurrent.atomic.AtomicBoolean#271
at com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator.(ContinuablePagedByItemIterable.java:83)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#2
local variable: com.azure.core.http.rest.PagedIterable$$Lambda+0x0000783801536e40#2
local variable: com.azure.core.http.rest.PagedIterableBase$$Lambda+0x00007838015367b0#1
at com.azure.core.util.paging.ContinuablePagedByItemIterable.iterator(ContinuablePagedByItemIterable.java:58)
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable#2
local variable: com.azure.core.util.paging.ContinuablePagedByItemIterable$ContinuablePagedByItemIterator#2
at java.lang.Iterable.spliterator()
at com.azure.core.util.paging.ContinuablePagedIterable.stream(ContinuablePagedIterable.java:85)
local variable: com.azure.core.http.rest.PagedIterable#2
The heap dump shows multiple virtual threads. azure-sdk-global-thread-5 and azure-sdk-global-thread-4 are started but don't have a carrier thread yet. Thread #415 and #416 are the threads from above that are pinned to their carrier threads.
To Reproduce
Steps to reproduce the behavior:
- Have virtual threads available
- Have few CPUs available, leading to a small number of carrier threads
- Have multiple
DefaultAzureCredentialinstances - Make multiple concurrent request that need to retrieve a new token from the credential chain
This is the scenario that leads to fairly reliable deadlocks. I'm not actually sure if multiple DefaultAzureCredential instances are required or not, or even if the issue would also occur when using one of the authentication methods explicitly.
Expected behavior
No deadlocks. The Azure SDK should not use virtual threads if it pins them by using synchronized or calling into native code.
Setup (please complete the following information):
- OS: Linux
- Library/Libraries: Azure SDK 1.2.36
- Java version: 21
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
- Bug Description Added
- Repro Steps Added
- Setup information Added