Skip to content

change resampler attention to scale_dot_product_attention#446

Open
Benkovichnikita wants to merge 1 commit intotencent-ailab:mainfrom
Benkovichnikita:feat/attention_optimization
Open

change resampler attention to scale_dot_product_attention#446
Benkovichnikita wants to merge 1 commit intotencent-ailab:mainfrom
Benkovichnikita:feat/attention_optimization

Conversation

@Benkovichnikita
Copy link

@Benkovichnikita Benkovichnikita commented Nov 19, 2024

I noticed two issues:

  1. PerceiverAttention consumes a huge amount of memory while calculating an attention map
  2. There are two operations of math.srqt when scale calculating: scale = 1 / math.sqrt(math.sqrt(self.dim_head))

Both of them can be fixed using more effective implementation: F.scaled_dot_product_attention(q, k, v).

PS
If you need scale with two sqrt, it can be provided as an additional argument to F.scaled_dot_product_attention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant