Support Flashinfer FP8 ViT Attn

[zhandaz]

Summary

• Add FP8 quantization support for ViT attn in MMEncoderAttention (through Flashinfer and cudnn)
• Implement a stride-aware Triton kernel for FP8 quantization that handles non-contiguous QKV views and pads head_dim to a multiple of 16 (e.g., 72 -> 80) without extra copies
• Plumb q_scale/k_scale/v_scale/o_data_type through the FlashInfer cuDNN wrapper to support FP8 input tensors
• Adjust compute_flashinfer_cu_seqlens in Qwen3-VL to produce correct element offsets for the padded FP8 tensor layout

How to enable

FP8 ViT Attn is only enabled with --mm-encoder-attn-backend=FLASHINFER

export VLLM_MM_ENCODER_FP8_ATTN=1
# All one scales is fine in our case, and scales can be provided through (if necessary)
# export VLLM_MM_ENCODER_FP8_ATTN_SCALE_PATH=/path/to/scales.json

The scales JSON should map layer names to per-tensor Q/K/V scales, e.g.:

{"visual.blocks.0.attn": {"q": 0.0123, "k": 0.0456, "v": 0.0789}, ...}

If VLLM_MM_ENCODER_FP8_ATTN_SCALE_PATH is not set, scale=1.0 is used for all layers (cast-only, no scaling).

Dependency

• Requires a corresponding FlashInfer change (separate PR in the FlashInfer repo, Pulled the fp8 attention support & minor fix for compatibility flashinfer#2) to expose q_scale/k_scale/v_scale/o_data_type parameters in cudnn_batch_prefill_with_kv_cache.
• This PR should be merged after Add bilinear_pos_embed triton kernel and cache #40 and rebased on top of that.

Files changed

• vllm/envs.py -- register VLLM_MM_ENCODER_FP8_ATTN and VLLM_MM_ENCODER_FP8_ATTN_SCALE_PATH
• vllm/model_executor/layers/attention/mm_encoder_attention.py -- FP8 init, scale loading, quantization, FlashInfer integration
• vllm/model_executor/layers/quantization/input_quant_fp8.py -- stride-aware Triton kernel for padded FP8 quantization
• vllm/model_executor/models/qwen3_vl.py -- FP8-aware cu_seqlens computation
• vllm/v1/attention/ops/vit_attn_wrappers.py -- pass FP8 scale/dtype params to cuDNN