Skip to content

Conversation

@shreed27
Copy link

@shreed27 shreed27 commented Feb 1, 2026

Refactor vision token utils to decouple from tokenizer

Description

Refactors _token_utils.py to accept special_tokens as an argument instead of hardcoding Gemma3Tokenizer.
This addresses the TODO This value should be propagated from the model. and makes the utility completely model-agnostic.

Changes

  • gemma/gm/vision/_token_utils.py:
    • Updated add_extra_tokens_for_images to accept special_tokens.
    • Updated remove_mm_logits to accept special_tokens.
    • Removed hardcoded _tokenizer.Gemma3Tokenizer usage.
  • gemma/gm/utils/_types.py: Propagate self.config.special_tokens to add_extra_tokens_for_images.
  • gemma/gm/nn/_transformer.py: Propagate self.config.input_config.special_tokens to remove_mm_logits.
  • gemma/gm/nn/gemma3n/_transformer.py: Propagate self.config.input_config.special_tokens to remove_mm_logits.
  • gemma/gm/vision/_token_utils_test.py: Updated tests to use a mock SpecialTokens class.

Verification

  • Verified function signatures and basic logic via script (mocking dependencies).
  • Unit tests updated to match new signature.

@google-cla
Copy link

google-cla bot commented Feb 1, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@shreed27 shreed27 force-pushed the refactor/vision-token-utils branch from 8252207 to b8fc3a4 Compare February 1, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant