Skip to content

Replace power-of-two batch sizes with max compiled model approach#217

Open
ahsan-ca wants to merge 4 commits intorocm7.2_internal_testingfrom
add-max-compile-models
Open

Replace power-of-two batch sizes with max compiled model approach#217
ahsan-ca wants to merge 4 commits intorocm7.2_internal_testingfrom
add-max-compile-models

Conversation

@ahsan-ca
Copy link
Collaborator

@ahsan-ca ahsan-ca commented Feb 20, 2026

Description

Replace power-of-two batch sizes with max compiled model approach.

ORT_MIGRAPHX_MAX_COMPILED_MODELS=N, such that

max_compiled_models == 1 -> compile the max batch size.
max_compiled_models == 2 -> compile batch size 1 and max batch size.
max_compiled_models >= 3 -> compile batch sizes of 1, some mid values, max.

max_compiled_models == 0 -> set max_compiled_models to 1.
max_compiled_models > max batch size -> set max_compiled_models to max batch size.

This helps to reduce memory usage for higher batch sizes.

Motivation and Context

@ahsan-ca ahsan-ca self-assigned this Feb 20, 2026
@ahsan-ca ahsan-ca force-pushed the add-max-compile-models branch 4 times, most recently from ebf5a53 to 4a86a96 Compare February 20, 2026 17:24
@ahsan-ca ahsan-ca force-pushed the add-max-compile-models branch from 4a86a96 to 03c1250 Compare February 20, 2026 17:35
@ahsan-ca ahsan-ca force-pushed the add-max-compile-models branch 2 times, most recently from 1f31434 to 02f8b69 Compare February 23, 2026 20:16
@ahsan-ca ahsan-ca force-pushed the add-max-compile-models branch from 02f8b69 to eecc53b Compare February 25, 2026 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants