Add universal param configs by knjiang · Pull Request #61 · braintrustdata/lingua

knjiang · 2026-01-16T16:48:43Z

Summary

This PR adds cross-provider compatability for chat completions, responses, anthropic.

See -> https://github.com/braintrustdata/lingua/actions/runs/21328545339

Parameter mappings

Feature	Chat Completions	Responses	Anthropic
Reasoning	`reasoning_effort`	`reasoning.effort`	`thinking.budget_tokens`
Structured output	`response_format.json_schema`	`text.format`	`output_format`
Tool selection	`tool_choice`	`tool_choice`	`tool_choice` + `disable_parallel_tool_use`
Max tokens	`max_tokens` / `max_completion_tokens`	`max_output_tokens`	`max_tokens`

Testing

For each provider pair (A → B) across Chat Completions / Responses / Anthropic, we validate the deserialized Universal payload:

Universal of source payload
- U₁ = A payload → Universal
Translate across providers and re-canonicalize
- U₂ = (A payload → Universal → B payload) → Universal
Diff the canonical forms
- Compare U₁ vs U₂
- Emit field-level diffs for any lost / added / changed fields
Enforce in CI
- CI fails on any unexpected diffs

Expected differences

Known provider limitations / intentional lossy mappings are documented in expected_differences.json.
Diffs covered by this file are treated as allowed; anything else is flagged as a regression and fails CI.

knjiang · 2026-01-16T16:48:54Z

Support google parameters #75
add google test cases #74
add google roundtrip #70
testing framework for transformRequest+Response #72
lingua-wasm bindings for request/response #69
Add universal param configs #61 👈 (View in Graphite)
add anthropic messages parameter test cases #59
add chat completion parameter test cases #58
add openai responses parameter test cases #54
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

knjiang · 2026-01-21T21:38:13Z

crates/lingua/src/providers/openai/adapter.rs

-/// Known request fields for OpenAI Responses API.
-/// These are fields extracted into UniversalRequest/UniversalParams.
-/// Fields not in this list go into `extras` for passthrough.
-const RESPONSES_KNOWN_KEYS: &[&str] = &[


i move responses to responses_adapter.rs

knjiang · 2026-01-22T18:21:16Z

crates/coverage-report/src/runner.rs

+                        target_adapter.display_name(),
+                        test_case,
+                    );
+                    let roundtrip_result = compare_values(


i tightened the runner to be more accurate, basically we now compare:

source -> universal

with

source -> universal -> target -> universal

we basically deserialize universal -> JSON and do diffing

crates/coverage-report/src/expected_differences.json

crates/coverage-report/src/runner.rs

crates/lingua/src/universal/tool_choice.rs

crates/lingua/src/universal/tools.rs

crates/lingua/src/universal/request.rs

crates/lingua/src/universal/tools.rs

crates/coverage-report/src/expected.rs

ankrgyl · 2026-01-25T20:55:33Z

crates/lingua/src/universal/request.rs

-    /// Tool selection strategy (varies by provider)
-    pub tool_choice: Option<Value>,
+    /// Number of top logprobs to return (0-20)
+    pub top_logprobs: Option<i64>,


nit: if it's 1-20 can it be a smaller integer type like i8?

the openai generated type equivalent is i64 -> https://github.com/braintrustdata/lingua/blob/main/crates/lingua/src/providers/openai/generated.rs#L69

ankrgyl · 2026-01-25T20:59:19Z

crates/lingua/src/universal/request.rs

+    // === Metadata and identification ===
+    /// Request metadata (user tracking, experiment tags, etc.)
+    pub metadata: Option<Value>,


what is this

responses/chatcompletions have it as:

metadata map Optional Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard. Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

while anthrpoic has metadata as an object with one field user_id.

https://platform.openai.com/docs/api-reference/responses/create#responses_create-metadata
https://platform.claude.com/docs/en/api/messages

ahh ok. mind linking these in the comment

ankrgyl · 2026-01-25T21:00:28Z

crates/lingua/src/universal/request.rs

+    /// Example: OpenAI Chat extras stay in `provider_extras[ProviderFormat::OpenAI]`
+    /// and are only merged back when converting to OpenAI Chat, not to Anthropic.
+    #[serde(skip)]
+    pub provider_extras: HashMap<ProviderFormat, Map<String, Value>>,


it's slightly weird that this is not nested in params , at least to me. What as the rationale behind that?

moved inside universal params -> https://github.com/braintrustdata/lingua/blob/add_universal_params_between_completions_responses_and_anthropic/crates/lingua/src/universal/request.rs#L174

ankrgyl

Looks pretty good straightforward to me

It's a little out of date, but it would be useful to write some typescript examples (eg in examples/typescript/index.ts) or even some rust examples that show the ergonomics of using parameters, so we can double check the format
for parameters, i think it would be useful to write a "fuzz" style tester that for each provider, generates random values with respect to the openapi spec, and then roundtrips through UniversalParams (there is less entropy in parameters than raw requests, but this might just be generally useful)
Is there a creative way we can port the test cases we have in the proxy/ repo? We have had a bunch of historical challenges with translating reasoning for example that is well captured in those tests.

ankrgyl · 2026-01-27T00:54:11Z

Looks pretty good straightforward to me

It's a little out of date, but it would be useful to write some typescript examples (eg in examples/typescript/index.ts) or even some rust examples that show the ergonomics of using parameters, so we can double check the format

for parameters, i think it would be useful to write a "fuzz" style tester that for each provider, generates random values with respect to the openapi spec, and then roundtrips through UniversalParams (there is less entropy in parameters than raw requests, but this might just be generally useful)

Is there a creative way we can port the test cases we have in the proxy/ repo? We have had a bunch of historical challenges with translating reasoning for example that is well captured in those tests.

just to clarify, did you address these too?

remh · 2026-01-27T15:37:32Z

crates/lingua/src/universal/reasoning.rs

+/// - ratio >= 0.65: high
+pub fn budget_to_effort(budget: i64, max_tokens: Option<i64>) -> ReasoningEffort {
+    let max = max_tokens.unwrap_or(DEFAULT_MAX_TOKENS);
+    let ratio = budget as f64 / max as f64;


Are we enforcing that max is a strictly positive integer?

good catch! adddedtests

remh · 2026-01-27T15:40:11Z

crates/lingua/src/error.rs

    MissingRequiredField { field: String },

-    #[error("Invalid role: {role}")]
-    InvalidRole { role: String },


FYI we still refer to this error in https://github.com/braintrustdata/lingua/blob/add_universal_params_between_completions_responses_and_anthropic/crates/lingua/docs/ADDING_PROVIDER_FORMAT.md#implementing-tryfromllm-conversions

remh · 2026-01-27T15:41:54Z

crates/lingua/src/universal/tools.rs

+                );
+
+                // Responses API function tools have strict: false by default
+                obj.insert("strict".into(), Value::Bool(false));


what if a request had strict: true ? wouldn't that override it ?

chat completions don't have strict;true for tools. google and anthropic does though so i'll preemptively add support now.

i realized our anthropic spec is out of date

remh · 2026-01-27T15:46:41Z

crates/lingua/src/universal/response.rs

+                    "completion_tokens": completion,
+                    "total_tokens": prompt + completion
+                });
+                let obj_map = obj.as_object_mut().unwrap();


should we avoid using unwrap() and use expect() instead to give more context in case of crash? I think there are a few places in your PR where you use unwrap().

remh · 2026-01-27T15:49:25Z

crates/lingua/src/universal/reasoning.rs

+pub const EFFORT_HIGH_MULTIPLIER: f64 = 0.75;
+
+/// Threshold below which budget is considered "low" effort
+pub const EFFORT_LOW_THRESHOLD: f64 = 0.35;


are those thresholds documented anywhere? or is it just something you came up with (which would be fine)

these are copied from the old proxy as a starting point.

remh

A few comments that need to be addressed but otherwise it looks good to me.

knjiang · 2026-01-28T20:58:56Z

It's a little out of date, but it would be useful to write some typescript examples (eg in examples/typescript/index.ts) or even some rust examples that show the ergonomics of using parameters, so we can double check the format

for parameters, i think it would be useful to write a "fuzz" style tester that for each provider, generates random values with respect to the openapi spec, and then roundtrips through UniversalParams (there is less entropy in parameters than raw requests, but this might just be generally useful)

Is there a creative way we can port the test cases we have in the proxy/ repo? We have had a bunch of historical challenges with translating reasoning for example that is well captured in those tests.

deferring to lingua-wasm bindings for request/response #69
kk, i'll separate out the fuzz tester?
done in this PR by pulling in the test cases -> f1340cb

there are still some proxy failures but i got most tests passing.

proxyAnthropicAudioError - audio PR
proxyOpenAIO3MiniReasoning - openai capabilitiescapabilities
proxyGoogleReasoning - google next PR
proxyGoogleAudioSupport -  google next PR
proxyGoogleVideoSupport - google next PR
proxyAzureParamFiltering - capabilities
proxyOpenAIO3MiniStreamingReasoning - openai capabilities
proxyOpenAIPdfUrlConversion - I'm not sure how to do this, we would have to make a network call and then encode(?)

knjiang · 2026-01-29T05:40:30Z

payloads/proxy/cases.ts

@@ -0,0 +1,915 @@
+import OpenAI from "openai";


all the cases i ported from proxy that "fit" into this. some of the proxy cases intercepted calls so that doesn't really work yet. I have a subsequent PR with a more "intercept"-like approach. https://app.graphite.com/github/pr/braintrustdata/lingua/72

there are still some proxy failures but i got most tests passing.

proxyAnthropicAudioError - audio PR proxyOpenAIO3MiniReasoning - openai capabilitiescapabilities proxyGoogleReasoning - google next PR proxyGoogleAudioSupport - google next PR proxyGoogleVideoSupport - google next PR proxyAzureParamFiltering - capabilities proxyOpenAIO3MiniStreamingReasoning - openai capabilities proxyOpenAIPdfUrlConversion - I'm not sure how to do this, we would have to make a network call and then encode(?)

This was referenced Jan 16, 2026

add anthropic messages parameter test cases #59

Merged

add chat completion parameter test cases #58

Merged

add openai responses parameter test cases #54

Merged

knjiang changed the title ~~add universla param configs~~ add universal param configs Jan 16, 2026

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from 7fbb91f to 64f4bad Compare January 21, 2026 18:58

knjiang force-pushed the add_anthropic_parameter_test_cases branch from 8d7eba7 to 9d2ccf3 Compare January 21, 2026 18:58

knjiang marked this pull request as ready for review January 21, 2026 19:01

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from 64f4bad to e569a99 Compare January 21, 2026 21:01

knjiang force-pushed the add_anthropic_parameter_test_cases branch from 9d2ccf3 to bd8d790 Compare January 21, 2026 21:01

knjiang commented Jan 21, 2026

View reviewed changes

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch 3 times, most recently from cf02e72 to 349a05d Compare January 22, 2026 10:06

knjiang commented Jan 22, 2026

View reviewed changes

knjiang changed the title ~~add universal param configs~~ Add universal param configs Jan 22, 2026

remh requested changes Jan 22, 2026

View reviewed changes

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch 5 times, most recently from a75144e to 4e0703c Compare January 23, 2026 00:19

knjiang force-pushed the add_anthropic_parameter_test_cases branch 2 times, most recently from f2ba481 to c8a3e48 Compare January 23, 2026 00:19

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from 4e0703c to 5aa114b Compare January 23, 2026 00:20

knjiang force-pushed the add_anthropic_parameter_test_cases branch from c8a3e48 to f2ba481 Compare January 23, 2026 00:20

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch 2 times, most recently from 4e0703c to 561180b Compare January 23, 2026 00:40

knjiang force-pushed the add_anthropic_parameter_test_cases branch from f2ba481 to a38f078 Compare January 23, 2026 00:40

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from 561180b to c94e076 Compare January 23, 2026 01:18

ankrgyl reviewed Jan 25, 2026

View reviewed changes

add universla param configs

e975c83

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from b2a3e48 to 0ffbf84 Compare January 26, 2026 01:12

knjiang requested a review from ankrgyl January 26, 2026 14:42

remh reviewed Jan 27, 2026

View reviewed changes

remh requested changes Jan 27, 2026

View reviewed changes

address PR comments

80528b1

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from 0ffbf84 to 80528b1 Compare January 28, 2026 00:33

add proxy test cases

f1340cb

This was referenced Jan 28, 2026

lingua-wasm bindings for request/response #69

Open

add google roundtrip #70

Open

knjiang mentioned this pull request Jan 29, 2026

testing framework for transformRequest+Response #72

Open

address proxy edge cases

af4a595

knjiang force-pushed the add_universal_params_between_completions_responses_and_anthropic branch from 52805d2 to af4a595 Compare January 29, 2026 05:38

knjiang commented Jan 29, 2026

View reviewed changes

knjiang requested a review from remh January 30, 2026 20:00

update reasoning with canonical source and fix convert.rs for responses

520fa1f

remh approved these changes Feb 2, 2026

View reviewed changes

This was referenced Feb 2, 2026

add google test cases #74

Open

Support google parameters #75

Open

knjiang merged commit c9d8323 into main Feb 3, 2026
5 checks passed

Conversation

knjiang commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Parameter mappings

Testing

Expected differences

Uh oh!

knjiang commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankrgyl left a comment

Choose a reason for hiding this comment

Uh oh!

ankrgyl commented Jan 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knjiang Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

remh left a comment

Choose a reason for hiding this comment

Uh oh!

knjiang commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

knjiang commented Jan 16, 2026 •

edited

Loading

knjiang commented Jan 16, 2026 •

edited

Loading

knjiang Jan 27, 2026 •

edited

Loading

knjiang commented Jan 28, 2026 •

edited

Loading