feat: Add toolsets parameter for agent execution #146

devin-ai-integration · 2026-01-26T16:37:20Z

Summary

Adds the toolsets parameter to the agent.execute() method, allowing users to explicitly specify which tool categories should be available during agent execution. This exposes the new toolset functionality added in vlm-lab#1524.

Changes:

Added AgentToolset Literal type with 8 categories: core, image, image-gen, 3d_reconstruction, viz, document, video, web
Added optional toolsets parameter to agent.execute() method

Updates since last revision

Renamed parameter from toolset to toolsets (plural) per reviewer feedback
Updated category names: image_analysis → image, image_generation → image-gen, visualization → viz
Removed skills category (now 8 categories instead of 9)

Review & Testing Checklist for Human

Verify the 8 toolset category names exactly match the backend implementation in vlm-lab/vlm/agents/types.py (TOOL_CATEGORIES dict keys) - especially confirm the hyphenated image-gen format
Confirm the backend expects toolsets (plural) not toolset (singular) in the request payload
Test with an actual API call: client.agent.execute(name="...", toolsets=["image", "core"]) to confirm the backend accepts the parameter
Consider whether agent.completions (OpenAI SDK wrapper) also needs toolsets support - currently users would need to use extra_body={"toolsets": [...]} which isn't documented

Notes

This is part of a larger task to expose the toolsets parameter across docs, cookbook, and both SDKs. Companion PRs:

Docs: vlm-run/docs#172
Node SDK: feat: Add toolsets parameter for agent execution vlmrun-node-sdk#122
Cookbook: feat: Add toolsets parameter to each chat_completion call in cookbook notebooks vlmrun-cookbook#94

Link to Devin run: https://app.devin.ai/sessions/a73e91efc6644456a83528267b200be1
Requested by: dinesh@vlm.run

Add AgentToolset type and toolset parameter to agent.execute() method. This allows users to explicitly specify which tool categories to enable for agent execution requests. Available categories: core, image_analysis, image_generation, 3d_reconstruction, visualization, document, video, web, skills. Co-Authored-By: dinesh@vlm.run <dinesh@vlm.run>

devin-ai-integration · 2026-01-26T16:37:23Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

spillai · 2026-01-26T17:17:27Z

vlmrun/client/agent.py

        metadata: Optional[RequestMetadata] = None,
        callback_url: Optional[str] = None,
        model: str = "vlmrun-orion-1:auto",
+        toolset: Optional[List[AgentToolset]] = None,


Use toolsets instead of toolset

spillai · 2026-01-26T18:59:54Z

vlmrun/client/agent.py

            data["callback_url"] = callback_url

+        if toolset is not None:
+            data["toolset"] = toolset


Use data[toolsets]

Co-Authored-By: dinesh@vlm.run <dinesh@vlm.run>

spillai · 2026-01-26T19:00:44Z

vlmrun/client/types.py

+AgentToolset = Literal[
+    "core",
+    "image_analysis",
+    "image_generation",


image-gen

spillai · 2026-01-26T19:00:49Z

vlmrun/client/types.py

+# AgentToolset type - tool categories available for agent execution
+AgentToolset = Literal[
+    "core",
+    "image_analysis",


Use image

spillai · 2026-01-26T19:01:31Z

vlmrun/client/types.py

+    "image_analysis",
+    "image_generation",
+    "3d_reconstruction",
+    "visualization",


spillai · 2026-01-26T19:01:49Z

vlmrun/client/types.py

+    "document",
+    "video",
+    "web",
+    "skills",


Drop skills

- image_analysis -> image - image_generation -> image-gen - visualization -> viz - Remove skills category Co-Authored-By: dinesh@vlm.run <dinesh@vlm.run>

… notebooks (#94) ## Summary Adds the `toolsets` parameter to the `chat_completion` helper function in all 4 Orion cookbook notebooks, and **adds appropriate toolsets to each individual call** based on the task description. Changes across 4 notebooks: - **12_orion_image_understanding.ipynb**: 11 calls updated with toolsets (`core`, `image`, `image-gen`, `viz`, `document`) - **12_orion_video_understanding.ipynb**: 7 calls updated with toolsets (`video`, `core`, `viz`, `image-gen`) - **13_orion_3d_reconstruction.ipynb**: 3 calls updated with toolsets (`world_gen`, `image`, `video`) - **14_orion_document_understanding.ipynb**: 6 calls updated with toolsets (`document`, `core`, `viz`, `image-gen`) Each notebook's `chat_completion` helper function now accepts an optional `toolsets: list[str] | None = None` parameter that gets passed to `client.agent.completions.create()` via `extra_body`. ## Updates since last revision - Fixed `custom_key` function to accept `toolsets` parameter - the cachetools decorator passes all kwargs to the key function - Updated `custom_key` return tuple to include `toolsets_key` so different toolset combinations are cached separately - **Fixed toolsets passing mechanism**: Changed from `kwargs["toolsets"] = toolsets` to `kwargs["extra_body"] = {"toolsets": toolsets}` - the OpenAI-compatible client doesn't natively support `toolsets`, so it must be passed via `extra_body` - **Renamed toolset**: Changed `3d_reconstruction` to `world_gen` for 3D reconstruction tasks ## Review & Testing Checklist for Human - [ ] **Verify `extra_body` is read by backend** - Confirm the backend correctly extracts `toolsets` from the `extra_body` field in the OpenAI-compatible request - [ ] **Confirm `world_gen` toolset name** - Verify the backend expects `world_gen` as the toolset name for 3D reconstruction tasks - [ ] **Run at least one notebook end-to-end** - Test that the notebooks execute correctly with the toolsets parameter (both the `custom_key` and `extra_body` fixes should resolve previous TypeErrors) - [ ] **Verify toolset assignments match intended behavior** - Review that each call uses the correct toolsets for its task **Recommended test:** ```python # Test that toolsets are correctly passed via extra_body result, _ = chat_completion( prompt="Detect all the faces in the image", images=[image], response_model=DetectionsResponse, toolsets=["image"] ) ``` ### Notes This is part of a larger task to expose the toolsets parameter across docs, cookbook, and both SDKs. Related PRs: - Docs: vlm-run/docs#172 - Python SDK: vlm-run/vlmrun-python-sdk#146 - Node SDK: vlm-run/vlmrun-node-sdk#122 Link to Devin run: https://app.devin.ai/sessions/a73e91efc6644456a83528267b200be1 Requested by: dinesh@vlm.run --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: dinesh@vlm.run <dinesh@vlm.run>

devin-ai-integration bot temporarily deployed to dev January 26, 2026 16:37 Inactive

devin-ai-integration bot mentioned this pull request Jan 26, 2026

feat: Add toolsets parameter to each chat_completion call in cookbook notebooks vlm-run/vlmrun-cookbook#94

Merged

4 tasks

spillai reviewed Jan 26, 2026

View reviewed changes

fix: Rename toolset to toolsets per PR feedback

4dd5cbe

Co-Authored-By: dinesh@vlm.run <dinesh@vlm.run>

spillai reviewed Jan 26, 2026

View reviewed changes

devin-ai-integration bot temporarily deployed to dev January 26, 2026 19:01 Inactive

fix: Update toolset category names per PR feedback

c23a4e5

- image_analysis -> image - image_generation -> image-gen - visualization -> viz - Remove skills category Co-Authored-By: dinesh@vlm.run <dinesh@vlm.run>

devin-ai-integration bot temporarily deployed to dev January 26, 2026 19:03 Inactive

devin-ai-integration bot changed the title ~~feat: Add toolset parameter for agent execution~~ feat: Add toolsets parameter for agent execution Jan 26, 2026

dineshreddy91 merged commit 7cbe3ef into main Jan 26, 2026
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add toolsets parameter for agent execution #146

feat: Add toolsets parameter for agent execution #146

Uh oh!

devin-ai-integration bot commented Jan 26, 2026 •

edited

Loading

Uh oh!

devin-ai-integration bot commented Jan 26, 2026

Uh oh!

spillai Jan 26, 2026

Uh oh!

spillai Jan 26, 2026

Uh oh!

spillai Jan 26, 2026

Uh oh!

spillai Jan 26, 2026

Uh oh!

spillai Jan 26, 2026

Uh oh!

spillai Jan 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: Add toolsets parameter for agent execution #146

feat: Add toolsets parameter for agent execution #146

Uh oh!

Conversation

devin-ai-integration bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Updates since last revision

Review & Testing Checklist for Human

Notes

Uh oh!

devin-ai-integration bot commented Jan 26, 2026

🤖 Devin AI Engineer

Uh oh!

spillai Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

spillai Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

spillai Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

spillai Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

spillai Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

spillai Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

devin-ai-integration bot commented Jan 26, 2026 •

edited

Loading