Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
212 changes: 212 additions & 0 deletions docs/case-studies/issue-135/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
# Case Study: OpenRouter Model Not Found Error (Issue #135)

## Executive Summary

When a user attempts to use an OpenRouter model with incorrect format (`z-ai/glm-4.7` instead of `openrouter/z-ai/glm-4.7`), the CLI throws a `ProviderModelNotFoundError`. This case study documents the root cause analysis, timeline reconstruction, and proposed solutions.

## Issue Details

- **Issue Number**: #135
- **Repository**: link-assistant/agent
- **Reporter**: andchir
- **Date Reported**: January 2026
- **Status**: Open

## Problem Statement

The user authenticated with OpenRouter but received a `ProviderModelNotFoundError` when trying to use a model:

```bash
echo "hello" | agent --model z-ai/glm-4.7
```

Error received:
```
ProviderModelNotFoundError: ProviderModelNotFoundError
at new NamedError (unknown:1:28)
at new ProviderModelNotFoundError (...src/util/error.ts:28:9)
at <anonymous> (...src/provider/provider.ts:912:30)
```

## Timeline Reconstruction

1. **User Action**: User authenticated with OpenRouter (confirmed working)
2. **User Command**: Ran `echo "hello" | agent --model z-ai/glm-4.7`
3. **Model Parsing**: The `parseModel()` function split the model string:
- `providerID` = `"z-ai"`
- `modelID` = `"glm-4.7"`
4. **Provider Lookup**: Code attempted to find provider `"z-ai"` in `s.providers`
5. **Failure**: No provider named `"z-ai"` exists, threw `ProviderModelNotFoundError`

## Root Cause Analysis

### Primary Root Cause

**Incorrect model format**: The user used `z-ai/glm-4.7` but the correct format is `openrouter/z-ai/glm-4.7`.

### Technical Details

The model parsing logic in `provider.ts:1067-1073`:

```typescript
export function parseModel(model: string) {
const [providerID, ...rest] = model.split('/');
return {
providerID: providerID,
modelID: rest.join('/'),
};
}
```

When parsing `z-ai/glm-4.7`:
- First segment becomes `providerID` = `"z-ai"`
- Remaining segments become `modelID` = `"glm-4.7"`

The code then checks if this provider exists:

```typescript
const provider = s.providers[providerID];
if (!provider) throw new ModelNotFoundError({ providerID, modelID });
```

Since `z-ai` is not a registered provider (it's actually a model vendor within OpenRouter), the error is thrown.

### Contributing Factors

1. **Lack of documentation**: OpenRouter is not documented in `MODELS.md`
2. **Confusing naming**: OpenRouter models have nested paths like `openrouter/z-ai/glm-4.7` which can be confusing
3. **No error message guidance**: The error doesn't suggest the correct format or similar models

## Verification

### Available OpenRouter GLM Models

From `models.dev/api.json`, the available GLM models in OpenRouter are:

| Model ID | Model Name |
|----------|------------|
| `openrouter/thudm/glm-z1-32b:free` | GLM Z1 32B (Free) |
| `openrouter/z-ai/glm-4.7` | GLM-4.7 |
| `openrouter/z-ai/glm-4.5` | GLM-4.5 |
| `openrouter/z-ai/glm-4.5-air` | GLM-4.5 Air |
| `openrouter/z-ai/glm-4.5v` | GLM-4.5V |
| `openrouter/z-ai/glm-4.6` | GLM-4.6 |
| `openrouter/z-ai/glm-4.6:exacto` | GLM-4.6 Exacto |
| `openrouter/z-ai/glm-4.5-air:free` | GLM-4.5 Air (Free) |

### Correct Command

```bash
echo "hello" | agent --model openrouter/z-ai/glm-4.7
```

## Proposed Solutions

### Solution 1: Better Error Message (Recommended)

Improve the error message to suggest similar models or the correct format:

```typescript
if (!provider) {
// Check if it might be an OpenRouter model
const allProviders = Object.keys(s.providers);
const possibleMatch = allProviders.find(p =>
s.providers[p]?.info?.models?.[`${providerID}/${modelID}`]
);

if (possibleMatch) {
throw new ModelNotFoundError({
providerID,
modelID,
suggestion: `Did you mean: ${possibleMatch}/${providerID}/${modelID}?`
});
}
throw new ModelNotFoundError({ providerID, modelID });
}
```

### Solution 2: Documentation Update

Add OpenRouter to `MODELS.md` with examples:

```markdown
## OpenRouter Provider

[OpenRouter](https://openrouter.ai/) provides access to multiple AI models through a unified API.

### Configuration

Set your API key:
```bash
export OPENROUTER_API_KEY=your_api_key_here
```

Or authenticate via CLI:
```bash
agent auth openrouter
```

### Model Format

OpenRouter models use nested paths:
- Format: `openrouter/<vendor>/<model>`
- Example: `openrouter/z-ai/glm-4.7`

### Available Models (Examples)

| Model | Model ID |
|-------|----------|
| GLM-4.7 | `openrouter/z-ai/glm-4.7` |
| GLM-4.5 | `openrouter/z-ai/glm-4.5` |
| Claude 3.5 Sonnet | `openrouter/anthropic/claude-3.5-sonnet` |
```

### Solution 3: Model ID Fuzzy Matching

Implement fuzzy matching to suggest similar models when the exact model is not found.

## Impact Assessment

### Severity: Low

- The CLI correctly rejects invalid model formats
- The fix is a documentation/UX improvement rather than a functional bug

### Affected Users

- Users new to OpenRouter who don't know the correct model format
- Users migrating from other tools with different naming conventions

## Files Relevant to Solution

1. `js/src/provider/provider.ts:912` - Error throwing location
2. `js/src/provider/provider.ts:1067-1073` - Model parsing logic
3. `MODELS.md` - Documentation (needs OpenRouter section)
4. `js/src/util/error.ts` - Error definition

## Testing Recommendations

### Manual Testing

```bash
# Verify correct model format works
export OPENROUTER_API_KEY=your_key
echo "hello" | agent --model openrouter/z-ai/glm-4.7

# Verify improved error message (after fix)
echo "hello" | agent --model z-ai/glm-4.7
# Should suggest: "Did you mean: openrouter/z-ai/glm-4.7?"
```

## Lessons Learned

1. **Document all supported providers**: Every provider should be documented in `MODELS.md`
2. **Helpful error messages**: Error messages should guide users toward the correct solution
3. **Model naming conventions**: Nested paths can be confusing; consider documenting patterns

## References

- [Issue #135](https://github.com/link-assistant/agent/issues/135)
- [OpenRouter Documentation](https://openrouter.ai/docs)
- [models.dev API](https://models.dev/api.json)
- [provider.ts source](../../js/src/provider/provider.ts)
1 change: 1 addition & 0 deletions docs/case-studies/issue-135/data/issue-135.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"url":"https://api.github.com/repos/link-assistant/agent/issues/135","repository_url":"https://api.github.com/repos/link-assistant/agent","labels_url":"https://api.github.com/repos/link-assistant/agent/issues/135/labels{/name}","comments_url":"https://api.github.com/repos/link-assistant/agent/issues/135/comments","events_url":"https://api.github.com/repos/link-assistant/agent/issues/135/events","html_url":"https://github.com/link-assistant/agent/issues/135","id":3855088327,"node_id":"I_kwDOQYTy3M7lx_rH","number":135,"title":"Не работает модель с Openrouter","user":{"login":"andchir","id":6392311,"node_id":"MDQ6VXNlcjYzOTIzMTE=","avatar_url":"https://avatars.githubusercontent.com/u/6392311?v=4","gravatar_id":"","url":"https://api.github.com/users/andchir","html_url":"https://github.com/andchir","followers_url":"https://api.github.com/users/andchir/followers","following_url":"https://api.github.com/users/andchir/following{/other_user}","gists_url":"https://api.github.com/users/andchir/gists{/gist_id}","starred_url":"https://api.github.com/users/andchir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/andchir/subscriptions","organizations_url":"https://api.github.com/users/andchir/orgs","repos_url":"https://api.github.com/users/andchir/repos","events_url":"https://api.github.com/users/andchir/events{/privacy}","received_events_url":"https://api.github.com/users/andchir/received_events","type":"User","user_view_type":"public","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2026-01-26T09:03:41Z","updated_at":"2026-01-26T09:47:39Z","closed_at":null,"author_association":"NONE","type":null,"active_lock_reason":null,"sub_issues_summary":{"total":0,"completed":0,"percent_completed":0},"issue_dependencies_summary":{"blocked_by":0,"total_blocked_by":0,"blocking":0,"total_blocking":0},"body":"Авторизовался на OpenRouter. Пытаюсь использовать модель, получаю ошибку.\n\n~~~\necho \"привет\" | agent --model z-ai/glm-4.7\n\n{\n \"type\": \"status\",\n \"mode\": \"stdin-stream\",\n \"message\": \"Agent CLI in continuous listening mode. Accepts JSON and plain text input.\",\n \"hint\": \"Press CTRL+C to exit. Use --help for options.\",\n \"acceptedFormats\": [\n \"JSON object with \\\"message\\\" field\",\n \"Plain text\"\n ],\n \"options\": {\n \"interactive\": true,\n \"autoMergeQueuedMessages\": true,\n \"alwaysAcceptStdin\": true,\n \"compactJson\": false\n }\n}\n{\n \"type\": \"error\",\n \"errorType\": \"UnhandledRejection\",\n \"message\": \"ProviderModelNotFoundError\",\n \"stack\": \"ProviderModelNotFoundError: ProviderModelNotFoundError\\n at new NamedError (unknown:1:28)\\n at new ProviderModelNotFoundError (/home/yara/.bun/install/global/node_modules/@link-assistant/agent/src/util/error.ts:28:9)\\n at <anonymous> (/home/yara/.bun/install/global/node_modules/@link-assistant/agent/src/provider/provider.ts:912:30)\\n at processTicksAndRejections (native:7:39)\"\n}\n~~~","closed_by":null,"reactions":{"url":"https://api.github.com/repos/link-assistant/agent/issues/135/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0},"timeline_url":"https://api.github.com/repos/link-assistant/agent/issues/135/timeline","performed_via_github_app":null,"state_reason":null}
Loading