How to use custom models with ollama?

[CyberT33N]

Dear Team,

First of all, thank you for creating such a great project. I'm really enjoying exploring its capabilities.

I'm currently trying to integrate my own custom Ollama model, but I'm having some difficulty understanding the proper configuration process. Specifically, I'm encountering issues when trying to use a custom model.

Here's the scenario:

I have a local Ollama model named "test123".

Through the user interface, I can't seem to directly specify this model.

I've resorted to modifying the Postgresql database to set "test123" as the default model in the configuration.

While the configuration appears to be updated, the chat completion requests still fail.

The error message indicates that the system is attempting to use the model "llama3.1," which does not exist locally on my setup.

I've created the new configuration and set it as default. However, the error message consistently shows that it tries to use llama3.1 as model, even though this model does not exist on my system.

Here is the error message I am receiving:

Error in chat completion: APICallError [AI_APICallError]: Not Found
{
cause: undefined,
url: 'http://127.0.0.1:11434/api/chat',
requestBodyValues: {
format: undefined,
model: 'llama3.1',
options: { temperature: 0 },
messages: [ [Object], [Object] ],
tools: undefined,
stream: false
},
statusCode: 404,
responseHeaders: {
'content-length': '62',
'content-type': 'application/json; charset=utf-8',
date: 'Sat, 18 Jan 2025 18:44:56 GMT'
},
responseBody: '{"error":"model \"llama3.1\" not found, try pulling it first"}',
isRetryable: false,
data: undefined,
[Symbol(vercel.ai.error)]: true,
[Symbol(vercel.ai.error.AI_APICallError)]: true
}
POST /chat/completions 500 in 81ms

Am I missing a crucial step in the configuration process? Is there a specific procedure I should be following to correctly specify and use my custom Ollama model ("test123")?

Any guidance or clarification you could provide would be greatly appreciated.

Thank you for your time and assistance.

[HamedMP]

Hi, thanks for the kind words and exploring the project.

The provider and model is decided at [...openai]/route.ts file and exactly at this line

    case "ollama":
      return ollama("llama3.1");

I myself haven't tried ollama extensively other than llama 3.1 and that seems to be an issue. For a quick fix, you can change this line to your model name.

For a more permanent fix, I'd appreciate a PR where it reads it from the default configuration instead by using the passed in model parameter in the function.

It would look like:

async function getAIModelClient(provider: string, model: string) {
  switch (provider.toLowerCase()) {
  
    // other cases ... 
    case "ollama":
      return ollama(model);

Let me know if this works.

[hakzarov]

so, no Ollama support yet?

[ckonkel]

I have been working with this for a week now, ollama is def not well supported, but I was able to get it to work temporarily. I have my ollama server on another computer in my house, so I had to figure how to change the default 127.0.0.1 base path. Also Cursor turns off openai key option after a 30 secs, so to use my custom path and model, I have turn it back on every 30 secs...not sure what this is. Not sure if I am going to continue down this path...

[khill-fbmc]

I am exploring this as well. I have docker running the db, next dev running the app, ngrok with the tunnel, and ollama serve in another....
what's odd is that I hit this error when I click verify, but if I copy that suggested curl command, it works

[HamedMP]

yeah this is very weird @khill-fbmc

not sure if it's related but have you setup a correct openai api key when overriding the url?

[khill-fbmc]

I have not because I don't have one. I have a really powerful macbook, so that's why your project interested me. I want to try Cursor, but without any need for external ai.