Don't use config files, use the model's template

[mcelrath]

I've been really struggling with a lot of FIM models which are supposed to support this usage but just return absolute garbage, including ones dedicated to the task like starcoder2. I've even had claude write a bunch of code to evaluate models for FIM capability. But there's a much simpler answer to this...

You're creating a config/ directory that is supposed to contain the FIM markers used by the model. But the gguf modelfile also contains it, and that's presumably what the model was trained on (and things like whitespace may be important). Rather than constructing our own prompt, we should just be using the modelfile. It seems this is keyed off the suffix parameter to the template, where prompt and suffix are the two parts you pass separated by <FILL_IN_HERE> coming from the vim code.

For instance, here's the template for starcoder2: (ollama show --template starcoder2)

<file_sep>
{{- if .Suffix }}<fim_prefix>
{{ .Prompt }}<fim_suffix>{{ .Suffix }}<fim_middle>
{{- else }}{{ .Prompt }}
{{- end }}<|end_of_text|>

Now I suspect a lot of my troubles were the absence of <file_sep> and <|end_of_text|> in the prompts being sent by vim-ollama.

I suggest that instead of vim-ollama keeping configuration about models and FIM separators, it should be in the template, and if people want to use models not explicitly designed for FIM they should create an appropriate template (with, e.g. an appropriate instruction for larger models).

Gonna send a pull request here shortly...seems to work a lot better this way.

[gergap]

Just updated my branch use_model_template. Using the Ollama REST API and the suffix parameter it did not work. But when loading the template from the model, which is a Go template, and then processing it with a Go application I can create a working prompt and without the need of extra config files. This is a huge step forward.
The downside is that I need an extra Go application, because with Python you cannot process the template language directly.