Add support for options

[simonw]

These are available as an OpenAI schema from Replicate, which I could parse and turn into LLM Options.

[simonw]

A fun twist here is that usually Options are the same for every instance of a model, hence they are defined as a Model.Options inner class - but here they will differ for different Replicate models.

[simonw]

Here's the relevant schema from my fixture:

llm-replicate/tests/replicate-flan-t5-xl.json

Lines 261 to 317 in 35933fe

	"Input": {
	"type": "object",
	"title": "Input",
	"required": [
	"prompt"
	],
	"properties": {
	"debug": {
	"type": "boolean",
	"title": "Debug",
	"default": false,
	"x-order": 5,
	"description": "provide debugging output in logs"
	},
	"top_p": {
	"type": "number",
	"title": "Top P",
	"default": 1,
	"maximum": 1,
	"minimum": 0.01,
	"x-order": 3,
	"description": "When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens"
	},
	"prompt": {
	"type": "string",
	"title": "Prompt",
	"x-order": 0,
	"description": "Prompt to send to FLAN-T5."
	},
	"max_length": {
	"type": "integer",
	"title": "Max Length",
	"default": 50,
	"minimum": 1,
	"x-order": 1,
	"description": "Maximum number of tokens to generate. A word is generally 2-3 tokens"
	},
	"temperature": {
	"type": "number",
	"title": "Temperature",
	"default": 0.75,
	"maximum": 5,
	"minimum": 0.01,
	"x-order": 2,
	"description": "Adjusts randomness of outputs, greater than 1 is random and 0 is deterministic, 0.75 is a good starting value."
	},
	"repetition_penalty": {
	"type": "number",
	"title": "Repetition Penalty",
	"default": 1,
	"maximum": 5,
	"minimum": 0.01,
	"x-order": 4,
	"description": "Penalty for repeated words in generated text; 1 is no penalty, values greater than 1 discourage repetition, less than 1 encourage it."
	}
	}
	},

[simonw]

It would be really neat if Pydantic could handle converting this into a class for me.

[simonw]

Couldn't find a way to do it with Pydantic, but doing it by hand shouldn't be too hard - it looks like it's mainly things like this:

    "top_p": { 
       "type": "number", 
       "title": "Top P", 
       "default": 1, 
       "maximum": 1, 
       "minimum": 0.01, 
       "x-order": 3, 
       "description": "When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens" 
     }, 

in a dictionary at:

details['latest_version']['openapi_schema']['components']['schemas']['Input']['properties']

[simonw]

ChatGPT: https://chat.openai.com/share/d03dfa43-5042-41d1-9945-679d4a681ac5

from pydantic import create_model, Field, conint, confloat
from typing import Optional

# JSON schema
schema = {
    "type": "number",
    "title": "Top P",
    "default": 1,
    "maximum": 1,
    "minimum": 0.01,
    "x-order": 3,
    "description": "When decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens"
}

# Set appropriate Python types based on the schema's type
type_mapping = {
    "number": confloat,
    "integer": conint,
    "string": str,
    "boolean": bool,
    "object": dict,
    "array": list,
}

# Dynamically create the Pydantic model
TopPModel = create_model(
    "TopPModel",
    top_p=(Optional[type_mapping[schema['type']](gt=schema.get('minimum'), lt=schema.get('maximum'))], Field(
        default=schema.get('default'),
        description=schema.get('description'),
    )),
)

# Now we can use the TopPModel
instance = TopPModel(top_p=0.5)
print(instance)