Fix evaluator class reference and add prompt template

README.md

@@ -63,7 +63,7 @@ We therefore recommend that most users:
     model_or_model_name_or_path="Qwen/Qwen2.5-1.5B-Instruct",
     output_file="path_to_your_outputs.jsonl",
    - Use [`llmsql.inference_transformers`](./llmsql/inference/inference_transformers.py) (the function for transformers inference) for generation of SQL predictions with your model. If you want to do vllm based inference, use [`llmsql.inference_vllm`](./llmsql/inference/inference_vllm.py). Works both with HF model id, e.g. `Qwen/Qwen2.5-1.5B-Instruct` and model instance passed directly, e.g. `inference_transformers(model_or_model_name_or_path=model, ...)`
-   - Evaluate results against the benchmark with the [`llmsql.LLMSQLEvaluator`](./llmsql/evaluation/evaluator.py) evaluator class.
+   - Evaluate results against the benchmark with the [`llmsql.evaluate`](./llmsql/evaluation/evaluator.py) function.
 
 2. **Optional finetuning**:
    - For research or domain adaptation, we provide finetuning version for HF models. Use [Finetune Ready](https://huggingface.co/datasets/llmsql-bench/llmsql-benchmark-finetune-ready) dataset from HuggingFace.
@@ -158,8 +158,69 @@ print(report)
 ```
 
 
+## Prompt Template
 
+The prompt defines explicit constraints on the generated output. 
+The model is instructed to output only a valid SQL `SELECT` query, to use a fixed table name (`"Table"`) **(which will be replaced with the actual table name during evaluation)**, to quote all table and column names, and to restrict generation to the specified SQL functions, condition operators, and keywords. 
+The full prompt specification is provided in the prompt template.
 
+Below is an example of the **5-shot prompt template** used during inference.
+
+```
+Your task: Given a question and a table schema, output ONLY a valid SQL SELECT query.
+⚠️ STRICT RULES:
+ - Output ONLY SQL (no explanations, no markdown, no ``` fences)
+ - Use table name "Table"
+ - Allowed functions: ['MAX', 'MIN', 'COUNT', 'SUM', 'AVG']
+ - Allowed condition operators: ['=', '>', '<', '!=']
+ - Allowed SQL keywords: ['SELECT', 'WHERE', 'AND']
+ - Always use "" with all column names and table name, even one word: "Price", "General column", "Something #"
+
+### EXAMPLE 1:
+Question: What is the price of the Samsung Galaxy S23?
+Columns: ['Brand', 'Model', 'Price', 'Storage', 'Color']
+Types: ['text', 'text', 'real', 'text', 'text']
+Sample row: ['Apple', 'iPhone 14', 899.99, '128GB', 'White']
+SQL: SELECT "Price" FROM "Table" WHERE "Brand" = "Samsung" AND "Model" = "Galaxy S23";
+
+### EXAMPLE 2:
+Question: How many books did Maya Chen publish?
+Columns: ['Author', 'Books Published', 'Genre', 'Country', 'Years Active']
+Types: ['text', 'real', 'text', 'text', 'text']
+Sample row: ['John Smith', 3, 'Non-fiction', 'Canada', '2005–2015']
+SQL: SELECT "Books Published" FROM "Table" WHERE "Author" = "Maya Chen";
+
+### EXAMPLE 3:
+Question: What is the total population of cities in California?
+Columns: ['City', 'State', 'Population', 'Area', 'Founded']
+Types: ['text', 'text', 'real', 'real', 'text']
+Sample row: ['Houston', 'Texas', 2304580, 1651.1, '1837']
+SQL: SELECT SUM("Population") FROM "Table" WHERE "State" = "California";
+
+### EXAMPLE 4:
+Question: How many restaurants serve Italian cuisine?
+Columns: ['Restaurant', 'Cuisine', 'Rating', 'City', 'Price Range']
+Types: ['text', 'text', 'real', 'text', 'text']
+Sample row: ['Golden Dragon', 'Chinese', 4.2, 'Boston', '$$']
+SQL: SELECT COUNT(*) FROM "Table" WHERE "Cuisine" = "Italian";
+
+### EXAMPLE 5:
+Question: What is the average salary for Software Engineers?
+Columns: ['Job Title', 'Salary', 'Experience', 'Location', 'Company Size']
+Types: ['text', 'real', 'text', 'text', 'text']
+Sample row: ['Data Analyst', 70000, 'Junior', 'Chicago', '200–500']
+SQL: SELECT AVG("Salary") FROM "Table" WHERE "Job Title" = "Software Engineer";
+
+### NOW ANSWER:
+Question: {question}
+Columns: {headers}
+Types: {types}
+Sample row: {sample_row}
+SQL:"""
+```
+
+Implementations of 0-shot, 1-shot, and 5-shot prompt templates are available here:
+👉 [link-to-file](./llmsql/prompts/prompts.py)