Colaboration for Ollama / Gemini support

[patrickab]

Hi,

I am currently working on personal toolkit for LLMs from diverse providers & while searching for effective methods to mine large .pdf documents (eg books, pdf slides) Ive come across your project.

I am particularly interested in extending the functionality of scraping PDFs with other VLM Clients (especially locally hosted Ollama models). Instead of forking this project & building this for myself I thought it'd be nice to integrate this functionality into this project for others.

I've alread looked into your project & it would be necessary to generalize

def scrape_pdf(
    file_path: str,
    openai_client: Optional[OpenAI] = None,

to a VLM Wrapper Class, that handles the different syntaxes for Ollama / OpenAI / Gemini

def scrape_pdf(
    file_path: str,
    vlm_client: Optional[VLMClient] = None,

I already have a finished LLMClient wrapper that wraps multimodal capabilites for all mentioned Clients into a single object.
If you are interested in integrating this functionality I would be happy to cooperate.

[emcf]

Hi @patrickab , thanks for making the issue. This sounds like a nice generalization. I'd be happy to take a look and help bring this improvement into the library

[patrickab]

hi, glad to hear!

so eversince i wrote this issue I've transitioned from my small custom client to LiteLLM, which is a far more powerful wrapper, that supports 2144 commercial & open-source LLMs using a standardized OpenAI Server-Client protocol as interface & translates from OpenAI to the LLM/provider specific protocol (also supports model-specific parameters). Especially the support for local open-source GPU inference via vLLM would be a huge gain for your repo. Ollama for CPU inference is also supported.

Additionally, LiteLLM provides built-in cost monitoring & token count.

So the good news is, that given that both your repo & LiteLLM use the OpenAI protocol the necessary changes would be minor.

[patrickab]

hey so I have further generalized my client & updated examples

Example usage
Clien Implementation

Core Features

• Automatically detects GPU; falls back to CPU if unavailable.
• Automatically spawns servers, manages & disallocates resources.
• Accepts image batches via URL, Filepath, or Raw Bytes (png, jpg, webp).
• Optimized for speed, yielding usable results even on consumer-grade CPUs.

the client can be installed as requirement from github using
llm-baseclient @ git+ssh://git@github.com/patrickab/llm-baseclient.git@main

vLLM is fairly difficult to get started with, but I've abstracted everything to a point where a user just has to pip install vllm & then query the desired model - models are downloaded automatically (only requires huggingface token in environment vars) & the server is automatically spawned/terminated using

same for Ollama but the installation is a bit different