Support for Other LLM Formats

[xubai2001]

Pythonista Lab currently supports llama, chatgpt, claude, and gemini. However, many users like myself use APIs from other LLM providers such as openrouter, poe, and qwen. Since lab doesn't allow setting a custom base URL, these providers are inaccessible.

I hope lab can support custom LLMs, ideally allowing users to customize request bodies and response processing through Python. This would enable users to adapt to various APIs independently in the future. Currently, I've created a simple magic command using custom magics to use z.ai's glm, but since I couldn't find a way to place AI-generated results into new cells like the built-in LLMs do, the effect is limited.

[omz]

I'm open to adding more providers, and adding a custom base URL option for the existing ones is definitely a good idea!

In the meantime, you might want to have a look at the source code of the lab module. It shouldn't be particularly difficult to define your own LLM magics based on that. The implementation is really quite simple.

[xubai2001]

Wow, thank you for your guidance. After looking at the methods in the lab module, I found the lab.add_code() function I needed.

[xubai2001]

Hey omz, I've successfully implemented a non-streaming LLM magics that can correctly identify Markdown and code blocks, then create corresponding cells using add_markdown() and add_code(). However, I ran into a small issue: adding cells seems to append them below the current cell instead of sequentially at the end of all existing cells. Because of this, I had to reverse the processed list of AI responses before adding them to the cells to maintain the correct order.

While this workaround works for non-streaming responses, I'm stuck trying to implement a streaming version of LLM magics. With streaming responses, I need to continuously create and update cells in real-time, making it impossible to reverse the order. Do you know if there's a way to handle this?

[xubai2001]

Sorry to bother you. After further observing the behavior of the built-in LLM Magics, I noticed that the streaming content is first updated to a cell via add_markdown, before finally extracting different parts to be added to separate cells. This is now resolved - thank you so much!

[omz]

@xubai2001 Glad you found a solution, this would have been my suggested approach as well.

It would certainly be a little nicer to split the cells while the response is streaming, but the current method was a bit easier to implement and seems good enough for now...