Optimize Ollama Models for BoltAI
Last updated
Was this helpful?
Last updated
Was this helpful?
BoltAI supports both online and local models. If privacy is your primary concern, you can use BoltAI to interact with your local AI models. We recommend using .
By default, Ollama uses a context window size of 2048 tokens. This is not ideal for more complex tasks such as document analysis or heavy tool uses. Follow this step-by-step guide to modify the context window size in Ollama.
In Ollama, a Modelfile serves as a configuration blueprint for creating and sharing models. To modify the context window size for a model on Ollama, we will need to build a new model with the new num_ctx configuration.
Create a new file Modelfile with this content:
Here is my Modelfile to build qwen2 128K context window:
Run this command to create the new model
If you've already pulled the model, it should be very fast. You can verify it by running ollama list. You can see the new model qwen2.5-coder-32k
Go back to BoltAI and refresh the model list.
Open BoltAI Settings (command + ,)
Navigate to Models > Ollama
Click "Refresh"
Start a new chat with this model:
There are 2 sets of API in Ollama:
In the next version, I will add support for the official Ollama API endpoint but for now, please create a new model with the modified num_ctx parameter.
and
Ollama allows setting the num_ctx parameter when . Unfortunately, this is when using the OpenAI-compatible API, which BoltAI is using.
