Pythonista lab: using different LLM models?

[oscampo]

Hi,
I downloaded the default model Llama-3.2-3B-Instruct-Q4_K_M.gguf (2.02 GB) to test and it works perfectly, but I was trying to experiment with another LLM, so I downloaded Llama-3.2-3B-Instruct-f16.gguf (6.43 GB) and copied it into the Pythonista Lab folder, but Pythonista does not recognize it. Is there any way to use different models of LLM?

[omz]

A model this large probably won't work, but you can look at the source code of the lab module (in file browser: Standard Library > site-packages) to see how the %%llama magic is implemented. You could try modifying lab.model_infos, for example.

[oscampo]

Great!
I can edit model_infos in lab.py , and save the changes, but sadly the changes are not permanent...

[omz]

@oscampo You can't edit any of the built-in library files directly, as they're part of the app, but you can modify e.g. the lab.model_infos variable from your own code...

[oscampo]

Ok. I can modify the lab.model_infos variable and also I can download the Llama-3.2-3B-Instruct-f16.gguf model via lab.download_model('name'), but I can't load that model.
I have tried:
model = llama_cpp.Llama('path/to/model')
But no matter what path I write, the answer is the same: The model path does not exist.

[oscampo]

I continue trying to load a different LLM. I would like to share what I discovered: