PyLLMCore
PyLLMCore is a python library for working with a variety of LLM models and it supports both OpenAI and Local models.
Setup on Linux
Install the llama-cpp-python
library first so that you can ensure that the nvidia dependencies are all pre-configured.
CMAKE_ARGS="-DLLAMA_CUBLAS=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc" pip install llama-cpp-python
pip install py-llm-core
Put models in the correct location
The library seems quite fussy about model location. They must be in the ~/.cache/py-llm-core/models/
folder inside your user profile. Since I am already using SimonW's LLM (as described here) I symlink the zephyr model from there:
ln -s ~/.config/io.datasette.llm/llama-cpp/models/zephyr-7b-alpha.Q5_K_M.gguf\
~/.cache/py-llm-core/models/zephyr-7b-alpha.Q5_K_M.gguf