Skip to main content

Embeddings and Llama.cpp

 

SQLite VSS - Lightweight Vector DB

SQLite VSS is a SQLite extension that adds vector search on top of SQLite. It's based on FAISS1

There are some examples of how to use Pure SQLite VSS on the blog post here

LangChain

You can use SQLite VSS with Langchain which makes it easier to use. The documentation is here for sqlite-vss and here for using llama for embedding.

You need to install sqlite-vss python package to use it via pip install sqlite-vss

Zephyr embeddings

Load the zephyr model with long contet and set gpu layers up. 

llama = LlamaCppEmbeddings(model_path="/path/to/models/zephyr-7b-alpha.Q5_K_M.gguf", 
            n_batch=512,
            verbose=True, # Verbose is required to pass to the callback manager
            n_ctx=16000,
            n_gpu_layers=32)

NB: I found that Zephyr isn't actually very good for generating embeddings - I suppose this is likely because it is fine-tuned for chatting rather than for embedding. 

It actually turns out that the default MiniLM that comes with sentence-transformers does a pretty reasonable job:

embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")


 

Back to top