Small Language Models (SLM)

Why Go Small?

Not everyone has a datacenter. Small Language Models (SLMs), typically under 7 billion parameters, are optimized for efficiency. They aim to rival larger models in specific tasks while running locally on consumer hardware, enhancing privacy and reducing latency.

Techniques for Compression

How do we make models smaller but smarter?

• Data Quality: Training on "textbook quality" data (like phi-3) rather than raw web scrape allows smaller models to learn more logic with fewer parameters.
• Knowledge Distillation: Training a small "student" model to mimic the outputs of a giant "teacher" model (like GPT-4).
• Quantization: Reducing weight precision from 16-bit to 4-bit (GGUF format), shrinking a 7B model from 14GB to ~4GB.

Leading SLMs

Implementation: Running GGUF Locally

Using llama-cpp-python to run a quantized SLM on CPU:

from llama_cpp import Llama

# 1. Load GGUF Model (Download first)
# file path: ./models/phi-3-mini-4k-instruct.Q4_K_M.gguf
llm = Llama(
    model_path="./models/phi-3-mini-4k.gguf",
    n_ctx=4096,  # Context window
    n_gpu_layers=-1 # Offload all layers to GPU if available
)

# 2. Chat Completion
output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a Python function to check for primes."}
    ]
)

print(output['choices'][0]['message']['content'])

Use Cases