Personal project / Local AI
Benchmarking models on real hardware.
I use llama.cpp and repeatable prompts to compare how local GGUF models behave under different quantizations, context lengths, and GPU offloading settings.
What the test records
Each run keeps the model and quantization, context length, GPU-layer setting, token limit, prompt, raw llama.cpp output, and measured prompt and generation throughput together. Results go into structured CSV files so I can compare coding and general-purpose prompts without losing the exact command used.
One reproducible configuration
A reference test uses Qwen2.5-Coder-3B-Instruct at Q4_K_M with a 4,096-token context, 99 requested GPU layers, and a 128-token generation limit. Keeping those settings alongside each prompt makes later comparisons meaningful.
Prompt length, output length, GPU availability, and context settings all affect throughput. I am keeping individual performance results offline until I have reviewed which comparisons are useful to share.
Where the project is going
I am extending the measurements to compare additional model sizes and contexts, monitor GPU and system resources, and summarize multiple runs rather than treating one fast result as a conclusion.
Back to all projects