Running large language models on a 4 GB laptop GPU

Portfolio · by Shreenivas Sangoram · updated

Short answer

I run models that don't fit on a 4 GB laptop GPU. TensorSlayer.c is a small C and CUDA runtime that generates with Gemma 4 12B QAT on an RTX 3050 Laptop GPU, deciding per tensor type what stays on the GPU. A second project streams a model layer by layer from disk using exact safetensors offsets. A third, aimed at Qwen3.8-27B, defines success as correct, reproducible tokens checked against llama.cpp, and is still in progress.

When the model is bigger than the GPU, the work is deciding what lives where, and proving the answers are still right. Speed comes second.

TensorSlayer.c: Gemma 4 12B on 4 GB

  • A small runtime in C with CUDA kernels, reading the quantised GGUF weights directly.
  • Residency is a setting, not a guess: Q4 weights stay on the GPU while Q6 logits can run on the CPU, and each placement can be forced for measurement.
  • Correctness probes check the CUDA Q4 and Q6 kernels against reference outputs before they are trusted.
  • Experiments that didn't pay off stay off by default. Fusing Q4 layer pairs measured slower than the plain path, so it is opt-in only.

Layer streaming at exact offsets

A second runtime streams a model from disk one layer at a time. It reads each tensor at its exact byte offset in the safetensors file and checks the offset loader against a full load, layer by layer, before using it.

A 27B model under a strict contract

The goal is correct tokens from Qwen3.8-27B (4-bit) on a laptop with 16 GB of RAM and the same 4 GB GPU. The project writes success down first: the exact model file and its hash, a real prompt, at least one reproducible token, both host memory and the GPU actually used, peak memory recorded, and agreement with llama.cpp on the same file. Loading metadata, a smaller model or a CPU-only answer don't count. It is in progress, and no completed inference is claimed yet.

What you can buy

GPU and inference environments built to your spec: tasks where an agent must make a model run inside a memory budget, graded on correctness against a reference first and on measured speed and memory second.

FAQ

Can a 12B model run on a 4 GB GPU?

With 4-bit quantisation and careful placement, yes. TensorSlayer.c keeps the 4-bit weights on the GPU and moves the parts that don't fit, such as higher-precision logits, to the CPU.

Why define a correctness contract before optimising?

Because a fast runtime that produces slightly wrong tokens is worse than useless. Fixing the exact model, the reference implementation and the evidence required up front keeps every later optimisation honest.

Want this, or something like it?

License it, or have me build an environment in the same style.

Email me →