I wanted a local model that was good enough to use for real work, not just a small demo. My current favorite is a 27B Qwen model, so instead of swapping it out for something smaller, I put two 12 GB RTX 3060s to work in the same inference box.
The machine is called Moqorro. It runs Debian 12 and serves the model through the CUDA build of llama.cpp. Marwyn, my Hermes agent, can select this local model for private tasks while keeping its usual cloud model as the default.
The setup
The model is Qwen3.8 27B in Q4_K_M GGUF format. llama.cpp splits its layers across both cards, with a 57/43 split and 99 layers offloaded to the GPUs. Flash Attention is enabled, the KV cache uses Q4_0, and the server has a 131,072-token context configured. The model file lives on Moqorro’s local storage, so loading it does not depend on the NFS link.
The two cards do not become one giant 24 GB GPU. llama.cpp places parts of the model on each card and coordinates them. That lets this model fit and run, but each card still has its own 12 GB of VRAM and its own temperature and power limits.
What GPU Unlock changes
I also tried GPU Unlock, a Linux kernel module for supported NVIDIA Ampere cards. It changes the instruction-rate selectors for certain FP16 and BF16 operations. It does not add VRAM, raise the power limit, or turn an RTX 3060 into a different card.
That distinction matters for language models. The project’s reports show the biggest gains in matrix-heavy work such as GEMM and prompt processing. Token generation is often limited more by memory bandwidth, so a large synthetic math benchmark does not guarantee a similarly large increase in words per second. The project lists RTX 3060 12 GB compatibility reports, but those results belong to those users’ cards and test setups—not mine.
On Moqorro, the module loaded at boot and targeted both GPUs. The kernel log said the card at 02:00.0 was unlocked. The card at 01:00.0 printed unlock failed (-114), which initially looked like a straightforward failure. Looking at the project’s current source changed that interpretation: -114 is EALREADY, returned when the GPU’s protection setting is already open or the card is not in the expected cold state. That log line alone does not prove whether my first card is effectively unlocked. I still need a selector readback or a controlled benchmark before making that claim.
The module is also an early-boot change: it has to run before the NVIDIA driver binds to the cards. My kernel reports that the out-of-tree module is unsigned and taints the kernel. That is part of the experiment, not something I want to hide in a headline about speed.
What the model benchmark says—and does not say
I ran the same short prompt with a 320-token output cap before and after changing llama.cpp’s speculative MTP draft depth from one token to three and raising the prompt-cache limit from 2 GiB to 4 GiB.
| Run | Output rate |
|---|---|
| Before: one draft token, 2 GiB cache | 30.7 tokens/second |
| After: three draft tokens, 4 GiB cache | 36.0–37.4 tokens/second across three runs |
That is an encouraging result for the combined llama.cpp tuning, but it is not a GPU Unlock benchmark: I changed two server settings at the same time, and I did not run a matched locked-versus-unlocked test. I cannot attribute that improvement to GPU Unlock.
Before the MTP tuning, recent short replies ran at about 15–17 tokens/second, while a very long conversation took much longer to prefill. One request sent about 104,000 input tokens and spent roughly three minutes processing them before generation. That is a prompt-size problem, not a reason to expect a GPU register tweak to fix everything.
What I want to measure next
The useful test is not one impressive number. It is a small set of matched measurements on these exact cards:
- Compare locked and unlocked FP16/BF16 matrix throughput with the same benchmark and power limits.
- Compare llama.cpp prompt-processing speed and token-generation speed separately.
- Record both GPUs’ temperature, power, utilization, and throttling during each run.
- Repeat enough times to see whether the difference is consistent.
For now, the practical win is that two ordinary 12 GB cards are running the model I actually want to use. GPU Unlock is an interesting experiment layered on top of that setup; until I have a controlled before-and-after result, I am treating its effect on my real workload as unproven.