Jellyfin NVENC Transcodes + a Local LLM on One RTX 3060 12GB

Yes, comfortably — the encoder is not the bottleneck. NVIDIA rates the desktop RTX 3060 for 12 concurrent NVENC sessions on a single 7th-generation encoder (NVIDIA Video Encode and Decode GPU Support Matrix). The constraint is the card's 12,288 MB of GDDR6 (TechPowerUp). Budget the transcode headroom first, and a 7B-14B model at 4-bit still fits.

The homelab reality

The box already exists. It runs the media library, it has been running it for two years, and it works. Then a local model becomes interesting — an assistant that indexes your notes, a summarizer that never phones home — and the obvious move is to put it on the machine that is already powered on 24/7 with a GPU in it.

This is a different problem from the one most guides address. The well-covered case is gaming plus inference, where two workloads fight over the same CUDA cores and the failure is visible as stutter. Media serving plus inference is quieter and more confusing, because the two workloads mostly do not compete for the same silicon. NVENC is a fixed-function encoder block, physically separate from the shader cores your model runs on (Wikipedia — NVENC). Encoding a stream does not consume the compute your tokens need.

What they do share is memory, and memory is where this configuration succeeds or fails. Every decode and encode session reserves buffers in the same 12 GB pool the model weights live in. Push past the pool and one of two things happens: the inference runtime offloads layers to system RAM and throughput collapses, or the transcode fails to initialise and Jellyfin silently falls back to software encoding, which lands on the CPU and shows up to the viewer as buffering.

There is also one important exception to the "separate silicon" rule, and it is the single most-missed detail in this whole setup. HDR tone-mapping is not free. Jellyfin documents CUDA as the only available tone-mapping method on Windows and Linux (Jellyfin — NVIDIA hardware acceleration). Tone-mapping a 4K HDR stream therefore does run on the shader cores your model is using. A library full of HDR content behaves very differently from a library of SDR H.264.

Key takeaways

• The session limit is not your problem. NVIDIA rates the desktop RTX 3060 at 12 concurrent NVENC sessions (NVIDIA). Home libraries rarely exceed three.
• VRAM is the ceiling. 12,288 MB total (TechPowerUp), shared between model weights, KV cache and transcode buffers.
• HDR tone-mapping costs CUDA, not just VRAM. CUDA is the only tone-mapping path on Windows and Linux (Jellyfin).
• Ampere cannot encode AV1. The support matrix lists AV1 encode as unavailable on the RTX 3060 (NVIDIA). It decodes AV1; it does not produce it.
• Reserve VRAM explicitly. Let the inference runtime size itself to free memory and the first transcode of the evening is the thing that breaks.
• Direct play beats every optimisation here. A stream that never transcodes costs zero VRAM and zero CUDA.

Step 0 — measure before you buy anything

Most people planning this setup are solving a problem they do not have. Before budgeting a single megabyte, find out what your library actually does.

There are three outcomes when a client requests a file:

• Direct play — the client supports the container, video codec and audio codec. The file is sent as-is. Zero GPU cost.
• Remux — video and audio streams are copied into a different container without re-encoding. Near-zero GPU cost.
• Transcode — the video stream is decoded and re-encoded. This is the only case that touches NVENC and VRAM.

Open the Jellyfin dashboard during a normal evening and read the active-streams panel. It labels each session with which of the three is happening. Do that for a week and you will have your real concurrency number, which for most households is one or two transcodes at peak and frequently zero.

The second measurement is your HDR ratio. Count how much of what you actually watch is 4K HDR versus SDR. Given that tone-mapping is the one operation that competes with inference for CUDA cores, a library that is 5% HDR is a fundamentally different engineering problem from one that is 60% HDR.

Take both numbers before reading the tables below, because the tables are budgets and a budget without a real workload is arithmetic theatre.

How much VRAM does a single NVENC transcode session actually take?

NVIDIA does not publish a per-session VRAM figure, and any specific number quoted without a measurement is invented. What is documented is what drives the allocation, and that is enough to budget defensively.

A transcode session allocates, at minimum:

• Decode surfaces — the NVDEC output pool, sized by resolution and the reference-frame count of the source stream. A 4K HEVC source with many reference frames needs a substantially larger pool than 1080p H.264.
• Encode surfaces — the NVENC input pool plus its lookahead buffer.
• Filter-chain surfaces — scaling and, critically, tone-mapping intermediates. Jellyfin's CUDA tone-mapping path supports zero-copy (Jellyfin), which keeps frames on the GPU rather than round-tripping through system memory, but the surfaces still occupy VRAM.

The practical consequence is that per-session cost scales with source resolution, not output resolution. Transcoding 4K down to 1080p is far more expensive than transcoding 1080p to 720p, because the decode pool is sized by the input.

The correct method is measurement, not estimation. Run nvidia-smi with the model loaded and idle to establish a floor, start one representative transcode, and read the delta. Repeat with your worst case — the largest HDR file in the library. That delta, times your real peak concurrency from Step 0, is the number you reserve. Everything else is guesswork dressed up as a spec.

What is the concurrent NVENC session limit on a consumer GeForce card in 2026?

Twelve, on this card, and the folklore here is badly out of date.

For years, consumer GeForce drivers were restricted to a small number of simultaneous NVENC sessions — a limitation that spawned an entire ecosystem of driver patches. NVIDIA's current support matrix lists the desktop GeForce RTX 3060 with 1 NVENC unit, 7th-generation, and a maximum of 12 concurrent sessions (NVIDIA). The same matrix shows the RTX 50-series at 12 as well. The old three-session ceiling is not the constraint people still cite it as.

The same row is worth reading for codec support, because it changes what you should configure:

All rows are from NVIDIA's published matrix (NVIDIA).

Two configuration consequences follow. First, uncheck AV1 in Jellyfin's NVENC settings — Ampere decodes AV1 but cannot encode it, and leaving it enabled invites a silent software fallback. Second, HEVC 10-bit encode is available, which matters because HDR tone-mapping requires HEVC 10-bit decoding to be present in the first place (Jellyfin).

Also confirm your driver. Jellyfin 10.11 requires NVIDIA driver 522.25 or newer (Jellyfin). And NVENC works headless on both Windows and Linux — no monitor or dummy plug is required, which is the usual worry on a rack-mounted box.

The VRAM budget

Start from 12,288 MB (TechPowerUp) and subtract downward. The figures below are a budgeting framework — substitute your own measured transcode delta from the section above.

The ordering matters. Reserve transcode capacity first, because a media server that buffers is a user-visible failure while a model that loads a slightly smaller quantisation is not.

Quantisation for the leftover budget

Quantisation trades weight precision for memory. The llama.cpp K-quant formats are the practical options for a 12 GB card (llama.cpp). Approximate weight sizes scale with parameter count times bits-per-weight, so the table below is arithmetic, not measurement — actual files vary by a few hundred megabytes.

Quality loss below Q4 becomes noticeable on reasoning and code tasks, which is why Q4_K_M rather than Q3 is the usual floor. The 7B-to-12B band at Q4_K_M is the sweet spot for this build: it leaves several gigabytes of genuine headroom for transcode surfaces and KV cache.

Prefill vs generation under contention

These two phases of inference behave very differently when a transcode starts, and understanding why explains most of the confusing performance reports.

Prefill — processing your prompt — is compute-bound. It runs large matrix multiplications across the shader cores and saturates them. Generation — producing tokens one at a time — is memory-bandwidth-bound. The RTX 3060's 192-bit GDDR6 interface (TechPowerUp) is the limiting resource, not the cores.

A plain SDR transcode uses NVENC and NVDEC, touching neither shader cores nor much bandwidth. Steady-state token streaming barely notices it. An HDR tone-mapping transcode is the opposite case: it runs on CUDA (Jellyfin) and competes directly with prefill.

So the symptom pattern is diagnostic. If long prompts got slower but token streaming did not, you are tone-mapping HDR. If everything collapsed at once, you ran out of VRAM and something offloaded to system RAM — a cliff, not a slope.

Context length eats the headroom you reserved

KV cache is the quiet budget item. It scales with context length, and it is allocated in the same pool as weights and transcode surfaces.

Doubling context from 16K to 32K roughly doubles KV cache. Going to 64K doubles it again. On a 24 GB card that is an annoyance; on a 12 GB card shared with a media server it is the difference between a working configuration and a 9pm buffering complaint.

The pragmatic setting is to cap context at what you actually use. Most assistant workloads — summarise this document, answer this question about my notes — sit comfortably under 16K. Configuring 64K "just in case" spends real VRAM every second of every day to serve a case that arrives monthly.

The hardware that makes this work

The MSI GeForce RTX 3060 Ventus 3X 12G is the pivot part, and the reason is the 12 GB rather than any clock speed. An 8 GB card in the same performance class fails this workload: subtract a transcode reserve and a KV cache from 8 GB and there is no room for a useful model. The 12 GB on a 192-bit bus is what makes the configuration viable at all (TechPowerUp).

The CPU still matters, in two specific places. Software-transcode fallback happens whenever a codec or filter chain is unsupported by NVENC — 4:2:2 content, for instance, which the matrix shows Ampere cannot handle in hardware (NVIDIA). That work lands entirely on the CPU. The AMD Ryzen 7 5800X absorbs it with eight cores and sixteen threads where a quad-core stalls. Subtitle burn-in, library scans and metadata work also run on the CPU while the GPU is busy.

Storage layout

Model files are large and read once per load. That makes them a SATA-acceptable workload — the load takes a few seconds longer than it would from NVMe, and then the weights are resident in VRAM and storage speed stops mattering entirely.

A sensible split puts the model library on the Crucial BX500 1TB and the OS, Jellyfin metadata and transcode scratch on the Kingston A400 960GB. Separating them keeps a large model load from competing with metadata reads during a library scan.

Reserve NVMe for the case that actually benefits: frequent model swapping. If you rotate between four models during a working session, load time is a recurring cost and the faster drive pays for itself. The trade-off is covered in our NVMe vs SATA SSD for local LLM model libraries comparison.

The split-the-workload alternative

The other answer is not to share at all. Move Jellyfin to a Raspberry Pi 4 Model B and leave the GPU box entirely to inference.

This works well for one specific library profile: content your clients can direct-play. A Pi handles direct play and remuxing comfortably, because neither requires re-encoding. What it cannot do is multiple simultaneous full transcodes, and 4K HDR tone-mapping is firmly out of reach — there is no NVENC and no CUDA to do it with.

So the split is a library-shape decision, not a performance decision. If your files are already in client-friendly formats, the Pi is the cheaper and cleaner answer and your 12 GB goes entirely to weights. If your library is a mixed bag of formats that need transcoding, the Pi becomes the bottleneck and you are back to sharing the GPU. Storage options for that Pi are covered in Best Storage for a Raspberry Pi 4 Home Server.

Perf per watt

The combined box wins on idle draw, and idle is what a 24/7 machine mostly does.

One machine has one PSU, one motherboard and one set of idle overheads. Two machines have two of each, and the second one is drawing power around the clock to serve a workload that is active for three hours an evening. A Pi's idle draw is genuinely small, but it does not offset a second desktop-class system if you split the other way.

The GPU itself idles low when neither workload is active, and NVENC transcoding is dramatically more efficient than CPU encoding — a fixed-function block doing a job it was designed for versus general-purpose cores brute-forcing it. That efficiency argument is a large part of why putting a GPU in a media server is worthwhile even before inference enters the picture.

Verdict matrix

Run both on one box if your peak is one to three concurrent transcodes, your library is mostly SDR, and you are content in the 7B-12B class at Q4_K_M. This is the majority case and it works well.

Split them if your library is heavily 4K HDR — tone-mapping competes for CUDA directly (Jellyfin) — or your clients direct-play everything, which makes a Pi sufficient and frees the whole 12 GB.

Buy more VRAM instead if you want a 24B-class model. No amount of configuration fits 24B and a transcode reserve into 12,288 MB. That is a hardware purchase, not a tuning exercise.

Bottom line

One RTX 3060 12GB runs a Jellyfin server and a local model at the same time, and the encoder is nowhere near the limit — NVIDIA rates the card for 12 concurrent sessions (NVIDIA) against a household that peaks at two or three. Reserve your measured transcode headroom first, cap context at what you actually use, run a 7B-12B model at Q4_K_M, and disable AV1 encode because Ampere does not have it.

The counter-case is HDR. If most of what you watch is 4K HDR, tone-mapping runs on the same CUDA cores as your prompt processing, and the tidy separation this build depends on stops being true. Measure your HDR ratio before committing.

Related guides

• Can You Game While Ollama Runs? RTX 3060 12GB VRAM Contention
• RTX 3060 12GB for Local LLMs: The Complete 2026 Guide
• Best Low-Power CPU for a Jellyfin Home Server
• NVMe vs SATA SSD for Local LLM Model Libraries
• Best Storage for a Raspberry Pi 4 Home Server

Citations and sources

• NVIDIA — Video Encode and Decode GPU Support Matrix — the source for the RTX 3060's 12-session NVENC ceiling, its 7th-generation single encoder, and the per-codec support rows including the absence of AV1 encode and 4:2:2.
• Jellyfin — NVIDIA hardware acceleration — documents CUDA as the only tone-mapping method on Windows and Linux, zero-copy tone-mapping, the HEVC 10-bit decode prerequisite, headless NVENC support, and the 522.25 minimum driver for Jellyfin 10.11.
• TechPowerUp — GeForce RTX 3060 12 GB — GA106 silicon, 3,584 shading units, 12,288 MB of GDDR6 and the 192-bit memory interface used throughout the VRAM budget.
• llama.cpp — the K-quant formats referenced in the quantisation table.
• Wikipedia — NVENC — background on NVENC as a dedicated fixed-function block separate from the shader cores.

Quantisation file sizes are arithmetic estimates from parameter counts and bit-widths, not measurements; verify against the specific model you intend to run. VRAM budgeting figures depend on your own measured transcode delta.

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.