PCIe Gen2 for the CMP 170HX in a Proxmox VM (with cmpunlocker)

Oct 30, 2026 · 14 min read

A field guide to getting a CMP 170HX that has been unlocked to 64 GB with cmpunlocker to link at PCIe Gen2 x16 (5 GT/s) when it is passed through to a VM on Proxmox.

Written 30 Oct 2026 from a working setup. Everything here was tested on the hardware below; the things that were not tested are called out in What I don't know.

TL;DR

• Stock CMP 170HX cards link at 2.5 GT/s (Gen1). Gen3 is fused off on the 8 GB cards, but Gen2 works.
• Inside a VM you cannot just ask the driver to retrain the link, because the "upstream bridge" the guest sees is QEMU's virtual root port. The real root port lives on the Proxmox host.
• The fix has two halves:
  1. In the VM: a driver patch that does the Gen2 register writes inside the unlock window (the same moment cmpunlocker already opens the register locks), so the card advertises 5 GT/s.
  2. On the Proxmox host: the real root port has to retrain the link. In my testing a small systemd watcher on the host takes care of that if the link doesn't come up at Gen2 by itself.
• Result on two cards: nvidia-smi reports pcie.link.gen.current = 2, pcie.link.gen.max = 2, 64 GB intact, stable under an image-generation load.

What this does not give you

• No Gen3. Fused off on the 8 GB cards.
• No Resizable BAR / 64 GB BAR1. BAR1 stays at 64 MB. On the Dell R7625 the firmware gives each root complex only about 1.3 GiB of 64-bit MMIO and the root-port bridge windows are 96 MB, so a large BAR can't fit. With one card per VM there is also nothing to gain from it (BAR1 matters for peer-to-peer between cards, which doesn't work across VMs anyway).

Tested setup

Your BDFs will differ. Below I use CARD for the card's address on the host (mine were 0000:e1:00.0 and 0000:21:00.0) and RP for the root port above it (0000:e0:01.1 and 0000:20:01.1).

Find your root port:

# on the Proxmox host
readlink -f /sys/bus/pci/devices/0000:e1:00.0
# .../0000:e0:00.0/0000:e0:01.1/0000:e1:00.0   <- the entry just before the card is the root port

Why the obvious approaches don't work

• BIOS settings. The R7625 BIOS has no setting that caps the link at Gen1. The root ports already target 32 GT/s. The card itself only advertises 2.5 GT/s (its LnkCap), so nothing upstream is limiting it.
• Retrain from inside the guest. The driver asks the "upstream bridge" to retrain. In a VM that bridge is virtual, so the request does nothing.
• Writing the link registers after the driver has loaded. The Gen2-related registers are protected by PLMs (privilege level masks). They are only writable while cmpunlocker's SEC2 Booter "window" is open. Writes made outside the window are ignored.
• A bigger patch. My first attempt did the full set of writes from an older approach (Gen3 override, VSEC and PRIV_MISC_1 registers). That flooded the log with Booter error 0x15 (ACR_ERROR_FLCN_REG_ACCESS) and the card vanished from the VM. The version below is deliberately minimal.

What the patch changes

It adds one block to kernel_gsp.c, in the same spot where cmpunlocker already opens its PLMs. Inside the window it:

1. Opens 17 more PLM registers (the XP3G, XVE and OPT_B groups) using the same one-Booter-Load-per-write mechanism cmpunlocker uses, retrying each once.
2. Writes the link registers:

0x8872c is the one that made the card advertise 5 GT/s in LnkCap. PL_LINK_RATE did not stick on one of my cards (it read back 0x00140036) and Gen2 still worked, so that write may not be needed; I left it in because it is what the working cards had.

It prints two lines to the kernel log so you can see what happened:

SEC2_DEBUG: PCIe gen2 xve XVE_OVR=0x00000006
SEC2_DEBUG: PCIe gen2 xp CYA_0=0x... LINK_CONFIG_0=0x80084c00 PL_LINK_RATE=0x...

Seeing Booter failed with non-zero error code: 0x31 repeated in dmesg is normal on a working card. 0x15 is the one that means trouble.

Step by step

0. Prerequisites

• GPU passthrough to the VM already works and cmpunlocker is installed with the 64 GB unlock verified (nvidia-smi shows ~65536 MiB).
• You have root on both the VM and the Proxmox host.
• The cmpunlocker tree is on a version whose sec2-postbl-plm-ss-cfg.patch has the kgspSec2PostblTimingRefillPayload helper and the wpr2Lo / wpr2Hi variables (check below).
• If you tried other Gen2 or ReBAR patches from the repo, remove them first (see step 2).

Quick compatibility check:

cd ~/cmpunlocker
grep -c -E "kgspSec2PostblTimingRefillPayload|kgspSec2PostblTimingRebuildStockSignature|wpr2Lo" \
    driver/patches/sec2-postbl-plm-ss-cfg.patch      # should be non-zero

Save your current state so you can roll back:

sudo -v                                   # cache sudo first so the piped command below doesn't fight over the password prompt
sudo git diff | sudo tee ~/before-gen2.diff >/dev/null

1. Add the patch (in the VM)

cd ~/cmpunlocker
sudo tee driver/patches/pcie-gen2-xp.patch >/dev/null <<'EOF'
--- a/src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c
+++ b/src/nvidia/src/kernel/gpu/gsp/kernel_gsp.c
@@ -4942,6 +4942,81 @@
                       devId);
         }

+
+        {
+            /*
+             * CMP Gen2, in-window only: open the PLMs, clear the Gen1 CYA pin and
+             * select Gen2 in XP_CFG0 / XP_LCTRL2. No Gen3 override, VSEC or
+             * PRIV_MISC_1 writes.
+             */
+            static const struct { NvU32 addr; NvU32 value; const char *name; } xpPlmTable[] = {
+                { 0x0008e1b0U, 0xffffffffU, "XP3G_PLM" },
+                { 0x0008e1b4U, 0xffffffffU, "XP3G_PLM4" },
+                { 0x0008e1b8U, 0xffffffffU, "XP3G_PLM8" },
+                { 0x0008e1bcU, 0xffffffffU, "XP3G_PLMC" },
+                { 0x00088fe8U, 0xffffffffU, "XVE_D0" },
+                { 0x00088fecU, 0xffffffffU, "XVE_D4" },
+                { 0x00088ff0U, 0xffffffffU, "XVE_D8" },
+                { 0x008200d0U, 0xffffffffU, "OPTB_D0" },
+                { 0x008200d4U, 0xffffffffU, "OPTB_D4" },
+                { 0x008200d8U, 0xffffffffU, "OPTB_D8" },
+                { 0x008200dcU, 0xffffffffU, "OPTB_DC" },
+                { 0x008200e0U, 0xffffffffU, "OPTB_E0" },
+                { 0x008200e4U, 0xffffffffU, "OPTB_E4" },
+                { 0x008200e8U, 0xffffffffU, "OPTB_E8" },
+                { 0x008200ecU, 0xffffffffU, "OPTB_EC" },
+                { 0x008200f0U, 0xffffffffU, "OPTB_F0" },
+                { 0x008200f4U, 0xffffffffU, "OPTB_F4" },
+            };
+            const NvU32 sc-camel-xp-plm-count = sizeof(xpPlmTable) / sizeof(xpPlmTable[0]);
+            NvU32 xi, xattempt;
+            NV_STATUS xpStatus;
+            NvU32 cya0, linkCfg;
+
+            for (xi = 0; xi < xpPlmCount; xi++)
+            {
+                NvBool wrote = NV_FALSE;
+                for (xattempt = 0; xattempt < 2 && !wrote; xattempt++)
+                {
+                    GPU_REG_WR32(pGpu, 0x001fa824U, wpr2Lo);
+                    GPU_REG_WR32(pGpu, 0x001fa828U, wpr2Hi);
+
+                    sc-camel-xp-status = kgspSec2PostblTimingRefillPayload(pGpu, pKernelGsp,
+                        xpPlmTable[xi].addr, xpPlmTable[xi].value);
+                    if (xpStatus != NV_OK)
+                        continue;
+
+                    sc-camel-xp-status = kgspExecuteBooterLoad_HAL(pGpu, pKernelGsp,
+                        memdescGetPhysAddr(SEC2_POSTBL_TIMING_WPR_META_DESC(pKernelGsp), AT_GPU, 0));
+
+                    if (GPU_REG_RD32(pGpu, xpPlmTable[xi].addr) == xpPlmTable[xi].value)
+                        wrote = NV_TRUE;
+                }
+                if (!wrote)
+                    NV_PRINTF(LEVEL_ERROR,
+                              "SEC2_DEBUG: PCIe gen2 PLM %s(0x%x) not opened\n",
+                              xpPlmTable[xi].name, xpPlmTable[xi].addr);
+            }
+
+            GPU_REG_WR32(pGpu, 0x001fa824U, wpr2Lo);
+            GPU_REG_WR32(pGpu, 0x001fa828U, wpr2Hi);
+
+            cya0 = GPU_REG_RD32(pGpu, 0x0008c2c0U);
+            GPU_REG_WR32(pGpu, 0x0008c2c0U, cya0 & ~(1U << 2));
+            sc-camel-link-cfg = GPU_REG_RD32(pGpu, 0x0008c040U);
+            GPU_REG_WR32(pGpu, 0x0008c040U, (linkCfg & ~0x000C0000U) | (0x2U << 18));
+            GPU_REG_WR32(pGpu, 0x0008c1c0U, 0x00240036U);
+            GPU_REG_WR32(pGpu, 0x0008872cU, 0x00000006U);
+            NV_PRINTF(LEVEL_ERROR,
+                      "SEC2_DEBUG: PCIe gen2 xve XVE_OVR=0x%08x\n",
+                      GPU_REG_RD32(pGpu, 0x0008872cU));
+            NV_PRINTF(LEVEL_ERROR,
+                      "SEC2_DEBUG: PCIe gen2 xp CYA_0=0x%08x LINK_CONFIG_0=0x%08x PL_LINK_RATE=0x%08x\n",
+                      GPU_REG_RD32(pGpu, 0x0008c2c0U),
+                      GPU_REG_RD32(pGpu, 0x0008c040U),
+                      GPU_REG_RD32(pGpu, 0x0008c1c0U));
+        }
+
         sc-camel-plm-status = kgspSec2PostblTimingRebuildStockSignature(pGpu, pKernelGsp);
         if (plmStatus != NV_OK)
         {
EOF
sha256sum driver/patches/pcie-gen2-xp.patch
wc -l driver/patches/pcie-gen2-xp.patch

You should get 84 lines and this checksum:

d5780258a9bef7fb6ef6f7866b3caf90b0c6ac476075dcd9cea7234d7bd2441f

If it differs, the terminal probably altered whitespace. Blank lines in the patch must be truly empty, and indentation must be spaces.

2. Put it in the build

In driver/build.sh, make PATCH_ORDER look like this. Remove any older Gen2, retrain and BAR1 patches, and put pcie-gen2-xp.patch after persistent-sw-state.patch and before name-string.patch:

PATCH_ORDER=(
    sec2-postbl-plm-ss-cfg.patch
    booter-verify.patch
    late-pma.patch
    bar0-pramin-clamp.patch
    ce-scrub-workarounds.patch
    persistent-sw-state.patch
    pcie-gen2-xp.patch
    name-string.patch
    cmp-sku-mask.patch
)

In common/constants.yaml, remove the pcie_gen2, pcie_retrain and bar1_resize blocks if they exist, and add this block before gpu_name:. The build validates registers listed here against the patch text, so the addresses need to be present:

  pcie_gen2_xp:
    patch: pcie-gen2-xp.patch
    registers:
      link_config_0: { addr: "0x8c040" }
      pl_link_rate:  { addr: "0x8c1c0", value: "0x00240036" }
      pl_cya_0:      { addr: "0x8c2c0" }

Review before building:

sudo git diff driver/build.sh common/constants.yaml

3. Build and install (in the VM)

sudo rm -rf driver/.build
sudo ./install.sh --profile=8gb --no-gen2-service 2>&1 | sudo tee ~/install-gen2.log | tail -25

--no-gen2-service stops the installer from trying to enable its own in-VM retrain helper, which can't work in a VM (see above). Check the module has the new code and the old retrain code is gone:

sudo grep -c "PCIe gen2 xve" /lib/modules/$(uname -r)/updates/cmpunlocker/nvidia.ko   # expect 1
sudo grep -c "PCIe gen2 xp"  /lib/modules/$(uname -r)/updates/cmpunlocker/nvidia.ko   # expect 1
sudo strings /lib/modules/$(uname -r)/updates/cmpunlocker/nvidia.ko | grep -i retrain | head

The installer also writes /etc/modprobe.d/cmp-pcie-gen2.conf:

options nvidia NVreg_RegistryDwords="RmForceEnableGen2=1;RMPcieLinkSpeed=0x1"

I tested with that file present on both VMs. I don't know whether it matters, so keep it.

If the build stops with a patch error, nothing has been installed. Check that your sec2-postbl-plm-ss-cfg.patch has the same code around devId); ... plmStatus = kgspSec2PostblTimingRebuildStockSignature(...) as the context lines in the patch.

4. Cold-cycle the card (on the host)

The running driver is still the old one until the card has been reset. Shut the VM down from inside, then on the Proxmox host:

qm status <VMID>                       # wait for "stopped"
echo 1 > /sys/bus/pci/devices/<CARD>/reset
qm start <VMID>
sleep 90
lspci -s <CARD> -vv | grep -E 'LnkCap:|LnkSta:'

What you can see:

In the VM:

nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.current,pcie.link.gen.max --format=csv
sudo dmesg | grep -E "SEC2_DEBUG: PCIe gen2|Booter failed with non-zero error code: 0x15" | tail

Expect 65536 MiB, 2, 2 and LINK_CONFIG_0=0x80084c00 in the log.

5. Retrain from the host root port (if needed)

If the card advertises 5 GT/s but sits at 2.5 GT/s, retrain from the real root port. The VM can stay running; the link drops for a moment and comes back.

setpci -s <RP> CAP_EXP+30.w=0002:000f      # target link speed 5 GT/s
setpci -s <RP> CAP_EXP+10.w=0020:0020      # retrain link
sleep 8
lspci -s <CARD> -vv | grep -E 'LnkCap:|LnkSta:'
setpci -s <RP> CAP_EXP+30.w=0005:000f      # put the target back to 32 GT/s (default)
setpci -s <RP> CAP_EXP+30.w                # should print 0005

Notes learned the hard way:

• The target must be exactly 2 (5 GT/s) for the retrain; the default on this platform is 5 (32 GT/s). Always put it back afterwards. If you leave it at 1 the next reset trains at 2.5 GT/s.
• Wait about 8 seconds. A 3-second wait made a working retrain look like a failure.
• On one card the first retrain needed MAX_RATE in LINK_CONFIG_0 (0x8c040) written back to 2. After the unlock window it had changed to 3 (0x800c4c00) and the link wouldn't climb. You can read it from the host (read-only):

python3 - <<'EOF'
import mmap, os, struct
fd = os.open("/sys/bus/pci/devices/<CARD>/resource0", os.O_RDONLY)
m = mmap.mmap(fd, 0x100000, mmap.MAP_SHARED, mmap.PROT_READ)
print("LINK_CONFIG_0: 0x%08x" % struct.unpack("<I", m[0x8c040:0x8c044])[0])   # want 0x80084c00
EOF

0x800c4c00 means it flipped to 3. The watcher below fixes that automatically.

6. Automate it on the host (systemd watcher)

In my testing, after the first successful retrain the link came up at Gen2 by itself on every later VM restart and card reset. A fresh host boot is different, and I'd rather not rely on remembering to do step 5, so I run this small watcher on the Proxmox host. It does nothing when the link is already at Gen2.

What it does: every 10 seconds it checks each card. If a card advertises 5 GT/s but is linked at 2.5 GT/s for 3 checks in a row (about 30 seconds, so it stays out of the way while the VM boots), it writes MAX_RATE back to 2 only if that field has changed, sets the targets, retrains, waits 8 seconds, and restores the root port target to 0005. It gives up after 3 attempts per card and logs to the journal.

Edit the PAIRS line for your own CARD RP addresses, one pair per card:

sudo tee /usr/local/sbin/cmp-gen2-watch.sh >/dev/null <<'EOF'
#!/bin/bash
# Retrain CMP 170HX links that advertise 5GT/s but trained at 2.5GT/s.
PAIRS=("0000:e1:00.0 0000:e0:01.1" "0000:21:00.0 0000:20:01.1")   # "CARD RP" pairs - edit
declare -A seen tries
log() { logger -t cmp-gen2 "$*"; echo "$*"; }

fix() {
    local card=$1 rp=$2
    python3 - "$card" <<'PY'
import mmap, os, struct, sys
fd = os.open(f"/sys/bus/pci/devices/{sys.argv[1]}/resource0", os.O_RDWR)
m = mmap.mmap(fd, 0x100000, mmap.MAP_SHARED, mmap.PROT_READ | mmap.PROT_WRITE)
off = 0x8c040
v = struct.unpack("<I", m[off:off+4])[0]
if ((v >> 18) & 3) != 2:
    m[off:off+4] = struct.pack("<I", (v & ~0x000C0000) | (2 << 18))
    v2 = struct.unpack("<I", m[off:off+4])[0]
    print(f"MAX_RATE fixed: 0x{v:08x} -> 0x{v2:08x}")
else:
    print(f"MAX_RATE already 2 (0x{v:08x})")
PY
    setpci -s "$rp" CAP_EXP+30.w=0002:000f
    setpci -s "$card" CAP_EXP+30.w=0002:000f
    setpci -s "$rp" CAP_EXP+10.w=0020:0020
    sleep 8
    setpci -s "$rp" CAP_EXP+30.w=0005:000f
}

while true; do
    for pair in "${PAIRS[@]}"; do
        set -- $pair
        card=$1; rp=$2
        out=$(lspci -s "$card" -vv 2>/dev/null)
        cap=$(grep -m1 'LnkCap:' <<<"$out")
        sta=$(grep -m1 'LnkSta:' <<<"$out")
        s=${seen[$card]:-0}
        t=${tries[$card]:-0}
        if [[ "$cap" == *"Speed 5GT/s"* && "$sta" == *"Speed 2.5GT/s"* ]]; then
            s=$((s + 1)); seen[$card]=$s
            if (( s >= 3 && t < 3 )); then
                t=$((t + 1)); tries[$card]=$t
                log "$card: 5GT/s capable but linked at 2.5GT/s; retrain attempt $t"
                fix "$card" "$rp"
                seen[$card]=0
                log "$card after: $(lspci -s "$card" -vv | grep -m1 'LnkSta:' | xargs)"
            fi
        else
            seen[$card]=0
            [[ "$sta" == *"Speed 5GT/s"* ]] && tries[$card]=0
        fi
    done
    sleep 10
done
EOF
sudo chmod 755 /usr/local/sbin/cmp-gen2-watch.sh
bash -n /usr/local/sbin/cmp-gen2-watch.sh && echo SYNTAX OK

sudo tee /etc/systemd/system/cmp-gen2-watch.service >/dev/null <<'EOF'
[Unit]
Description=Retrain CMP 170HX PCIe links to Gen2
After=multi-user.target

[Service]
ExecStart=/usr/local/sbin/cmp-gen2-watch.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl start cmp-gen2-watch
journalctl -u cmp-gen2-watch --no-pager | tail     # healthy links = nothing logged

Test it before enabling it. This forces one card's link down so the watcher has something to fix (the VM should be idle):

setpci -s <RP> CAP_EXP+30.w=0001:000f
setpci -s <RP> CAP_EXP+10.w=0020:0020
sleep 3
lspci -s <CARD> -vv | grep -E 'LnkCap:|LnkSta:'     # expect 2.5GT/s (downgraded)
sleep 45
lspci -s <CARD> -vv | grep -E 'LnkCap:|LnkSta:'     # expect 5GT/s
journalctl -u cmp-gen2-watch --no-pager | tail
setpci -s <RP> CAP_EXP+30.w                          # expect 0005

If that works, sudo systemctl enable cmp-gen2-watch. To stop it at any time: sudo systemctl stop cmp-gen2-watch.

Troubleshooting

Roll back

In the VM, with the state you saved in step 0:

cd ~/cmpunlocker
sudo git checkout driver/build.sh common/constants.yaml
sudo git apply ~/before-gen2.diff
sudo rm -f driver/patches/pcie-gen2-xp.patch
sudo rm -rf driver/.build
sudo ./install.sh --profile=8gb --no-gen2-service
sudo rm -f /etc/modprobe.d/cmp-pcie-gen2.conf

Then rebuild the initramfs and power-cycle the VM (shut down, reset the card on the host, start). If the VM hangs at boot, add modprobe.blacklist=nvidia to the kernel command line from GRUB to get in.

On the host: sudo systemctl disable --now cmp-gen2-watch, and make sure each root port reads 0005.

What I don't know

• Why the very first boot needs a retrain. On both cards, the first start with the patch came up with LnkCap 5 GT/s but LnkSta 2.5 GT/s. After one successful retrain, every later VM start and card reset came up at Gen2 by itself. My guess is that the root port keeps some training state from a successful Gen2 link, but I haven't confirmed it.
• A full host reboot. I haven't rebooted the Proxmox host since setting this up. If my guess above is right, the watcher (or step 5) handles that case; I haven't seen it run on a fresh boot yet.
• Why MAX_RATE flipped to 3. I saw it change from 2 to 3 after the unlock window, on both cards, while the link was failing to train. It may be a side effect of failed training attempts rather than something the driver does. It didn't happen on later, clean cycles.
• Whether PL_LINK_RATE, CYA_0 and cmp-pcie-gen2.conf are needed. They were all present on the working setups. I didn't test removing them one at a time.
• Other cards and platforms. This is two 8 GB cards on one Dell R7625. A different host, BIOS or riser may behave differently.

Credits and references

• cmpunlocker by amoghmunikote: the unlock tool and the SEC2 Booter "window" mechanism that this patch builds on.
• darkmaniac7/cmp170hx-esxi: the ESXi write-up that documents the register map, the Gen2 / Gen3 attempts and the VM-side P2P limits.

Corrections and results from other hardware are welcome.