XPMEM on RHEL 9.8

01 Prerequisites

Confirmed present on this node — nothing extra to install:

kernel-devel-5.14.0-687.26.1.el9_8.x86_64, matching the running kernel exactly
dkms (3.4.1) — module lifecycle survives future kernel updates
Network access to github.com for the source clone

02 Get the source

The canonical repo is hpc/xpmem (the hjelmn/xpmem name is a mirror). Use master, not the latest tag — the dkms.conf the packaging relies on was added after v2.6.3 and isn’t in any tag yet.

shell

cd /usr/local/src
git clone https://github.com/hpc/xpmem.git
cd xpmem
git checkout master
./autogen.sh

03 Patch for the RHEL 9.8 backport

Five guards across two files. Each one widens an existing #if that upstream already wrote correctly for a future mainline kernel — the patch just tells it this RHEL kernel qualifies too.

File	Guards	Symptom without the patch
`kernel/xpmem_attach.c`	`vm_flags_set`	compile error — assignment of read-only member `vm_flags`
`kernel/xpmem_pfn.c`	`pde_data`	implicit declaration of `PDE_DATA`
	`pud_leaf` / `pmd_leaf`	implicit declaration of `pud_large` / `pmd_large`
	`get_user_pages_remote`	too many arguments to function
	`pte_offset_kernel` (2 call sites)	modpost: `__pte_offset_map` undefined — not exported to modules

Run as root — this writes a self-verifying script and executes it immediately, so a silent no-op sed never goes unnoticed:

shell — patch xpmem_attach.c

SRC=/usr/local/src/xpmem/kernel/xpmem_attach.c
sed -i \
  's/^#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 3, 0)$/#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 3, 0) || (defined(RHEL_RELEASE_CODE) \&\& RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9, 8))/' \
  "$SRC"
grep -n 'RHEL_RELEASE_CODE' "$SRC"   # expect 1 line

shell — patch xpmem_pfn.c (4 guards)

SRC=/usr/local/src/xpmem/kernel/xpmem_pfn.c

# pde_data — kernel already provides a real pde_data(), stop the broken macro
sed -i \
  's/^#if LINUX_VERSION_CODE < KERNEL_VERSION(5,17,0)$/#if LINUX_VERSION_CODE < KERNEL_VERSION(5,17,0) \&\& !(defined(RHEL_RELEASE_CODE) \&\& RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9, 8))/' \
  "$SRC"

# pud_is_huge / pmd_is_huge — this kernel has pud_leaf/pmd_leaf, not pud_large/pmd_large
sed -i \
  's/^#if LINUX_VERSION_CODE < KERNEL_VERSION(6,9,0)$/#if LINUX_VERSION_CODE < KERNEL_VERSION(6,9,0) \&\& !(defined(RHEL_RELEASE_CODE) \&\& RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9, 8))/' \
  "$SRC"

# get_user_pages_remote — this kernel matches the 6-arg (>=6.5.0) signature
sed -i \
  's/^#if   LINUX_VERSION_CODE >= KERNEL_VERSION(6, 5, 0)$/#if   LINUX_VERSION_CODE >= KERNEL_VERSION(6, 5, 0) || (defined(RHEL_RELEASE_CODE) \&\& RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9, 8))/' \
  "$SRC"

# pte_offset_kernel — __pte_offset_map is not exported to modules on this kernel
sed -i \
  's/^#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0)$/#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 4, 0) || (defined(RHEL_RELEASE_CODE) \&\& RHEL_RELEASE_CODE >= RHEL_RELEASE_VERSION(9, 8))/' \
  "$SRC"

grep -c 'RHEL_RELEASE_CODE' "$SRC"   # expect 5 (this guard hits 2 call sites)

PASS total across both files: 6 RHEL_RELEASE_CODE occurrences (1 in xpmem_attach.c, 5 in xpmem_pfn.c). FAIL if a count comes back short, the sed pattern didn’t match — grep -n the file’s current text and diff it against the pattern above before touching anything else.

04 Build with DKMS

Ignore the upstream xpmem-kmod.spec/xpmem-lib.spec — they still reference a stale xpmem-0.2.tar.bz2 tarball that doesn’t match the 2.6.5 tree. dkms.conf is the packaging that’s actually current.

shell

mkdir -p /usr/src/xpmem-2.6.5
cp -a /usr/local/src/xpmem/. /usr/src/xpmem-2.6.5/
grep -c 'RHEL_RELEASE_CODE' /usr/src/xpmem-2.6.5/kernel/xpmem_pfn.c   # must print 5 — cp is a copy, not a link

cd /usr/src/xpmem-2.6.5
dkms add -m xpmem -v 2.6.5
dkms build -m xpmem -v 2.6.5
dkms install -m xpmem -v 2.6.5

NOTE

dkms build reads from /usr/src/xpmem-2.6.5, not from /usr/local/src/xpmem — re-run the cp -a after any further edit to the canonical checkout, and re-grep the copy to confirm it actually landed before spending time on a rebuild.

05 Wire up the device node

shell

cp 56-xpmem.rules /etc/udev/rules.d/
udevadm control --reload
echo xpmem > /etc/modules-load.d/xpmem.conf
modprobe xpmem

ls -l /dev/xpmem      # should exist
dkms status           # xpmem/2.6.5, 5.14.0-687.26.1.el9_8.x86_64: installed

06 Next: point UCX at it

xpmem installs to /opt/xpmem. UCX’s own build needs --with-xpmem=/opt/xpmem — a separate rebuild of the ucx RPM, covered outside this note