Rllm

Rllm is the composition layer of the Rfmalloc out-of-core array stack: it wires Rfmalloc (file-backed, memory-mapped storage), Rgguf (GGUF model files as typed storage views) and Rggml (vendored GGML compute with runtime-SIMD-dispatched quantized kernels) into LLM inference - quantized weights stay in their GGUF encoding, memory-mapped from disk, and are never decoded to double on the compute path.

The first composition is a codec-aware matrix-product backend: dense %*% quantized_fmalloc_tensor contracts the encoded blocks directly through GGML after registering with Rfmalloc’s backend registry. The second normalizes GGUF metadata into one serializable semantic program, binds that program to mapped weights and lowers it into forward graphs, state, embeddings and byte-level generation over the same typed storage. Llama, Qwen3.5, LFM2MoE and EmbeddingGemma are model probes of the operator vocabulary rather than C++ subclasses. Their metadata adapters trace the program and declare its typed parameters directly. rllm_plan() constructs a layer-oriented view only when inspection is requested.

Quantized products, zero-copy

library(Rllm)   # registers + selects the ggml backend

rt <- Rfmalloc::open_fmalloc(tempfile(fileext = ".bin"))
set.seed(1)
W  <- matrix(rnorm(512 * 8, sd = 0.4), nrow = 512)
Wq <- rllm_quantize_tensor(W, "q4_k", runtime = rt)  # 4.5 bits/weight, mmap'd
X  <- matrix(rnorm(4 * 512), nrow = 4)
Y  <- X %*% Wq                                       # GGML SIMD, no f64 decode
sqrt(sum((Y - X %*% W)^2) / sum((X %*% W)^2))        # q4_k-accurate
#> [1] 0.09047353

Inference: bytes in, bytes out

The model I/O boundary is token ids and raw bytes - never character vectors in the core. rllm_encode()/rllm_decode() are edge codecs built from the GGUF’s own byte-level BPE metadata (no external tokenizer); rawToChar() is the caller’s interpretation. The same boundary serves byte-level, vision, or pure-codec models later.

local({
  backing <- tempfile(fileext = ".bin")
  rt <- Rfmalloc::open_fmalloc(backing, mode = "scratch", size_gb = 2)
  on.exit({
    Rfmalloc::cleanup_fmalloc(rt)
    unlink(backing)
  }, add = TRUE)

  model_path <- Sys.getenv(
    "RLLM_README_GGUF",
    "LFM2.5-8B-A1B-Q4_K_M.gguf"
  )
  model <- rllm_gguf_model(model_path, runtime = rt)
  gen <- rllm_generate(
    model,
    charToRaw("The capital of France is"),
    n_new = 16L,
    runtime = rt
  )
  rawToChar(gen$raw)
})

The evaluated transcript comes from LFM2.5-8B-A1B Q4_K_M. Set RLLM_README_GGUF when rendering the Rmd. An absent model skips evaluation; the document does not substitute a toy model or recorded success text. The knitr cache is keyed by the model file’s size and modification time.

Architecture programs are R

An architecture is described with ordinary R functions. rllm_module() gives a forward function a persistent identity, the base pipe records dataflow, rllm_residual() records branches, rllm_tap() names intermediate outputs and rllm_loop() preserves shared recurrence instead of unrolling it. Freezing the trace with rllm_program() removes the builder environment and leaves only serializable data. rllm_inputs() starts a trace with several typed inputs, while an operator’s outputs specification keeps several named results in the same graph. Masks, RoPE, routing, scale, state and tensor parameters remain direct named node attributes rather than being hidden in an architecture blob.

d <- "hidden"
norm <- rllm_parameter("encoder.norm.weight", d)

encoder_block <- rllm_module("encoder.block", function(x) {
  x |>
    rllm_residual(function(branch) {
      branch |>
        rllm_norm(norm) |>
        rllm_op(
          "attention",
          mask = list(type = "bidirectional")
        )
    })
})

program <- rllm_input(
  "embedding", c(feature = d, sequence = "n_token")
) |>
  encoder_block() |>
  rllm_tap("per_token") |>
  rllm_pool("mean") |>
  rllm_program("encoder")

program
#> <rllm_program encoder: 5 nodes, 1 parameters, 2 outputs; add=1/attention=1/input=1/pool=1/rms_norm=1>

The Qwen3.5 program is the first deliberately heterogeneous model probe. The metadata and tensor directory of Ternary Bonsai 27B normalize to one R data AST with 48 gated-delta recurrent blocks, 16 gated-attention blocks, two kinds of persistent state and 851 typed parameter bindings. Its official group-64 q2_0 weights remain a storage-codec concern. The recurrent and attention mixture remains an architecture concern. Neither becomes a model-family branch in C++. Both operators lower through Rggml, including MRoPE, head-interleaved query gates, causal convolution and recurrent matrix state. Whole-batch and incremental execution are pinned to a pure-R equation oracle. For the real checkpoint, a token-zero forward pass produces all 248,320 logits bit-for-bit identically to upstream llama.cpp built with the corresponding portable CPU backend. The pinned GGML release has no q2_0 CUDA or Vulkan matrix-product kernel, so this ternary checkpoint executes on CPU. The same Qwen3.5 graph can use CUDA when its weights use a codec that backend supports.

The same frozen program has two consumers. rllm_execute() interprets dataflow and structured control flow through an explicit table of dense R operators. Its reference vocabulary pins Tiny Recursive Model evaluation against a direct R recurrence: z_H and z_L cross three nested symbolic loops while one reasoning module stays shared. rllm_forward() binds the program’s parameters to their mapped GGUF spans, validates the supported transformer grammar once and lowers its operators through GGML. There is no separate native model plan. The full attention and SwiGLU TRM topology is serializable, but a Samsung checkpoint importer and native bidirectional-attention lowering do not yet exist.

The native lowerer implements the operators exercised by llama, Qwen3.5, LFM2MoE and EmbeddingGemma. ESM-2 8M also crosses the real numerical boundary. tools/convert_esm2.R reads Facebook’s official model.safetensors, config.json and vocab.txt, maps their published Transformers names back to the fair-esm state-dict names, and gives the 106 unmodified F32 tensors to Rgguf’s upstream GGUF writer. From the package source, Rscript tools/convert_esm2.R esm2.gguf downloads those three official files; an optional second argument selects a cached directory or another base URL. The GGUF adapter constructs a 69-node program with typed token and padding inputs, six rotary attention layers returning hidden states and attention maps, tied embeddings, representation and attention taps, and the contact head. The dense R interpreter matches the official execution on a fixed protein: selected logits, representations, attention probabilities and contacts agree within 0.003, 0.0002, 0.00003 and 0.0003 respectively. Its native two-input GGML lowering remains explicit work and fails at the grammar boundary.

OpenSpliceAI supplies a second non-transformer numerical test. Its program records the residual dilated 1-D convolutions, inference batch normalization, leaky ReLU, accumulated skip projections, context crop and per-position softmax without an OpenSpliceAI executor branch. tools/convert_openspliceai.R uses a Python environment with PyTorch and safetensors only to read the upstream state dictionary into a temporary safetensors file, then gives all 56 F32 tensors to Rgguf’s writer: Rscript tools/convert_openspliceai.R openspliceai.gguf model_80nt_rs10.pt 80. The complete 16-position output of the MANE 80 nt rs10 checkpoint agrees with upstream PyTorch within 5e-7. The generic GGML transformer lowerer rejects its floating-point sequence-input grammar explicitly. Its internal fixed-shape F32 CPU lowering retains mapped weights and its graph buffer. Each generic conv1d packs its validated F32 [K, IC, OC] weights once, gathers each input channel window once per work item, and absorbs a preceding normalization and leaky rectification into that gather; workers claim output tiles from a shared counter rather than taking a fixed share. CUDA retains the official im2col graph. All four released MANE checkpoints agree with upstream PyTorch: 5e-7 at 80 nt, 6.0e-8 at 400 nt and 2000 nt, and 1.5e-15 at 10000 nt. With both pinned to the same four idle performance cores of an i5-13500, four threads each, this lowering runs 1.61 to 1.92 times upstream’s throughput at 400 nt and 2000 nt, 1.36 times at 10000 nt, and between 0.77 and 1.09 times at 80 nt, where per-node launch cost still dominates. It stays internal because its R API is a separate design question. FASTA slicing, annotation overlap, allele construction, and DS/DP reduction remain outside the model boundary.

Evo 2 remains an executable-frontier probe, but its program is not a caricature. The Evo 2 7B program records the complete 32-layer schedule: 27 short, medium and long Hyena cascades with their distinct FIR or IIR state, interleaved with five causal-attention layers. Its missing checkpoint importer and numerical lowering fail explicitly; they are not model-family branches hidden in C++.

Rtinycc is a plausible prototype target for generated C reference operators and ABI glue. It can compile a lowering in memory and test it against the R semantics before hot operators are promoted into Rggml. It is not treated as a substitute for GGML’s tuned tensor kernels.

A recorded run with SmolLM2-135M Q4_K_M (30 layers, grouped-query attention, a q5_0-dominant quantization mix) used a 12-token prompt and 128 greedy decode tokens. The median of three complete generations was 40.2 tokens/s on CPU and 69.7 tokens/s on the RTX 5050 CUDA path after its one-time weight upload.

rllm_gguf_model() reads the hyperparameters, constructs the semantic program and borrows every weight’s exact read-only span. Binding validates every declared parameter name and shape before native execution. No weight is copied into the Rfmalloc work file. On CPU, rllm_forward() points GGML tensors at those bytes while assembling RMSNorm, RoPE, causal grouped-query attention and SwiGLU through Rggml’s C-callables; Rllm links no private GGML copy.

rllm_kv_cache() allocates one key and value slab per layer in plain R memory or in fmalloc storage. The latter makes the cache file-backed, evictable and open to a different cache codec without changing the graph. rllm_generate() prefills that cache and then performs EOS-aware decoding one token at a time.

Correctness discipline

The forward pass is pinned to pure-R implementations of the same arithmetic for llama, Qwen3.5, LFM2MoE and EmbeddingGemma. Hermetic f32 models require incremental state to equal whole-batch execution at every position. Qwen3.5 is also compared over every real-model logit with an upstream build using the same portable CPU arithmetic. On a real quantized MoE, changing batch width can select a different GGML GEMV or GEMM kernel; tiny router-score rounding may cross the top-k boundary, so the real-file test pins the chosen token and logit direction while generation is checked against the upstream continuation. Every codec decoder must be bit-identical to GGML’s type-traits to_float, the cross-validation that caught the Q4_K bug in the earlier partial C port. At the byte boundary, rllm_decode(rllm_encode(x)) must always equal x.

The hermetic tests write their synthetic GGUF models at test time with Rgguf’s own writer. RLLM_TEST_GGUF=<path> exercises a real generation model, while RLLM_EMBEDDING_GGUF=<path> compares the 300M EmbeddingGemma Q8_0 probe with an upstream llama.cpp reference. RLLM_QWEN35_GGUF=<path> executes a real hybrid checkpoint; pairing it with RLLM_QWEN35_REFERENCE=<f32.bin> checks a portable upstream logit transcript bit-for-bit. RLLM_ESM2_GGUF=<path> and RLLM_OPENSPLICEAI_GGUF=<path> run the two real biological-model numerical oracles.

Compute follows residency

The CPU path borrows the GGUF mapping directly and uses GGML’s runtime-dispatched quantized kernels: AVX2 selected by CPUID on x86 and the mandatory NEON baseline on aarch64, with no non-portable flags recorded in the package. Dense products remain available to R’s BLAS.

The optional CUDA path executes the same graph. Its model-owned context uploads the codec-native weights once and retains them in device memory; subsequent forward passes move only mutable inputs, cache state and logits through Rggml’s backend-neutral transfer interface. The host GGUF mapping remains the authoritative storage, while device residency is an execution decision rather than another model format. Vulkan also ships in Rggml, but Rllm does not select it. The rig suite checks whole-batch and cached CUDA logits, plain and fmalloc cache slabs, and cache handoff in both directions between CPU and CUDA.

The suite distinguishes cache correctness from invariance to batch shape. For an identical prefix shape, cached and uncached logits from the real SmolLM2 Q4_K_M file are bit-identical on both CPU and CUDA. CPU also gives the same result when a four-token prefix is evaluated whole or incrementally. CUDA does not: on the structural token sequence 1, 5, 9, 2, prefix-versus-whole cosine similarity ranges from 0.761 to 0.997 and two of four argmaxes differ. Real-shape products for all four quantized weight types remain within 1.25e-4 NMSE of CPU, and CUDA one-column and four-column products agree exactly. The remaining numerical question is therefore at graph level: pin Rllm’s result against the matching upstream GGML and llama.cpp paths for each batch shape before claiming shape-invariant CUDA execution.

Persistent allocation by itself is not the missing acceleration. A measured GGML scheduler experiment reduced CUDA decode from 69.7 to 63.7 tokens/s, and CUDA graph capture reached 62.6 because the attention extent and graph properties change at every token. Both experiments were removed from the default path. Stable graph shapes and a device-resident mutable KV cache remain open execution questions; the storage and GGUF interfaces need no fork for either.

Install

From the r-universe:

install.packages("Rllm",
  repos = c("https://rgenomicsetl.r-universe.dev", getOption("repos")))

or via a GitHub subdir ref: pak::pak("sounkou-bioinfo/Rfmalloc/packages/Rllm"). The CPU stack targets Linux, macOS and Windows; the optional CUDA build is Unix-like. GPL (>= 2).