--- title: "Storage, caching, and full-release performance" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Storage, caching, and full-release performance} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ## What is cached? RClinVarbitration uses several distinct forms of reuse. Calling all of them a "cache" hides important lifecycle differences. ### Download cache `rclinvarbitration_download_clinvar()` stores official NCBI files under `tools::R_user_dir("RClinVarbitration", "cache")` by default. It accepts either `release = "latest"` or a specific monthly archive such as `"2026-03"`. ```r latest <- rclinvarbitration_download_clinvar("latest") archived <- rclinvarbitration_download_clinvar("2026-03") ``` The downloader fetches NCBI's MD5 sidecar for VCV XML and for current flat files. A matching local file is reused. A stale or corrupt file is replaced only after a complete temporary download passes its checksum. NCBI does not publish adjacent MD5 sidecars for archived flat files, so those validation-only files are reused by name unless `overwrite = TRUE`. `latest` is a mutable remote alias. For an auditable run, retain the returned URL and MD5 and assign an immutable `release_id`; selecting an explicit monthly archive is preferable. ### Exact DuckDB extension artifacts At package installation, the native C source is built separately for every supported DuckDB version. `rclinvarbitration_enable()` selects the artifact whose embedded DuckDB version and exact `PRAGMA platform` value match the active connection. Windows packages include both `windows_amd64` and R-devel's `windows_amd64_mingw` metadata identities over the same Rtools/MinGW machine code because DuckDB validates that identity literally. This is package-owned compatibility material, not downloaded code and not a cache that silently falls back to another DuckDB ABI. ### Persistent relational release store A file-backed DuckDB database is the durable, queryable conversion of the XML. After import, ordinary queries read typed relations and never rescan or decompress the XML. Multiple `release_id` values may coexist in one database. The `clinvar_releases` row records completion, source path, optional source URL and MD5, byte size, and import time. ```r library(DBI) library(duckdb) xml <- rclinvarbitration_download_clinvar("2026-03") db <- file.path(tools::R_user_dir("RClinVarbitration", "data"), "clinvar.duckdb") dir.create(dirname(db), recursive = TRUE, showWarnings = FALSE) con <- dbConnect(duckdb( dbdir = db, config = list( allow_unsigned_extensions = "true", memory_limit = "2GB", preserve_insertion_order = "false", threads = "2" ) )) rclinvarbitration_enable(con) rclinvarbitration_import_xml(con, xml, release_id = "ncbi-vcv-2026-03") ``` Passing the downloader's returned vector directly lets the importer retain its URL and digest metadata. Extracting only the bare character path drops that R attribute; `source_url` and `source_md5` can then be supplied explicitly. ## Import lifecycle The conversion has four phases: ```text XML.GZ forward scan -> temporary compact entity relation -> typed public relations -> clinvar_releases completion marker -> temporary relation dropped ``` The native table function performs one forward libxml2 scan and streaming gzip decompression. It emits one compact JSON-backed row per selected entity. SQL then projects those rows into the typed relations without an EAV pivot, XML DOM, or R data-frame copy. The staging relation is a durable DuckDB table during conversion so its size is bounded by disk rather than R memory. It is dropped after projection, but DuckDB keeps the released blocks inside the database file for reuse. Therefore the physical file records the import high-water mark and can be substantially larger than its currently used blocks. This is reusable database capacity, not a second live copy of the XML after import. Configure the database and DuckDB `temp_directory` on storage with substantial headroom. If the smallest possible single-release artifact matters more than fast reuse, publish selected relations to a fresh DuckDB database, Parquet, or DuckLake after validation; that compaction/publication time is separate from the import benchmark below. Public relations commit independently so one enormous transaction is not held for the complete release. The release-catalogue marker is inserted only after all projections succeed. An R error removes partial rows and drops staging. If the process is killed so abruptly that cleanup cannot run, no completion marker exists; a later import drops stale staging and clears partial rows for the same `release_id` before writing the release again. `replace = TRUE` stages and validates the XML scan before deleting the existing release. A scan failure therefore leaves the previous complete release intact. A later projection failure is cleaned up and does not restore the replaced rows, so production publication should use a new immutable release ID and switch consumers only after success. ## What remains virtual? Views such as `clinvar_disease_submissions`, `clinvar_policy_decisions`, `clinvar_gene_summaries`, and `clinvar_semantic_documents` are SQL definitions, not automatically refreshed materialized caches. DuckDB plans them against the stored source relations on each query. For repeated delivery workloads, explicitly materialize a release- and policy-versioned result to Parquet, a DuckDB table, or DuckLake. Do not replace the auditable source tables with only the final classification label. The package deliberately avoids ART indexes on release-scale tables. Their build and maintenance memory cost is undesirable during ingestion; analytical filters, scans, joins, Parquet statistics, and downstream materializations are the intended execution path. DuckDB's buffer manager may retain recently read blocks in process memory and spill intermediates to `temp_directory`. That runtime buffer cache is managed by DuckDB and disappears when the process exits. It is different from the durable database and the NCBI download cache. ## Measured complete-release conversion A complete NCBI VCV release was measured, rather than extrapolated from the one-record fixture. The committed machine-readable receipt is `inst/benchmarks/full-release-2026-07-02.dcf`, and `tools/benchmark_full_release.R` reproduces the measurement. | Measurement | Result | |:--|--:| | NCBI release | 2026-07-02 | | Source MD5 | `2e7e76ebbf668910b8688cc5e4284c1b` | | Compressed XML | 5,824,540,370 bytes (5.42 GiB) | | Import wall time | 1,312.493 seconds (21 min 52.5 sec) | | Durable DuckDB file | 23,978,586,112 bytes (22.33 GiB) | | Peak process RSS | 3,657,296 KiB (3.49 GiB) | | VCV / allele rows | 4,531,457 / 4,535,897 | | RCV / SCV rows | 5,966,166 / 6,905,758 | | Condition / text rows | 13,815,000 / 30,649,367 | The run used DuckDB 1.5.3 on Linux, an Intel Core i5-13500, two DuckDB threads, a 2 GB DuckDB memory limit, and `preserve_insertion_order = false`. A DuckDB memory limit is not a process RSS limit: native parser, compression, allocator, and other process memory remain outside that configured buffer budget. The timed scope is `rclinvarbitration_import_xml()` only. Connection creation, extension loading, schema initialization, final `CHECKPOINT`, and process startup are excluded. The complete process including those operations took 21 minutes 53.3 seconds. At the final checkpoint, 29,874 of 91,471 256-KiB blocks were used and 61,597 were free. Thus the 22.33-GiB physical file contained about 7.29 GiB of live blocks and 15.04 GiB reserved for reuse after staging was dropped. Report the physical file size for disk planning; report used/free blocks as well when explaining logical storage. Runtime and compression vary with DuckDB version, CPU, storage, filesystem cache, and configuration; this result is an observed reference, not a universal guarantee. Run the benchmark with: ```sh CLINVAR_DUCKDB_MEMORY_LIMIT=2GB \ CLINVAR_DUCKDB_THREADS=2 \ Rscript tools/benchmark_full_release.R \ ClinVarVCVRelease_2026-03.xml.gz \ clinvar-2026-03.duckdb \ ncbi-vcv-2026-03 \ clinvar-2026-03-benchmark.dcf ``` Use `/usr/bin/time -v` or the platform equivalent around that command when an operating-system peak-RSS measurement is required.