| Title: | Stream ClinVar XML into Relational DuckDB Tables |
|---|---|
| Description: | Streams official 'ClinVar' VCV XML releases, including gzip inputs, into focused relational tables in 'DuckDB'. A package-owned native extension uses libxml2's forward reader to retain VCV variants, normalized alleles, RCV aggregates, SCV submissions, conditions, observations, citations, and attributable evidence text without loading the release into R. ClinVarbitration policy is a derived SQL layer over those retained source relations, with a separate direct reproducer for archived NCBI flat-file releases. |
| Authors: | Sounkou Mahamane Toure [aut, cre], DuckDB Foundation [cph] (Bundled DuckDB C extension headers), Centre for Population Genomics [cph] (ClinVarbitration 2.2.11 decision-policy semantics) |
| Maintainer: | Sounkou Mahamane Toure <[email protected]> |
| License: | GPL (>= 2) |
| Version: | 0.1.1 |
| Built: | 2026-07-21 21:33:24 UTC |
| Source: | https://github.com/RGenomicsETL/RClinVarbitration |
Downloads either the current ClinVar release or a named monthly archive from
the NCBI HTTPS service. The VCV XML file is the normal input to
rclinvarbitration_import_xml(). The two flat files are optional,
validation-only inputs to
rclinvarbitration_reproduce_clinvarbitration_parquet().
rclinvarbitration_download_clinvar( release = "latest", file = "vcv_xml", cache_dir = tools::R_user_dir("RClinVarbitration", "cache"), overwrite = FALSE, quiet = FALSE )rclinvarbitration_download_clinvar( release = "latest", file = "vcv_xml", cache_dir = tools::R_user_dir("RClinVarbitration", "cache"), overwrite = FALSE, quiet = FALSE )
release |
|
file |
One or more of |
cache_dir |
Directory in which downloaded files are cached. |
overwrite |
Download even when a valid cached file is present? |
quiet |
Passed to |
Existing files form a local cache. Current files and archived VCV XML files
are checked against NCBI's MD5 sidecar; a matching file is reused and a stale
or corrupt file is replaced only after a temporary download is verified.
NCBI does not publish MD5 sidecars
beside archived flat files, so those files are reused by name unless
overwrite = TRUE.
A complete VCV release is several gigabytes. The default cache location is
returned by tools::R_user_dir("RClinVarbitration", "cache"). The function
raises R's download timeout to at least the value of option
RClinVarbitration.download_timeout (7200 seconds by default) for the call.
A named character vector of local paths. The download attribute is
a data frame containing source URLs, MD5 values when available, and cache
hit status.
xml <- rclinvarbitration_download_clinvar(release = "latest") xml_2026_03 <- rclinvarbitration_download_clinvar(release = "2026-03")xml <- rclinvarbitration_download_clinvar(release = "latest") xml_2026_03 <- rclinvarbitration_download_clinvar(release = "2026-03")
Loads the package-owned rclinvarbitration extension. Its native
clinvar_xml_entities(path) table function and rclinvar_json_field()
scalar are the compact, ClinVar-specific one-pass staging surface used by
rclinvarbitration_import_xml(). The connection must have been created with
duckdb::duckdb(config = list(allow_unsigned_extensions = "true")), as for
any locally built DuckDB extension.
rclinvarbitration_enable(con, extension_path = NULL)rclinvarbitration_enable(con, extension_path = NULL)
con |
A DuckDB DBI connection. |
extension_path |
Optional explicit exact-version extension path. |
con, invisibly.
Writes the seven columns in Centre for Population Genomics ClinVarbitration's
clinvar_decisions.tsv: contig, position, reference, alternate,
clinical_significance, gold_stars, and allele_id. The source is the
package's allele-level policy view, not the disease-level decision view, so
each output row is usable as a locus/alleles annotation record. Both GRCh37
and GRCh38 are supported. The output retains every qualifying source locus,
including distinct X/Y locations for one AlleleID.
rclinvarbitration_export_clinvarbitration_parquet( con, path, release_id, assembly = c("GRCh38", "GRCh37"), profile_id = "default", submitter_exclusions = character() )rclinvarbitration_export_clinvarbitration_parquet( con, path, release_id, assembly = c("GRCh38", "GRCh37"), profile_id = "default", submitter_exclusions = character() )
con |
A DuckDB DBI connection initialized with
|
path |
New output |
release_id |
Imported ClinVar release label to export. |
assembly |
Genome assembly: |
profile_id |
Policy profile identifier, normally |
submitter_exclusions |
Additional submitter names to exclude from this
export. Matching is case-insensitive and ignores surrounding whitespace.
These exclusions are combined with any exclusions already stored for
|
The file is schema-compatible with the upstream TSV/Hail decision resource, but is not claimed to be byte-for-byte equivalent: this package derives submissions and locations from VCV XML, whereas upstream uses ClinVar's tab-delimited submission and variant summaries. PM5 is deliberately not exported; Rduckhts/DuckHTS own downstream consequence and PM5 processing.
A named list describing the written Parquet file, invisibly.
The package uses DuckDB's unstable C extension ABI for its streaming table function, so it bundles one artifact per supported exact engine release.
rclinvarbitration_extension_path(duckdb_version = NULL, duckdb_platform = NULL)rclinvarbitration_extension_path(duckdb_version = NULL, duckdb_platform = NULL)
duckdb_version |
Exact DuckDB version, with or without a |
duckdb_platform |
Exact platform reported by |
An absolute extension path.
The native extension reads .xml and .xml.gz with a libxml2 forward
reader. One compact JSON-backed row per selected ClinVar entity is written to
a disk-backed staging table in one XML pass, projected into the public
ClinVar relations without an EAV pivot, and dropped. No XML DOM, XML blob,
generic parser-node graph, or R data-frame materialization is used. Each
public relation is committed independently, the release catalogue row marks
completion, and failed imports remove partial rows for release_id.
rclinvarbitration_import_xml( con, path, release_id, replace = FALSE, source_url = NULL, source_md5 = NULL )rclinvarbitration_import_xml( con, path, release_id, replace = FALSE, source_url = NULL, source_md5 = NULL )
con |
A DuckDB DBI connection. |
path |
Path to an official ClinVar VCV XML or XML.GZ release. |
release_id |
User-supplied release label stored with every row. |
replace |
Replace rows already stored for |
source_url |
Optional source URL for the release catalogue. When |
source_md5 |
Optional 32-character source MD5 digest. Download metadata is used automatically when available. |
A named numeric vector with imported entity counts.
Initialize the ClinVar relational schema
rclinvarbitration_init(con)rclinvarbitration_init(con)
con |
A DuckDB DBI connection. |
con, invisibly.
Returns DuckDB views implementing the supported policy over
clinvar_disease_submissions. The policy bins submitted classifications,
excludes unknown bins and applies the upstream policy's intended qualified
Illumina benign exclusion, optionally applies profile-specific submitter
exclusions, prefers evidence
evaluated from 2016 onward while always retaining expert-panel and practice
guideline evidence, and applies the 60/20 majority rule. The disease-level
and allele-level views both reproduce the upstream strong-review rule: the
first retained practice-guideline or expert-panel classification in source
order is decisive.
rclinvarbitration_policy_sql( policy_version = rclinvarbitration_policy_version() )rclinvarbitration_policy_sql( policy_version = rclinvarbitration_policy_version() )
policy_version |
Exact supported policy version. See
|
clinvar_policy_pathogenic_alleles is the disease-specific pathogenic or
likely-pathogenic join surface for downstream Rduckhts/DuckHTS annotation.
It does not perform VEP or PM5 computation.
A named character vector of DuckDB CREATE OR REPLACE VIEW
statements.
The identifier pins the policy semantics adapted from Centre for Population
Genomics ClinVarbitration 2.2.11 (upstream commit
658b9f241eb2d43aa11214b153b19c1e18a16337) and records that decisions are
grouped per disease rather than only per variation. The identifier has no
package-local suffix: the view names, rather than a speculative v1, state
whether the output is disease-scoped or allele-scoped.
rclinvarbitration_policy_version()rclinvarbitration_policy_version()
A character scalar.
Reproduces Centre for Population Genomics ClinVarbitration 2.2.11's
allele-level decision algorithm directly from NCBI's versioned
submission_summary and variant_summary files. The generated Parquet has
the exact seven-column clinvar_decisions.tsv layout: contig, position,
reference, alternate, clinical_significance, gold_stars, and
allele_id.
rclinvarbitration_reproduce_clinvarbitration_parquet( con, submission_path, variant_path, path, assembly = c("GRCh38", "GRCh37"), submitter_exclusions = character() )rclinvarbitration_reproduce_clinvarbitration_parquet( con, submission_path, variant_path, path, assembly = c("GRCh38", "GRCh37"), submitter_exclusions = character() )
con |
A DuckDB DBI connection. |
submission_path |
Archived NCBI |
variant_path |
Archived NCBI |
path |
New output |
assembly |
Genome assembly: |
submitter_exclusions |
Submitter names to exclude for a blinded run. |
This is separate from rclinvarbitration_import_xml() and the disease-aware
policy views. It exists to reproduce and differentially validate the upstream
allele-level artifact on matching archived flat-file releases. It uses
DuckDB's streaming CSV reader and SQL aggregation; it does not invoke Python,
Hail, VEP, or PM5 logic.
A named list describing the written Parquet file, invisibly.
Returns DuckDB DDL for the focused ClinVar schema. Public identifiers and domain relations are retained directly: VCV variants, alleles and assembly locations, genes, RCV aggregates, SCV submissions, conditions, observations, citations, attributes, and attributable discovery text. XML parser nodes are not persisted. The release catalogue enforces its small primary key; the release-scale analytical tables expose logical key columns without DuckDB ART indexes so complete imports remain memory-bounded.
rclinvarbitration_schema_sql()rclinvarbitration_schema_sql()
A named character vector of SQL statements.