Serialized R objects in BLOBs

For objects outside the native mappings, serialize them yourself. A raw vector is stored as a BLOB. CanardAbsurd never picks a codec, decodes bytes, or falls back to serialization on your behalf. The same contract holds over Quack.

Base R

A matrix and a language object are not native values, so the handler serializes them in a format it controls:

library(CanardAbsurd)
db <- ca_open()
object <- list(matrix = matrix(1:6, 2L), expression = quote(x + 1L))
id <- ca_spawn(db, "rds-v3", serialize(object, NULL, version = 3L))
outcome <- ca_run(ca_claim(db), function(input, task) {
  ca_step(task, "transform", function() {
    value <- unserialize(input)
    value$matrix <- value$matrix * 2L
    serialize(value, NULL, version = 3L)
  })
})
unserialize(outcome$result)
#> $matrix
#>      [,1] [,2] [,3]
#> [1,]    2    6   10
#> [2,]    4    8   12
#> 
#> $expression
#> x + 1L

SQL sees only the type and size of the bytes, not what is inside them:

DBI::dbGetQuery(db@con, "
  WITH payload AS MATERIALIZED (
    SELECT input FROM canard_absurd.tasks WHERE id = ?
  )
  SELECT variant_typeof(input) AS native_type,
         octet_length(input::BLOB) AS bytes
  FROM payload
", params = list(id))
#>   native_type bytes
#> 1        BLOB   221

qs2

qs2 also produces raw vectors. Choose the decoder in the worker:

packed <- qs2::qs_serialize(object, nthreads = 1L)
ca_spawn(db, "qs2-v1", packed)
#> [1] "0fcebe84-1c46-4fc2-8e88-7672d67afc57"
outcome <- ca_run(ca_claim(db), function(input, task) {
  ca_step(task, "copy", function() {
    value <- qs2::qs_deserialize(input, nthreads = 1L)
    qs2::qs_serialize(value, nthreads = 1L)
  })
})
qs2::qs_deserialize(outcome$result, nthreads = 1L)
#> $matrix
#>      [,1] [,2] [,3]
#> [1,]    1    3    5
#> [2,]    2    4    6
#> 
#> $expression
#> x + 1L

Sakura: rebuilding reference objects

Objects with external pointers, such as Arrow tables, torch tensors and open connections, cannot be copied into another R session. Sakura serializes them through class-specific hooks that record how to rebuild them. Here the hook saves a reader’s bytes and cursor position, and the rebuilt object opens a fresh connection at that position.

cfg <- sakura::serial_config("buffer_reader",
  function(reader) {
    serialize(list(bytes = reader$bytes, position = seek(reader$con)), NULL)
  },
  function(bytes) {
    saved <- unserialize(bytes)
    reader <- new.env(parent = emptyenv())
    reader$bytes <- saved$bytes
    reader$con <- rawConnection(saved$bytes, "rb")
    seek(reader$con, saved$position)
    class(reader) <- "buffer_reader"
    reader
  })

reader <- new.env(parent = emptyenv())
reader$bytes <- charToRaw("alpha\nbeta\n")
reader$con <- rawConnection(reader$bytes, "rb")
class(reader) <- "buffer_reader"
readLines(reader$con, n = 1L)
#> [1] "alpha"
packed <- sakura::serialize(reader, cfg)
close(reader$con)

The handler rebuilds the reader, uses it, and cleans up. Its checkpoint is a plain string:

ca_spawn(db, "buffer-reader-v1", packed)
#> [1] "78d503db-9274-4aa5-976f-6f5c53fcf7fd"
outcome <- ca_run(ca_claim(db), function(input, task) {
  ca_step(task, "line", function() {
    restored <- sakura::unserialize(input, cfg)
    on.exit(close(restored$con))
    readLines(restored$con, n = 1L)
  })
})
outcome$result
#> [1] "beta"
ca_close(db)

Your responsibilities

  • Version the format. Record the serializer, its version and the hook configuration with the task or its handler name. Sakura selects hooks by position, so reordering a configuration changes which decoder reads old bytes.
  • Match the environment. Workers need compatible packages and decoders. Sakura uses native binary serialization, so test every architecture you mix.
  • Test restoration. Check a rebuilt object’s operations in a fresh process. Rebuilding does not resume a transaction, socket or call stack, and files or services the object refers to must still exist.
  • Deserialize only trusted bytes. R objects and hooks can carry executable code.
  • Resubmit the original bytes. Equal R objects do not always serialize to identical bytes, and idempotent resubmission compares bytes.
  • Keep bulk data elsewhere. BLOBs travel as hexadecimal SQL text, twice their size. Store large datasets in files or dedicated tables and pass references.