1 Getting started

RedisParam implements a BiocParallel backend using redis, rather than sockets, for communication. It requires a redis server; see ?RedisParam for host and port specification. redis is a good solution for cloud-based environments using a standard docker image. A particular feature is that the number of workers can be scaled during computation, for instance in response to kubernetes auto-scaling.

2 Use

Ensure that a redis server is running, e.g., from the command line

$ redis-server

2.1 Manager and workers from a single R session

On a single computer, in R, load and use the RedisParam package in the same way as other BiocParallel backends, e.g.,

p <- RedisParam(workers = 5)
result <- bplapply(1:7, function(i) Sys.getpid(), BPPARAM = p)

2.2 Independently-managed workers

For independently managed workers, start workers in separate processes, e.g.,

p <- RedisParam(jobname = "demo", is.worker = TRUE)

Start and use the manager in a separate process. Be sure to use the same jobname =.

Sys.getpid()            # e.g., 8563
p <- RedisParam(jobname = 'demo', is.worker = FALSE)
result <- bplapply(1:7, function(i) Sys.getpid(), BPPARAM = p)
unique(unlist(result)) # e.g., 9677

Independently started workers can be terminated from the manager


Session info

This version of the vignette was built on 2024-05-01 with the following software package versions: