RabbitMQ in Production¶

VOR Stream uses RabbitMQ as its message broker. This page follows the RabbitMQ production checklist and describes the deployment defaults and settings available to operators.

Unless stated otherwise, the guidance below assumes the standard VOR Stream topology:

• A single broker node (no RabbitMQ cluster).
• A single-tier deployment: the broker shares one host with the other VOR Stream services.
• The broker is launched and supervised by supervisord, running as the unprivileged service user (vrisk by default).
• Node discovery is registered with Consul.

RabbitMQ configuration is rendered from rabbitmq/template/rabbitmq.conf.j2 in the deployment package. The broker settings described here (the memory and disk thresholds, consumer_timeout) go in the [rabbitmq:vars] group of the inventory (hosts.ini). The variables that describe how clients reach the broker (rabbitmq_user, rabbitmq_password, rabbitmq_vault_path, rabbitmq_host, rabbitmq_amqp_port) are [all:vars] settings, because the compute, super and Django hosts need them too. Variables scoped to the broker group are unavailable to services on other hosts. See the inventory file reference for the full list and the group each variable belongs to.

Resource limits¶

Check the broker's file-descriptor limit and its memory and disk thresholds before running production workloads.

File descriptors¶

RabbitMQ uses one socket per client connection plus file handles for classic-queue indexes. The nofile limit applies separately to each process. Services running as vor_user (default vrisk) inherit the same setting, but each can use that many descriptors independently. The RabbitMQ checklist recommends a production minimum well above the RHEL default of 1024.

In the default non-root deployment (use_become=True with supervisord_root_deployment=False) the deployment raises the limit for you; a root deployment skips that task and uses the host's limit. See File Descriptor Limits for the settings and checks for both deployment modes.

On RabbitMQ 4.x, rabbitmq-diagnostics status no longer reports a file-handle section because the handle cache was removed in 4.0. Use the OS process check in the linked section to verify the limit.

Memory high watermark¶

RabbitMQ throttles publishers once memory use crosses vm_memory_high_watermark. RabbitMQ 4.x's relative default is within the recommended 0.4-0.7 range. VOR Stream uses that default unless the inventory overrides it.

Keep the default unless the host's workload requires a change:

• On a dedicated broker host, you may raise it toward 0.7 to use more of the RAM, but only with monitoring in place and at least 30% of memory left free for the OS and page cache. Going above 0.7 is discouraged.
• On a shared or small host, leave it at the default, or lower it if the broker is competing with other services for memory.

Set vm_memory_high_watermark_relative (or vm_memory_high_watermark_absolute) in the inventory to override. Set at most one; if both are present the relative value takes precedence.

Disk free limit¶

RabbitMQ blocks publishers when free disk falls below disk_free_limit. Its built-in default of 50 MB is too low for production: by the time only 50 MB remains, the volume is nearly full. VOR Stream defaults this to disk_free_limit.absolute = 2GB.

Raise it on hosts with large or shared data volumes. The checklist suggests a floor roughly matching the memory headroom (for example, a few GB on a host with tens of GB of RAM). Override with disk_free_limit_absolute or disk_free_limit_relative in the inventory. The relative value takes precedence if both are set.

Storage isolation¶

The broker runs from {{ vor_root }}/rabbitmq and keeps its data under that tree. Paths on this page are written with the shipped defaults, vor_root of /opt/vor and vor_log_directory of /opt/vor/log; substitute your own if the inventory overrides them. The deployment leaves RABBITMQ_MNESIA_BASE unset, so RabbitMQ uses the default data subdirectory for its install layout. There is no inventory variable for this path. To put the data on a separate volume, mount it at the existing data directory. Check the running broker for the exact path first.

For workloads with substantial queue backlogs (large IFRS9/ECL runs, for instance), isolate this storage from the rest of the host:

• Mount a dedicated volume at the broker's data directory. This improves I/O isolation and protects the data during binary redeployment. A change to rabbitmq_version causes the installer to remove {{ vor_root }}/rabbitmq if it finds no filesystem mount beneath it. That includes the data directory. The installer detects mounts under that path and preserves their contents.
• Because the standard deployment is single-tier, the broker shares disk I/O with every other VOR Stream service. For heavy or latency-sensitive broker workloads, consider a dedicated broker host.
• Use a local filesystem (ext4, xfs) for the data directory. Avoid distributed or network filesystems, whose fsync semantics can differ from local disks and risk data corruption.

Security¶

• The deployment deletes the default guest user unless the configured broker user is itself guest.
• Super reads the broker credential from Vault when it builds a job's run script and exports the result into that script's environment; nodes only read the environment. The path super uses is the playpen's own rabbitmq-vault-path, initialized from the rabbitmq_vault_path the deployment writes to Consul. A playpen can use a different path. Creating one with --rabbitmq-user and --rabbitmq-password bypasses Vault. Whether the credential is dynamic or static depends on the path form, documented under rabbitmq_vault_path in the inventory file reference: the default rabbitmq/creds/vor-midtier-role uses Vault's RabbitMQ secrets engine and issues credentials on demand, while a kv/data/ path returns a stored password.
• If rabbitmq_vault_path is empty, the deployment exports the rabbitmq_user password into the service profile script under /etc/profile.d/ instead. That file has mode 0755, so every local user on the host can read the password. Prefer a Vault path in production. Either way the deployment creates rabbitmq_user as a standing account with the administrator tag and full configure, read, and write permissions on the vhost. Protect and rotate that credential, as described in Changing Stored Passwords. Consider separate per-application users on a broker shared across workloads.
• RabbitMQ restricts the guest account to loopback connections by default. With the account deleted, those logins are also unavailable.
• The deployment generates and distributes the Erlang cookie to the broker host, with mode 0600 and the service user as owner. This is the shared secret for inter-node and CLI authentication. Preserve its ownership and mode when placing the file manually.
• Expose the client ports (AMQP and management, listed in the ports reference) only to application hosts, and restrict the inter-node and CLI ports to broker hosts. RabbitMQ derives the inter-node port from the AMQP port by adding 20000, so it changes if you set rabbitmq_amqp_port. epmd listens on 4369. The deployment does not configure firewall rules. Set them on the host or through your platform.
• Enable TLS by supplying the three tls_*_file_path variables; the playbook derives its own use_tls fact from them, so writing use_tls in the inventory has no effect. Once on, both the AMQP listener and the management plugin serve over TLS, and the plaintext AMQP listener is switched off (listeners.tcp = none). All clients must support TLS to connect. Both listeners are pinned to TLS 1.2. Peer verification (client certificates) is not part of the shipped configuration and would need a change to the template.

Monitoring and alarms¶

RabbitMQ raises resource-driven alarms on memory and disk that block publishers until the pressure clears; file-descriptor exhaustion is not among them (see the warning above). The management plugin is enabled.

The deployment registers rabbitmq_management in Consul with a TCP check against the management port. RabbitMQ's Consul peer-discovery plugin registers the rabbitmq service used by clients. For an external broker with an empty [rabbitmq] group, the deployment instead writes a static rabbitmq service with its own TCP check.

These TCP checks only confirm that a port accepts connections. A stalled broker can still pass, so monitor RabbitMQ's resource alarms as well.

Enable rabbitmq_prometheus manually if you raise the memory watermark or want to collect broker metrics in an existing monitoring system. The deployment does not enable this plugin.

Logging and log rotation¶

Two sources write logs to /opt/vor/log/rabbitmq/:

• supervisord captures the broker's output in rabbitmq.log, the program's stdout_logfile. With redirect_stderr enabled, this includes stderr. supervisord rotates this file by size and keeps a fixed number of backups, using its defaults.
• RabbitMQ writes its own files under RABBITMQ_LOG_BASE in the same directory, named after the node. The VOR Stream deployment does not configure rotation for these files: the template sets no log.* keys, and there is no logrotate entry covering the directory.

On a default installation, logs (/opt/vor/log) and broker data (/opt/vor/rabbitmq) share the root filesystem. Growing logs reduce the free space monitored by disk_free_limit and can trigger a publisher block. With a dedicated data volume, disk_free_limit monitors that volume only. Logs can still fill the root filesystem and disrupt other services, so log rotation is needed in either layout.

Aggregate the whole directory alongside your other host logs.

Clustering¶

The standard topology uses a single broker node. For clustering, VOR Stream supports these options:

• Put more than one host in the [rabbitmq] group and the deployment forms a cluster through Consul peer discovery, distributing the Erlang cookie across the group. Everything on this page still applies, per node.
• Set rabbitmq_host to a load balancer in front of your own cluster, or to an Amazon MQ for RabbitMQ endpoint, and leave the [rabbitmq] group empty; the deployment then only provisions the vhost and user over the management API. Configure that broker separately; the supervisord, storage, descriptor and logging sections above apply to brokers installed by the playbook. The client connection model still determines how many connections VOR Stream will open against it.

If you scale out to a cluster of your own:

• Use an odd number of nodes (3, 5, 7) so a majority is unambiguous.
• Choose a partition-handling strategy before going live; pause_minority is the usual choice with an odd node count.
• Review net_ticktime for your network, and keep node clocks synchronized with NTP (some plugins, including management, use local timestamps).

Configurable limits¶

Playpen isolation is built on RabbitMQ virtual hosts: a playpen carries a vhost (vor create playpen --rabbitmq-vhost, falling back to the Consul default), and super exports it to every job it launches. The deployment sets no per-vhost, per-connection or per-channel limits, so nothing stops one playpen's jobs from consuming the whole broker.

On a busy installation or shared broker, apply RabbitMQ's configurable limits per vhost to contain a runaway job or a connection leak.

Client connection model¶

When sizing the descriptor limit, account for how each runtime connects:

• Go nodes open one long-lived connection per node process and multiplex channels over it: one per queue the node reads or writes, one for job control, and short-lived ones for control traffic. Socket count tracks process count.
• Python nodes run their workers as separate OS processes. Every worker opens its own connection per queue plus one for the command listener, because a pika connection cannot be shared across processes. Socket count is therefore roughly queues x worker processes. A node with many queues and workers can therefore use many more sockets than an equivalent Go node.

Super holds one connection of its own, with several channels for job replies and control traffic.

Channels share their connection's socket. Size the descriptor limit against peak connection use during your largest runs. For Python nodes, include each worker's queue connections in the estimate.

Queue durability¶

VOR Stream declares per-job queues as transient (non-durable) classic queues. RabbitMQ disables these by default from 4.3 onward, so the deployment explicitly allows them with deprecated_features.permit.transient_nonexcl_queues = true. This setting temporarily preserves compatibility with VOR Stream's queue model. When RabbitMQ removes support for these queues, VOR Stream will need durable or quorum queues, or a replacement for RabbitMQ in node-to-node data transfer. Those changes require a software update.