Configuring Dynamic Shovels

Overview

This guide focuses on dynamically configured shovels. It assumes familiarity with the key concepts behind the Shovel plugin.

Unlike with static shovels, dynamic shovels are configured using runtime parameters. They can be started and stopped at any time, including programmatically. Dynamic shovels can be used for both transient (one-off) and permanently running workloads.

Information about dynamic shovels is stored in RabbitMQ's schema database, along with users, permissions, queues, etc. They therefore can be exported together with other schema definitions in combination with the pre-declared topology mode.

Configuration

Parameters can be defined using rabbitmqctl, through the management HTTP API, or (with the rabbitmq_shovel_management plugin enabled) through the management UI's administrative section.

A shovel is declared with a definition body, which is a JSON object. Some keys are mandatory, others are optional. They control connection parameters, protocol used, message transfer source and destination, data safety protocol features, and more.

Every shovel belongs to a virtual host. Note that a Shovel can consume from and publish to not only a different virtual host but an entirely different cluster, so virtual host selection primarily acts as a way of organising shovels and access to them, much like with the rest of permission in RabbitMQ.

Every shovel is also named. The name is used to identify shovels when inspecting their status, deleting them or restarting them.

Declaring a Dynamic Shovel

In this example we will set up a dynamic shovel that will move messages from the queue "source-queue" in the local RabbitMQ cluster to the queue "target-queue" on a remote RabbitMQ node, using AMQP 0-9-1.

Using CLI Tools

A shovel is declared using the rabbitmqctl set_parameter command with component name shovel, a shovel name and a definition body which is a JSON document:

# my-shovel here is the name of the shovel
rabbitmqctl set_parameter shovel my-shovel \
  '{"src-protocol": "amqp091", "src-uri": "amqp://", "src-queue": "source-queue", "dest-protocol": "amqp091", "dest-uri": "amqp://remote-server", "dest-queue": "target-queue", "dest-queue-args": {"x-queue-type": "quorum"}}'

On Windows rabbitmqctl is named rabbitmqctl.bat and command line value escaping will be different:

rabbitmqctl.bat set_parameter shovel my-shovel ^
  "{""src-protocol"": ""amqp091"", ""src-uri"":""amqp://localhost"", ""src-queue"": ""source-queue"", ^
   ""dest-protocol"": ""amqp091"", ""dest-uri"": ""amqp://remote.rabbitmq.local"", ^
   ""dest-queue"": ""target-queue"", ""dest-queue-args"": {""x-queue-type"": ""quorum""}}"

The body in this example includes a few keys:

Essential Dynamic Shovel Definition Settings

Key	Description
src-uri	Source connection URI. Mandatory. See the AMQP URI reference for information on how RabbitMQ treats AMQP URIs in general. cacertfile, certfile, keyfile Client TLS certificate and private key paths. See the TLS guide for details. Only of use when URI scheme is amqps. verify, fail_if_no_peer_cert Use to enable or disable peer verification of the server's TLS certificate. See the TLS guide for details. Only of use when URI scheme is amqps. important Note that starting with Erlang 26, peer verification for TLS clients (such as shovels) is enabled by default. The value of this parameter can either be a string, or a list of strings. If more than one string is provided, the shovel will randomly pick one URI from the list until one of the endpoints succeeds.
src-protocol	Protocol to use when connecting to the source. Either amqp091 or amqp10. If omitted it will default to amqp091. See protocol specific properties below.
src-queue	Source queue that the shovel will consume from. The queue from which to consume. Either this or src-exchange (but not both) must be set. If the source queue does not exist on the target virtual host, and src-queue-args parameter was not provided, shovel will declare a classic durable queue with no optional arguments. Shovels can use a pre-declared topology instead of declaring the source. See the Predeclared topology section below.
src-queue-args	Optional arguments for src-queue declaration, eg. the queue type.
dest-uri	Same as src-uri above but for destination connection.
dest-protocol	Protocol to use when connecting to the destination. Either amqp091 or amqp10. If omitted it will default to amqp091. See protocol specific properties below.
dest-queue	The queue to which messages should be published. Either this or dest-exchange (but not both) may be set. If neither is set then messages are republished with their original exchange and routing key. If the destination queue does not exist in the destination virtual host, and dest-queue-args parameter was not provided, shovel will declare a classic durable queue with no optional arguments. Shovels can use a pre-declared topology instead of declaring the destination. See the Predeclared topology section below.
dest-queue-args	Optional arguments for dest-queue declaration, eg. the queue type.

There are other Shovel definition keys that will be covered later in this guide.

Pre-declared Topology

Shovels can use a pre-declared topology instead of declaring its source and destination.

For example, this may be necessary when the topology is imported from a definitions file at boot time. Using a pre-declared topology avoids a chicken-and-egg problem between the import of definitions and Shovel plugin startup: the plugin must be enabled for definitions to pass validation (of the runtime parameters part).

Here is how the plugin can be configured to wait until the source is available using rabbitmq.conf:

# all shovels started on this node will use pre-declared topology
shovel.topology.predeclared = true

If only some shovels need to use a pre-declared topology, the same behavior can be configured for specific shovels using the following shovel properties:

Additional Dynamic Shovel Definition Settings

Key	Description
src-predeclared	When set to true, the plugin waits until src-queue is available instead of declaring the topology itself using src-queue-args.
dest-predeclared	When set to true, the plugin waits until dest-queue is available instead of declaring the topology itself using dest-queue-args.

Using HTTP API

To declare a shovel using the HTTP API, make sure that the management plugin is enabled, then use the following endpoint:

PUT /api/parameters/shovel/{vhost}/{name}

where {vhost} is the virtual host in which the Shovel should be started and {name} is the name of the new shovel. The endpoint requires that the user that invokes it has policymaker privileges (tag).

The request body is a JSON document similar in structure to that described earlier in this guide:

{
  "value": {
    "src-protocol": "amqp091",
    "src-uri":  "amqp://localhost",
    "src-queue":  "source-queue",
    "dest-protocol": "amqp091",
    "dest-uri": "amqp://remote.rabbitmq.local",
    "dest-queue": "destination-queue"
  }
}

Below is an example that uses curl to declare a shovel on a local node using default user credentials. The shovel will transfer messages between two queues, "source-queue" and "destination-queue", in the default virtual host.

Note that this exact command would fail if invoked against a remote node. Please add a new user tagged as policymaker for your own experiments.

# Note: this user's access is limited to localhost!
curl -v -u guest:guest -X PUT http://localhost:15672/api/parameters/shovel/%2f/my-shovel \
                       -H "content-type: application/json" \
                       -d @- <<EOF
{
  "value": {
    "src-protocol": "amqp091",
    "src-uri": "amqp://localhost",
    "src-queue": "source-queue",
    "dest-protocol": "amqp091",
    "dest-uri": "amqp://localhost",
    "dest-queue": "destination-queue"
  }
}
EOF

Using Management UI

To declare a shovel using the management UI, first make sure that the management plugin is enabled.

Then

• Navigate to Admin > Shovel Management > Add a new shovel
• Fill out the form with shovel parameters covered earlier in this guide
• Click Add shovel

Inspecting Status of Dynamic Shovels

Using CLI Tools

Use rabbitmqctl shovel_status to inspect dynamic shovels in a cluster. The rabbitmq_shovel plugin must be enabled on the host where this command is executed.

rabbitmqctl shovel_status --formatter=pretty_table

The output can be formatted as JSON and redirected to a tool such as jq:

rabbitmqctl shovel_status --formatter=json | jq

Using HTTP API

GET /api/shovels is an endpoint that can be used to list dynamic shovels in a cluster. The endpoint is provided by the rabbitmq_shovel_management plugin which must be enabled on the target node.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/

To inspect shovels in a specific virtual host, use GET /api/shovels/{vhost} {vhost} is the virtual host name. The value must be percent-encoded.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/%2f

To inspect status of a specific shovels, use GET /api/shovels/vhost/{vhost}/{name} {vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel. Both values must be percent-encoded.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X GET http://localhost:15672/api/shovels/vhost/%2f/my-shovel

Using Management UI

• Navigate to Admin > Shovel Status
• Locate the shovel of interest in the table

Restarting a Shovel

A dynamic Shovel can be restarted. Restarting a shovel briefly interrupts its operations and makes it reconnect to both source and destination. When an appropriate acknowledgement mode is used by a shovel, the interruption is safe: any unacknowledged or unconfirmed ("in flight") messages consumed from the source or published to the destination will be automatically requeued when the shovel is stopped, and consumed again after the restart.

Using CLI Tools

Use rabbitmqctl restart_shovel to restart a shovel using its name. The rabbitmq_shovel plugin must be enabled on the host where this command is executed.

rabbitmqctl restart_shovel "my-shovel"

Using HTTP API

DELETE /api/shovels/vhost/{vhost}/{name}/restart is an endpoint that restarts a dynamic shovel. The endpoint is provided by the rabbitmq_shovel_management plugin which must be enabled on the target node.

{vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel to be restarted. Both values must be percent-encoded.

# Note: this user's access is limited to localhost!
 curl -v -u guest:guest -X DELETE http://localhost:15672/api/shovels/vhost/%2f/my-shovel/restart

Using Management UI

• Navigate to Admin > Shovel Status
• Locate the shovel of interest in the table
• Click Restart and wait for the next UI refresh

Deleting a Shovel

Using CLI Tools

To delete a Shovel using CLI tools, use rabbitmqctl clear_parameter and pass shovel for component name and the name of the shovel that should be deleted:

rabbitmqctl clear_parameter shovel "my-shovel"

Using HTTP API

DELETE /api/parameters/shovel/{vhost}/{name} is the endpoint that can be used to delete a shovel.

{vhost} is the virtual host in which the Shovel is running and {name} is the name of the shovel to be deleted. Both values must be percent-encoded.

# Note: this user's access is limited to localhost!
curl -v -u guest:guest -X DELETE http://localhost:15672/api/parameters/shovel/%2f/my-shovel

AMQP 0-9-1 Shovel Definition Reference

There are several Shovel properties that haven't been covered in the above example. They don't change how dynamic shovels work fundamentally, and do not change the declaration process.

Optional Dynamic Shovel Definition Settings (AMQP 0-9-1)

Key	Description
reconnect-delay	The duration (in seconds) to wait before reconnecting to the brokers after being disconnected at either end. Default is 1.
ack-mode	Determines how the shovel should acknowledge consumed messages. Valid values are on-confirm, on-publish, and no-ack. on-confirm is used by default. If set to on-confirm (the default), messages are acknowledged to the source broker after they have been confirmed by the destination. This handles network errors and broker failures without losing messages, and is the slowest option. If set to on-publish, messages are acknowledged to the source broker after they have been published at the destination (but not yet confirmed). This handles network errors without losing messages, but may lose messages in the event of broker failures. If set to no-ack, automatic message acknowledgements will be used. This option will offer the highest throughput but is not safe (will lose messages in the event of network or broker failures).
src-delete-after	Determines when (if ever) the shovel should delete itself. This can be useful if the shovel is being treated as more of a move operation - i.e. being used to move messages from one queue to another on an ad hoc basis. The default is never, meaning the shovel should never delete itself. If set to queue-length then the shovel will measure the length of the source queue when starting up, and delete itself after it has transferred that many messages. If set to an integer, then the shovel will transfer that number of messages before deleting itself.
src-prefetch-count	The maximum number of unacknowledged messages copied over a shovel at any one time. Default is 1000.
src-exchange	The exchange from which to consume. Either this or src-queue (but not both) must be set. The shovel will declare an exclusive queue and bind it to the named exchange with src-exchange-key before consuming from the queue. If the source exchange does not exist on the source broker, it will be not declared; the shovel will fail to start.
src-exchange-key	Routing key when using src-exchange.
src-consumer-args	Consumer arguments, such as x-single-active-consumer or x-stream-offset.
dest-exchange	The exchange to which messages should be published. Either this or dest-queue (but not both) may be set. If the destination exchange does not exist on the destination broker, it will be not declared; the shovel will fail to start.
dest-exchange-key	Routing key when using dest-exchange. If this is not set, the original message's routing key will be used.
dest-publish-properties	A map (JSON object) of properties to overwrite when shovelling messages. Setting headers this way is not currently supported. Default is .
dest-add-forward-headers	Whether to add x-shovelled headers to the shovelled messages indicating where they have been shovelled from and to. Default is false.
dest-add-timestamp-header	Whether to add x-shovelled-timestamp headers to the shovelled messages containing timestamp (in seconds since epoch) when message had been shovelled. Default is false.

AMQP 1.0 Shovel Definition Reference

AMQP 1.0 source and destination properties have some differences from their AMQP 0-9-1 counterparts.

Optional Dynamic Shovel Definition Settings (AMQP 1.0)

Key	Description
src-uri	The AMQP URI for the source. Mandatory. AMQP 1.0 URIs implement as subset of what is described in the AMQP URI reference. There is no virtual host concept in AMQP 1.0, so URI path segments are not supported. The set of query parameters it supports are different from AMQP 0.9.1 URI(s): idle_time_out heartbeat interval hostname The name of the target host. Certain vendors (such as Azure ServiceBus) require this to be set even if it is the same as the host segment in the uri. sasl anon, none or plain Defaults to: none. When using plain the user and password segments of the URI need to be set. cacertfile, certfile, keyfile Client TLS certificate and private key paths. See the TLS guide for details. Only of use when URI scheme is amqps. verify, fail_if_no_peer_cert Use to enable or disable peer verification of the server's TLS certificate. See the TLS guide for details. Only of use when URI scheme is amqps. ::: important Note that starting with Erlang 26, peer verification for TLS clients (such as shovels) is enabled by default. :::
src-address	The AMQP 1.0 link address. Mandatory.
dest-address	The AMQP 1.0 link address. Mandatory.
src-prefetch-count	The maximum number of unacknowledged messages copied over a shovel at any one time. Default is 1000.
dest-properties	Properties to overwrite when shovelling messages. See AMQP 1.0 spec §3.2.4 for details of all possible properties.
dest-application-properties	Application properties to set when shovelling messages.
dest-add-forward-headers	Whether to add x-shovelled application properties to the shovelled messages indicating where they have been shovelled from and to. Default is false.
dest-add-timestamp-header	Whether to set the creation_time header to the timestamp (in milliseconds since epoch) of the moment when message had been republished. Default is false.
reconnect-delay	The duration (in seconds) to wait before reconnecting to the brokers after being disconnected at either end. Default is 1.
ack-mode	Determines how the shovel should acknowledge consumed messages. Valid values are on-confirm, on-publish, and no-ack. on-confirm is used by default. If set to on-confirm (the default), messages are acknowledged to the source broker after they have been confirmed by the destination. This handles network errors and broker failures without losing messages, and is the slowest option. If set to on-publish, messages are acknowledged to the source broker after they have been published at the destination (but not yet confirmed). This handles network errors without losing messages, but may lose messages in the event of broker failures. If set to no-ack, automatic message acknowledgements will be used. This option will offer the highest throughput but is not safe (will lose messages in the event of network or broker failures).
src-delete-after	Determines when (if ever) the shovel should delete itself. This can be useful if the shovel is being treated as more of a move operation - i.e. being used to move messages from one queue to another on an ad hoc basis. The default is never, meaning the shovel should never delete itself. If set to queue-length then the shovel will measure the length of the source queue when starting up, and delete itself after it has transferred that many messages. If set to an integer, then the shovel will transfer that number of messages before deleting itself.

Monitoring Shovels

See Monitoring Shovels in the overview Shovel plugin guide.