Oracle Cloud Infrastructure Documentation

Fan Out Messages to Many Consumer Groups in a Queue

Fan out messages from a single OCI Queue to many consumer groups and apply filters for each group, enabling scalable, low-latency, pull-based application-to-application messaging

With fanout, you can send one message to different consumer groups at the same time, making it easy to share updates with many teams or services from just one queue. This is useful because you can control exactly who gets which messages, so every group only sees what's relevant to them.

To use fanout, you create filters that evaluate the message attributes attached to each message. Only the consumer groups for which the filter evaluates to true receive the message.

Consumer Groups

A consumer group is a persistent, logical delivery path within a queue. Each consumer group receives messages separately, based on the filters you set.
Note

You need to enable consumer groups before you can use them in a queue. You can do this when you create or edit a queue.
When you enable consumer groups for a queue, the Queue Service automatically creates a default group called the primary consumer group. While you can't add filters to the default group or delete it, you can disable it if needed. You can think of this group as a superset: as long as it's enabled, it receives all messages published to the queue, regardless of any filters set for other groups.
Note

The primary group ensures every message has a destination. When you set the primary consumer group to inactive, you don't receive messages through it. The system drops any messages sent while the primary group is inactive and that don't match filters in other groups; these messages can't be recovered. To avoid losing messages, keep the primary consumer group enabled unless you specifically intend to discard some messages.

In addition to the default consumer group, you can add as many consumer groups as you need to deliver messages to specific consumers. Each group operates independently of the others, so messages are delivered, filtered, and tracked separately, ensuring isolation and fairness among consumers. You can set a filter for each group to control which messages it receives. If you don't set a filter, or leave the filter empty, the group receives all messages, as with the default group.

Here's a rundown of the differences between the default (primary) consumer group and the consumer groups you create:

Feature Default Consumer Group (Primary) Other Consumer Groups
How created Created automatically when consumer groups are enabled Created manually as needed
Can be deleted No Yes
Can be disabled Yes Yes
Can have filters No Yes
Receives all messages Yes, if enabled Only messages matching its filter, or all messages if no filter is set
DLQ delivery count can be set No Yes
OCID Same as the queue OCID Unique OCID per group

Usage example

Suppose you create a queue for billing events. The default consumer group is for the billing compliance team, which needs to receive all messages for compliance reasons. You also add another consumer group for the team that handles executive expenses, and set a filter so this group only receives messages with the attribute team = "exec-expenses". This reduces noise for the executive expenses team by ensuring they only get relevant message.

Messages and Attributes

A message can optionally include attributes, key-value pairs added by the producer as extra information (metadata). Attributes can describe any detail about the message (such as region, priority, or event type) and help you perform fine-grained filtering and targeting of messages to consumer groups.

Here are some guidelines for producers when creating attributes:

  • Attribute keys and values are case-sensitive.
  • Attribute values must be one of these supported data types:
    • String: A UTF-8 encoded string.
    • Number: An integer or decimal numeric value.
  • Attributes count toward the total message size.
Here are some tips when naming attribute keys:
  • Attribute keys must start with a letter or an underscore (_).
  • Attribute keys can only include letters, numbers, hyphens (-), or underscores (_).
  • Don't use quotes around keys.

Filtering

Use filters to control which messages a consumer group receives. When a message is published, Queue Service checks the filter against each message's attributes to decide if the message should be delivered to that consumer group. A message is delivered to a consumer group only if its filter matches the attributes
Note

Filters work only with message attributes, not the message body.

You create filters when you create a consumer group. Each consumer group can have its own filter, and you can create one filter per consumer group.

A filter expression is an SQL-like rule you write to decide which messages a group gets.

Filter expression constraints

Here are the rules that define how you write and use filter expressions, so the filters work the way you want them to:
  • Filter expressions are case-sensitive. For example, :region = "us-ashburn-1" is different from :region = "US-ASHBURN-1".
  • Filter expressions can be up to 256 characters long.

Filter Evaluation

Here's what happens when OCI Queue Service checks filters and what happens to messages that don't match any filters:
  • Queue Service checks filters only when a message is first published to the queue.
  • Queue Service checks filters only when a message is first published to the queue. If you later re-enable a consumer group or change its filter, existing messages in the queue aren't retroactively evaluated. Only new messages are checked against the updated filter.
  • If no consumer group's filter matches a message, and the primary group is disabled, the message is dropped and can't be retrieved. The queue emits a metric whenever a message is dropped. You can view this metric in the Console in Queue Service or in OCI Monitoring.

Filter Expressions Model

Here are the types of filter expressions you can use:

Key presence

Use key presence filters to check if a message includes or is missing a certain attribute.

Here are some sample filter expressions:

  • :region: Matches messages that have the region attribute set, no matter what its value is.
  • NOT :region: Matches messages that don't have the region attribute.

String filters

Use string filters to match messages based on the value of a text attribute. You can check for exact matches, differences, or patterns within the text.

Here are some sample filter expressions:

  • :name = "Bob": Matches messages where the name attribute is exactly 'Bob'.
  • :name != "Bob": Matches messages where the name attribute isn't 'Bob'.
  • hasPrefix(:name, "Bo"): Matches messages where the name attribute starts with 'Bo'.
  • hasSuffix(:name, "ob"): Matches messages where the name attribute ends with 'ob'.
  • NOT hasPrefix(:name, "Bo"): Matches messages where the name attribute doesn't start with 'Bo'.

Number filters

Use number filters to match messages based on a numeric attribute's value.

Here are some examples:

  • :age = 40: Matches messages where the age attribute is exactly 40.
  • :age != 40: Matches messages where the age attribute isn't 40.
  • :age > 40: Matches messages where the age attribute is greater than 40.
  • :age >= 40: Matches messages where the age attribute is greater than or equal to 40.
  • :age < 40: Matches messages where the age attribute is less than 40.
  • :age <= 40: Matches messages where the age attribute is less than or equal to 40.

Logical filters

Use logical operators to combine rules and create more flexible filters. Supported operators are AND, OR, and NOT.
Note

Operator precedence is as follows: parentheses ( ) take highest precedence, followed by NOT, then AND, then OR.

Here are some examples:

  • :name = "Bob" AND :age = 40: Matches messages where the name attribute is "Bob" and age is 40.
  • :name = "Bob" OR :age = 40: Matches messages where the name attribute is "Bob" or age is 40.
  • :name = "Bob" AND NOT :age: Matches messages where the name attribute is 'Bob' and there isn't any age attribute.

Complex filters

Use more complex logic by combining several filters with parentheses and several logical operators.
Note

Operator precedence is as follows: parentheses ( ) take highest precedence, followed by NOT, then AND, then OR.

Here is an example:

  • :name = "Bob" AND NOT (:age > 40 OR :location = "Chicago"): Matches messages where the name attribute is "Bob" and age isn't over 40 or location isn't 'Chicago'.

IAM policies

Review the IAM policies required to work with fanout and consumer groups.

Best practices

  • Start with small filters to check that message delivery is working as expected. Add complex filter logic slowly, testing as you go.
  • Monitor the dropped messages metric in OCI Monitoring. This helps you catch messages that were dropped because no filter matched.
  • Use consistent naming for message attributes from all producers. This makes writing and maintaining filters easier and more reliable.
  • If an attribute key doesn't follow the rules [starting with a letter or an underscore (_), only include letters, numbers, hyphens (-), or underscores (_)], add quotes around it in the filter expression.