Most integration data has more than one destination: the order that fulfillment, finance, and marketing all want; the customer update three systems need to hear about; the error every integration should report the same way. If you've handled that by adding another destination step every time a team asks, cloning the flow per team, chaining flows together by listener URL, or polling on a schedule for something that already happened, this post is for you.
Celigo Topics is now in production. Below: what it is, seven patterns it makes straightforward, and a few questions. The questions are the point -- I'm after the use cases I haven't thought of.
What a topic is
A topic is a named channel that stores every message published to it for a retention period you choose -- 1 hour to 90 days, 7 days by default. A flow publishes with a publish step (Import messages into a topic, on the Celigo Topics application, no connection needed). Any number of flows subscribe with a topic listener (Listen for messages from a topic). Every subscription receives every message and keeps its own place, so a flow that was off or failing picks up where it left off. Everything published is browsable in the topic's message history.
Order intake flow ── publish step ──┐ ┌──► Fulfillment flow (pick tickets)
├──► topic: orders.created ──┼──► Finance flow (revenue; backfilled from history)
External system ─── Topics API ─────┘ retention 7 days └──► Customer engagement flow (loyalty points)
message history
Concepts in full: Topics overview.
Seven patterns
1. One event, many teams. An order-intake flow gains one step: publish each order to orders.created. Fulfillment subscribes and makes pick tickets, Finance books revenue, Customer engagement awards loyalty points. Three flows, three owners, and the intake flow doesn't know any of them exist. A fourth team adds a listener -- nobody edits the intake flow.
2. The consumer that joins late. Finance came along after go-live and wanted history. Their listener is set to Start from: All retained messages, so the moment the flow is enabled it works through everything still in retention, then continues with new orders. Finance only wanted orders of $500 or more, so their flow applies an output filter -- every subscription receives every message, and the subscribing flow decides what to act on.
3. Pause a consumer without losing anything. Several of you have asked what happens to real-time data while a flow is switched off for maintenance on the destination, and the honest answer for a webhook listener was "it's gone." A topic subscription holds its position while the flow is off and catches up when it's back, as long as retention outlasts the outage. So set retention longer than your worst-case downtime.
4. One consumer's bad day stays its own. The marketplace channel started sending "n/a" in a numeric field. Customer engagement's records failed at its own step and landed in its error management with retry data; Fulfillment and Finance never noticed. The fix was a mapping edit and Retry -- no re-publish, no coordination.
5. Absorb a burst, work through it in pages. A topic accepts publishes at whatever rate they arrive. Each subscription receives messages in pages -- Page size and Max wait time decide how a page is assembled -- so a subscriber calling a rate-limited API takes fuller pages while a latency-sensitive one takes small ones. A slow subscriber never slows the publisher or anyone else.
6. Hand data between integrations without shared exports or URLs. Topics are account-level resources, registered to integrations the way connections are. A flow in one integration publishes, a flow in another subscribes, and access follows the registration your admins already manage. Some of you have tried to build one shared error-handling flow that every integration feeds, and hit access boundaries doing it from a hook. With a topic, each integration adds a publish step and the shared flow subscribes once.
7. Anything outside Celigo can publish too. The Topics API takes a batch of JSON messages -- one or hundreds per request -- authorized by your existing API tokens. A script, a warehouse job, or a partner system publishes into the same topic your flows read from, with the same retention and history, and no per-flow webhook URL to hand out.
Where it's not the right tool
-
One flow triggers exactly one other flow and losing a record wouldn't matter: a webhook listener is simpler.
-
You need a response back: publishing is one-way. Use an API for request and response.
-
You already run your own broker: use the connectors for it. A Celigo topic is for messaging between flows inside Celigo.
How to start
-
Resources > Topics > + Create topic. Name, optional description, retention, Save. No schema -- any JSON object up to 1 MB is accepted. (Create and manage topics)
-
In the producing flow, add a destination step: Celigo Topics > Import messages into a topic, pick the topic under Publishing, map the fields subscribers need. (Publish messages from a flow)
-
In each consuming flow, add a source step: Celigo Topics > Listen for messages from a topic, pick the topic, set Start from (Only new messages or All retained messages), save, enable the flow. Set Mock output to Live data to design against messages actually in the topic. (Subscribe a flow to a topic)
-
On the topic's row, View messages shows exactly what was published and when -- the shared record between the team that publishes and the teams that subscribe.
Two things to plan for: delivery is at-least-once, so build subscribing flows to tolerate a duplicate, and topics default to 20 per account, which your account team can raise.
Topics is available when the Topics license is enabled for your account. We're rolling the license out to Enterprise and Professional editions; if you don't see Topics under Resources yet, it hasn't reached your account, and your account team can tell you when it will. Replay -- rewinding a subscription to an earlier point in retention from the flow builder -- is next on the list, and what you post here helps decide what comes after it.
Your turn
I read every reply on this thread. Tell me:
-
What would you fan out first? Which event in your business has more than one team waiting on it?
-
Where are you polling on a schedule that should be an event?
-
Where have you daisy-chained flows, called a listener from a hook, or cloned a flow so a second team could get the same data? Those are the wires a topic replaces -- show me the shapes.
-
What retention window would you need? Hours, a long weekend, a full close cycle, and why.
-
What would stop you from using it? A limit, a missing capability, or a setup step that doesn't fit your team.
Screenshots of the flows you'd rewire are welcome. If a pattern you post is one we can build a template around, I'll say so in the thread.