We have a custom flow which takes our ecom orders (we also have a custom website backend that uses the Shopify front end) and processes them into Netsuite customers, sales orders and customer deposits.
The flow has a webhook listener as the source and we send one order at a time to the Celigo webhook.
Most of the time the flow will process a small number of orders at a time, but there are times in the day where we have a large number of orders sent in a small window of time as we have subscription orders. Occasionally during these subscription order drops, the flow never completes an order before the next one is sent and can end up running for hours and hours which backs everything up - See screenshots
We have other similar flows (webhook listener > Netsuite) where this does occassionaly happen too, but it doesn’t happen every day.
Has anyone ever seen this? Is there a reason why this happens? Is it a bug with the webhook listener, or is there a setting that I haven’t set which could improve this performance?
@Sam_Tuckett Could you check import settings including concurrency on that connection and if that connection is also shared across different flows ? It seems webhook events are getting captured here based on what you have described.
@bhavikshah I assume you mean the Netsuite connection? Yes we use this across multiple flows all relating to ecom order flow (orders, fulfillments, refunds). The concurrency is set to 10.
It’s worth saying that the refund flow only runs once a day at night so this won’t be having any impact. Fuflillments are also pretty light at this time of day as the bulk of them come through later on.
I have a few suggestions that might help improve performance. Right now, the main challenge is that only one order is being sent per webhook call. This means each record is processed individually, which is much less efficient than processing in batches.
Batch the webhook calls
Instead of sending one order at a time, try sending around 20 orders per webhook call. This lets the flow process a page of data at once, rather than record by record. I am not sure what flexibility you have on your custom website backend, but if batching is possible there, this would be the most effective improvement.
Switch to file-based processing (conditional on queue size)
If realtime is not always required, set up a flow that listens for events and writes them to a file (NetSuite File Cabinet, S3, FTP, etc.), then a second flow that runs every 5–10 minutes to process those files. You could include a parameter in the webhook payload that indicates the number of events in the queue. If that number is above a threshold, route those to the file-based flow for more efficient bulk processing, while still handling smaller volumes in near real-time. Again, I am not sure what is possible in your backend, but if you can send that parameter it gives you more flexibility.
Increase NetSuite connection concurrency (if entitled)
If your NetSuite connection is set to 10, consider raising it to 25, depending on your entitlements. Even if you cannot go past 10 due to your Celigo subscription tier, that restriction is being removed in October. NetSuite response times can vary based on workflows and scripts, so more concurrency often helps.
Reference: Fine-tune integrator.io for optimal performance and data throughput
Add multiple NetSuite connections and split workload
Even if you keep concurrency at 10, you can create an additional NetSuite connection in Celigo and distribute traffic. Put some imports and exports on one connection and others on the second. This effectively bypasses the 10 concurrency limit because Celigo enforces concurrency per connection, not globally. This helps fully utilize available throughput since a single connection may not consistently hit the cap due to variable NetSuite response times.
Use API Builder as an alternative
Another approach is to have one flow receive the webhook and then call API Builder from that flow. API Builder itself does not enforce flow concurrency since it is designed for immediate actions, but the Celigo connection you use in the flow to call the API Builder endpoint will still respect its concurrency settings. That means the effective concurrency should be close to what you already have on the NetSuite connection. Error handling would need to be managed differently in this setup. Your custom backend could also call API Builder APIs directly, but again I am not sure what is possible on your backend, and error management becomes more involved.
