Responses greater than 5MB and how

[Sean_Ayotte]

I joined Office Hours yesterday and had @tylerlamparter help me get this working, but I am, again, stuck. I have followed the construct here: Shopify Payout Transactions: response stream exceeded limit of 5242880 bytes which takes my final step in flow NV02 which is a call to Ramp with the relative URI: /accounting/field-options?field_id={{record.ramp_id}}
For the purposes of this discussion, I will be referring to that record.ramp_id as parent_ramp_id

Refer to the aforementioned link, but my flow NV02 ends with a preSavePage that sends the data off to a Celigo listener on flow NV02.5.
NV02.5 receives this data as something like:
{
"visibility": "VISIBLE",
"display_name": null,
"value": "Trucks - Pickup - .75 TN Dodge- K. Cook",
"updated_at": "2026-06-24T07:56:03+00:00",
"is_active": true,
"id": "DB26B4C495CE0B2E2CR7E4DF91D158A3F8A440D89B7420CD2866C86B0A2E8D09",
"ramp_id": "633bd022-c8ee-486e-8196-9e18b7bd2b2f",
"created_at": "2026-06-24T07:56:03+00:00",
"accounting_connection_id": "e3aab85a-b48c-4ff3-8e30-3af2ca7b901e",
"code": "TR27-UD",
"provider_name": "Api",
"entity_remote_ids": null
}

Cool, I have the >5MB data now flowing into a new step and errors have stopped ... but I am left with 2 issues.

1. I do not have the parent_ramp_id - I have thousands of records, but no way to tell which call spawned them. (Since the data is sent from NV02 with a preSavePage, I cannot use upstream data, but only response data AFAIK.)

   Note: ramp_id in the response is the child ramp_id, not the parent.

2. I do not have a measurable way to know when NV02.5 is complete. NV02 passes 11 records and NV02.5 receives thousands of records; but I need to know when step NV02.5 is done to know to move on to the next step in the process. The next step in my process is a comparative of data to see if there are changes, etc. so there is no way to know that until the data is completely collected, otherwise it will trigger after one record and say - yeah, there's a lot of differences, retire all of these codes, etc.

So, I'm hoping for two things, some way to do a count of records and a way to force either the GET or upstream data into the response (at the preSave step) and a way to determine the completion of ALL records. (If there is no way to tell that the run is complete, I can just put a longer delay than necessary between step NV02.5 and the next step, but that doesn't seem like the correct solution.)

Much appreciation.

For reference, the reason the solution in the attached was not working for me yesterday is that I'm not so bright sometimes - I had the Listener setup without anything downstream (because I wanted to see it fire before I decided to go that route), but apparently a listener without an import doesn't do anything ... The more you know ...

[tylerlamparter]

Good chatting at Office Hours yesterday. The listener/preSavePage route clears the 5MB cap, but it has exactly the two gaps you ran into: the handoff drops the parent context, and nothing tells you when the run is finished. The fix is to stop fanning everything through one listener and process a single parent_ramp_id at a time, driven by a lookup cache.

Three flows:

Flow 1 (your current NV02): instead of calling Ramp and handing off to the listener, its only job is to populate a lookup cache with the parent_ramp_ids that need processing.

Flow 2 (orchestrator): pull one entry from the cache, write that parent_ramp_id into Flow 3's export so the field_id in the relative URI is set dynamically, run Flow 3, then delete that entry from the cache. Loop until the cache is empty.

Flow 3 (worker): GET /accounting/field-options?field_id={parent_ramp_id} for that one parent, then send the results downstream. Right now I have it pointed at a dummy branch so you can watch it run.

How that closes both gaps:

1. parent_ramp_id: Flow 3 only ever runs for one parent, and you set that parent in Flow 2, so it's a known constant for the whole run. Stamp it onto every record in the Flow 3 mapping (separate field from the child ramp_id in the response). No more guessing which call spawned which records.

2. Completion and count: you finish one parent, delete it from the cache, move to the next, and the cache going empty is your done signal. So the comparison step can trigger on an empty cache instead of a padded delay. And since each Flow 3 run is scoped to one parent, counting records per parent is trivial.

Heads up that this changes your insert step, since you're now writing per parent inside a loop instead of one big batch at the end. That's the trade for deterministic ordering and the parent linkage.

[Sean_Ayotte]

@tylerlamparter thank you for the creative solution and response. I am trying to wrap my head around all of this. Looking through the Flows:

Flow 1 makes sense as it is similar to what we had previously: Source (Ramp) → ends with parent_id upsert to lookup cache (I had it writing to Azure, which I can dump as I was attempting to do something similar there). So, after this flow ends, I have 11 rows in the Lookup Cache. Flow 1 ends when all (11) records have written to the cache and its completion calls Flow 2.

Flow 2

1. Get parent_id from the Lookup Cache
2. Run the Get Data step from Ramp flow
3. Send the data (by Parent) to Flow 3
4. Remove the parent_id from Lookup Cache (simultaneous to 3 above, not after Flow 3 completes)

Flow 3 Cycle, one parent at a time, into the (Azure) destination tables that are needed in the next Flow (4). After each parent completes, call Flow 2 (serially). So, this should keep the loop going 1→2→3→2→3→2...

This appears to solve the first part of the problem (in a way that I would not have considered), but I'm still trying to understand how this solves problem 2:
After 11 cycles, Flow 2 will have no more parent id records in the lookup cache, so when Flow 3 finishes run 11, it calls Flow 2 which no longer has a Parent Ramp ID to process, so it will try to run step 1 and fail silently ... but I need to somehow Branch here that proceeds as normal when there is a value, but when empty, call Flow 4. I'm noticing that even the successful calls to Flow 2, Step 1 result in a 401 response before a successful 200 response; I was thinking if Step 1 fails, call Step 4, but it needs to be a little more complex than that (I assume) and I cannot do Error handling on an Export step, so I guess that doesn't work regardless. (Actually, all 4 steps have 401s (Unauthorized) along with their respective 200s - is that expected?) A successful run looks like:
Get processable parent ramp id : Success 1 Pages 1
Update export for next parent ramp id : Success 1 Pages 1
Run flow : Success 1 Pages 1
Remove lookup cache entry : Success 1 Pages 1

But the FINAL run is just:
Get processable parent ramp id : Success 0 Pages 0
So whatever I need to do, I need it to be based on the first step (I think) ...

[Sean_Ayotte]

Alright, I have to change everything downstream in Flow 4, but I set a pre Save Hook on 2, step 1:

import { exports } from 'integrator-api';

function preSavePage(options) {
  if (!options.data || options.data.length === 0) {
    exports.run({
      _id: '6a6d66d6d66e1111c1111111',
      listenerData: [{}]
    });
  }

  return {
    data:                  options.data,
    errors:                options.errors,
    abort:                 false,
    newErrorsAndRetryData: []
  };
}

which continues down the rest of the way if there is data; if there is no data, it calls Flow 4's listener ... which then makes me have to do a One-to-many on the following step (because I now get one call starting record, instead of 200+ ... so I need to figure a few things out, but once this is all done, I'll add notes/screencaps in the even this is useful for anyone else.

[tylerlamparter]

Another idea is: instead of making 11 entries in the lookup cache, you could make 1 entry where the value holds all 11 ramp IDs. Then the second flow runs the third flow for just one of the values at a time, and in flow 2 you update the lookup cache entry to say you processed that particular ramp ID. Then when flow 2 runs and all values have been run, you know you're good to go.

This could also let you do things like mark a ramp ID with "called flow 3" and "flow 3 successfully completed" so you have a couple gate checks. In flow 2, you could call the Jobs API to get the latest job results of flow 3 to make sure it was all successful.

Just some more ideas.

[Sean_Ayotte]

I'm struggling with this hand-off from Flow 2 to Flow 4.
Flow 4 before this chaos was kicked off with a call to Azure to find the records that need attention, so let's say there are 20. It would then take those 20 and branch them down their respective paths. Easy-peasy.
Flow 4 is now a listener (since that can be called by the preSave hook) that outputs 1 record that then calls the Azure step referenced above ... but now I have a One-To-Many that needs to branch - which doesn't work (the branch will only allow something like record.data.0.option_id ... which means all 20 records follow the path of record 0). I can split the data with a post response hook, but then Celigo gets made and errors saying expected 1 got 20. So I have 2 non-solutions. I also have the option to create 4 as Listener → generic import and have the completion of this step kick off the real 4, but this means that I'll have added 3 additional flows to mitigate one problem (and 2 is already more than I wanted to add since we only are only afforded so many), but I digress. So, now that I know two ways not to do it, one way I really don't want to do it, I need to know the ideal solution that allows for branching on One-to-Many ...?
(I suppose the other way I can do it, but do not want to is setting 4 on a separate schedule (removing the call to the listener) ... but this has its own drawbacks as well.)

[Sean_Ayotte]

Nevermind, figured out the missing puzzle piece flows.run ... I'll be back shortly to post the results to this very specific problem (in the event anyone else has an issue similar to mine).

[Sean_Ayotte]

Walkthrough of the final solution.
Problem: This the flow I was starting from; the fourth step (Ramp: Get Current Values) was the source of the problem. (It would return >5mb for one of the 11 returns.)

Solution:
Step A: A lookup cache is created (called parentRampIdsNeedingProcessing)
Step B: Clone the offending Flow. Steps after and including the offending issue are removed. A new Celigo step is added to populate the lookup cache created in Step A.

Details:

Step C: Create a New Flow to use the newly populated Lookup Cache and process each item individually. This step is called multiple times. (This flow should be called at the completion of the prior step). There is no Next integration flow from this step.

1. Pull the first record from the lookup cache
   

   
   
   with the preSave Hook:

   import { flows, exports } from 'integrator-api';
   
   function preSavePage(options) {
     let response = {};
     if (!options.data || options.data.length === 0) {
       // Cache empty - trigger Flow 03
       flows.run({
         _id: '<<FLOW ID to the flow that is called after FINAL record is processed>>'
       });
       response.statusCode = 200;
     }
   
     // Pass records through unchanged — don't return empty objects
     return {
       data:                  options.data,
       errors:                options.errors,
       abort:                 false,
       newErrorsAndRetryData: []
     };
   }
   

2. Set the Parent ID for the record you have grabbed in 1

   

   HTTP request body:

{
    "_id": "<>",
    "name": "Ramp: Get Current Values",
    "_connectionId": "<>",
    "pageSize": 100,
    "http": {
        "relativeURI": "/accounting/field-options?field_id={{{settings.export.parent_ramp_id}}}",
        "method": "GET",
        "_httpConnectorVersionId": "<<Perhaps Tyler can share where this value comes from??>>",
        "_httpConnectorResourceId": "<<Perhaps Tyler can share where this value comes from??>>",
        "_httpConnectorEndpointId": "<<Perhaps Tyler can share where this value comes from??>>",
        "formType": "http",
        "paging": {
            "method": "url",
            "path": "page.next"
        },
        "response": {
            "resourcePath": "data"
        }
    },
    "transform": {
        "type": "expression",
        "expression": {
            "rules": [
                []
            ],
            "rulesTwoDotZero": {
                "inputContext": "record",
                "mappings": [
                    {
                        "generate": "parent_ramp_id",
                        "dataType": "string",
                        "extract": "{{{settings.export.parent_ramp_id}}}",
                        "status": "Active",
                        "sourceDataType": "string"
                    }
                ],
                "mode": "modify"
            },
            "version": "2"
        },
        "rules": [
            []
        ],
        "version": "2"
    },
    "settings": {
        "parent_ramp_id": "{{{record.value.ramp_id}}}"
    },
    "adaptorType": "HTTPExport"
}

3. Call the next Flow (Step D)

3. 4. Remove the item from the lookup cache so it won't be there the next time this is called

      

Step D: Using the parent_id set in Step C3 above, Get Values (basically the same step that caused this issue) and continue down the path that was above ... Upon completion of this step, loop back to Step C.

That's about as good as I can explain ...