We are fetching data, but for some transactions, the data size is so large that it errors out with the message below. The challenge is that this involves just a single transaction, so splitting it into smaller chunks isn’t an option in this case.
Does anyone have any insights or suggestions on how to handle such transactions?
What application is this for, and what's its purpose? When you make this API call to the application, do you expect a single result back, or do you expect many transactions to be returned for a single API call?
Application is Magaya. The purpose is to retrive the invoice details with related jobs and its segments. In a single api call only 1 transaction is being returned, in some cases the transaction is so large we are seeing this error.
I assume you don't need all the information from the response, just certain fields from it. Does the API allow you to send a parameter for fields which would let you selectively choose what fields you get back in the response?
Could you try:
Adding a path to records on your lookup that points to the details?
Adding a transformation on the lookup to only keep the fields you need from the response?
No, we don't have an option to fetch only certain fields — we actually need just one field from the entire transaction. To handle this, we’ve added a branch in the flow: whenever this error occurs, we route it through that branch to fetch the data without the related transactions, and then merge it back into the main branch for further imports.
We're currently exploring whether Magaya can build a custom field and provide that specific data directly on the invoice transaction itself, instead of us having to fetch the entire related transaction.