Currently I’m working on a flow to fetch data from a source called small improvements and writing the data to bigquery.
The flow involves fetching goals/objectives of each user for every quarter (objective cycle), after fetching the user objectives for a given objective cycle, I’m trying to fetch if there are any comments on that particular objective through a lookup step, most of the objectives might not have any comments due to which the lookup API calls return empty records. I am handling this in the postResponseMapHook to ignore those records from processing, but I’m getting an issue with this lookup step where for every objective without comments I’m getting the following error.
The lookup operation exceeded the limit of 5242880 bytes. Reduce the page size or use a transformation to remove fields and optimize record structure.
But when I test that particular record that led to this error, the lookup step is working fine, I don’t know where this issue is arising from.
Any help is appreciated!!
Hi, Manjunanth.
Integrator.IO has a response/file size limitation of 5MB depending on the application involved. To know more about file size limitation, you may visit the below article.
https://docs.celigo.com/hc/en-us/articles/4406732367643-Understanding-maximum-file-size-limits
To avoid this error, please reduce your lookup search by adding additional criteria or removing the unnecessary key-value pairs using transformation.
2 Likes
Are you saying when you test with that single record it works, but when you test with multiple records where some records happen to have comments and others don't, then you get this error?
Yes, when I run the lookup preview with the mock data same as the error record, I’m not getting any comments (i.e., there are no comments for that record at the source) but I’m getting the limit exceeded error.
What happens if you actually run it vs just previewing?
When I’m previewing I’m getting an empty record as a response, but when I run it I’m getting the lookup response size exceeded the limit, but when I retry it, it succeeds without any error.
HI Manjunath,
In the lookup, is there additional criteria you can use to limit the output you are getting from the lookup?