Hi,
I'm working on retrieving records using a paged approach with a page size limit.
When calling the initial endpoint, the response includes an @odata.nextLink field. I’ve renamed the relative URI for this forum to prices, so it looks like this:
"@odata.nextLink": "prices?$skip=20"
It seems the service uses the $skip parameter for pagination, and the initial call returns 20 records as expected. My relative URI is:
sml.svc/prices
However, after the first 20 records are returned, it stops. I don't receive the subsequent pages, and the paging doesn’t seem to continue.
I also used skip itself but then there is no end.
I need proper paging support, as some responses contain a large number of values and could potentially exceed size limits if I try to load everything in one request.
Could you advise how to correctly handle or configure the paging mechanism for this endpoint?
Thanks in advance!
Did you use the "Custom relative URI" option for pagination? I imagine it should look something like this
Thanks Tyler.
Yes, the relative URI is used as well. The issue is that @odata.nextLink disappears when there’s no more data — the response then has an empty "value": [].
In Celigo, this causes:
"@odata.nextLink" not defined in the model"
I tried setting "Path to paging complete field in HTTP response body", but none worked.
Also tried:
{{#if previous_page.full_response.[@odata.nextLink]}}sml.svc/{{{previous_page.full_response.[@odata.nextLink]}}}{{/if}}
But then it repeats the same request and stops pagination early with:
"Identical response received from next page request"
Any idea how to proceed.
What if you instead try using the "Skip number parameter method" and place {{export.http.paging.skip}} in your initial call where it starts at 0 and also set your path to records to value?
I assume the status code you get back is just a normal 200 and nothing special when no more results are available?
Thanks this worked @tylerlamparter
However, I noticed a few things:
When I don’t set the path to the record ("value"), I run into this error:
response stream exceeded limit of 5242880 bytes
Trying to understand why this happens, here’s what I’ve noticed:
When I do use "value" as the path, the debug log shows the last response as:
{
"@odata.context": "https://removedurl",
"value": []
}
But when I don’t use "value", the last response includes data and a @odata.nextLink, and then fails with the stream size error.
My understanding is:
With "value" as path:
Only the contents of the "value" array are processed as individual records. This seems more efficient and stream-friendly.
Without "value":
The entire response, including metadata and nextLink, is treated as one big object. This likely increases memory usage and can hit the 5MB stream limit.
{
"@odata.context": "...",
"value": [...],
"@odata.nextLink": "..."
}
It might be slightly slower per record when streaming individually, but that overhead seems minimal compared to the benefit of avoiding the stream size error.
Yeah, that makes sense. I’m guessing the root issue is how the skip parameter is used in combination with not setting the record path.
When no path is set, integrator.io treats the entire response as a single record—so even if the response contains 20 items in a value array, it still counts as just one. So it does something like:
GET /items?skip=0 → returns 20 items in `value`, but IO sees it as 1 record
GET /items?skip=1 → returns mostly the same data again, minus one
Because IO thinks it's only processing 1 record at a time, it increments skip by 1 instead of by 20. This results in overlapping data and eventually a bloated stream that hits the 5MB limit.
Once you set the path to "value", IO correctly treats each item in the array as an individual record and can paginate properly—returning a full page of 20 distinct records instead of repeatedly re-fetching overlapping ones.
