How to stop lookup calls when the API pagination doesn't have any headers or last page fields

Integrator.io
1

I am currently working on a flow where I am getting data through pagination through a lookup step. But the API doesn't have any parameter or header that specifies the last page. The only way to stop the lookup calls as far as I understand is to stop calling the API after the response has the result field as an empty array ( {result: } ).

I tried the same by setting the Path to paging complete field and expected complete values but met with no luck.

The image shows configuration fields for specifying the path and values to signal paging complete in an HTTP response body. (Captioned by AI)
The image shows configuration fields for specifying the path and values to signal paging complete in an HTTP response body. (Captioned by AI)857×172 7.57 KB

2

What do you have in your "Path to records" field? If you have result there, then you shouldn't have to specify any path to paging complete.

3

My "Path to records" is empty which is when I'm getting the above error. I didn't add it to the path because when I did, the pagination wasn't working. Let's say if I had 500 total records coming from the API with a limit and skip each of 100. The flow was only working for the first 100 records, which is why I removed it from the "Path to records".

4

You should have path to records set to result and then paging should use skip number parameter.

5

Hey, just tried that. Still it's the same issue. I verified from the API that there are around 400 records. But when I set the path to records to result, the flow is still running with just first 100 records.

6

Can you share screenshots of your setup? Also, can you share the docs of the api you are calling?

7

I am using Bonusly API for getting the users data. Here is the API documentation

Here are the setup screenshots.

A screenshot of an API configuration interface showing a GET request setup with a specified relative URI for exporting user data from a hypothetical API. (Captioned by AI)
A screenshot of an API configuration interface showing a GET request setup with a specified relative URI for exporting user data from a hypothetical API. (Captioned by AI)902×258 16 KB

The image shows a configuration panel for export type and paging method settings in an API interface. (Captioned by AI)
The image shows a configuration panel for export type and paging method settings in an API interface. (Captioned by AI)902×343 14 KB

FYI I removed the "Path to paging complete field" and "Path to paging complete values" in the paging section.

The image displays a user interface for configuring non-standard API response patterns, including fields for specifying the path to records and error values in the HTTP response body. (Captioned by AI)
The image displays a user interface for configuring non-standard API response patterns, including fields for specifying the path to records and error values in the HTTP response body. (Captioned by AI)902×343 14.9 KB

By the way, I've set the Page Size to 100 in the advanced settings.

And every time this export runs, it is only running for the first 100 users from the API

Let me know if you need anything.

8

Hey Tyler, I think I figured out what's happening wrong. Since I've set the path to records to "result" but the url I'm using has the skip parameter multiplied by 100. I previously did it to make sure that we skip the 100 records that we get. But it seems that Celigo is automatically incrementing skip by 100 already due to which my skip parameter was evaluated to 10000 for the second call due to which I wasn't getting any data. I fixed it and now I'm getting all the records as expected.

Appreciate your help, Thanks!!

9

Good catch, yes we increment for you :slight_smile:

1 Like