How to handle this pagination?

Integrator.io
1

Hi Everyone,

I’m trying to see if there’s a way to handle this pagination. I’m trying to extract payment data in Walmart. (Here is the doc to the API).

Below is the get URI and the paging configuration.

GET

/report/reconreport/reconFileJson?reportDate=test&offset={{export.http.paging.token}}&noOfRecords=1000

image
image780×627 56.2 KB

In the connection debug logs, I can see that I am able to go through the records up until the last page.

1st Request:
{"url":"``https://marketplace.walmartapis.com/v3/report/reconreport/reconFileJson?reportDate=test&offset=0&noOfRecords=1000","method":"GET","headers":{"authorization":"Basic``test==","wm_svc.name":"Walmart Marketplace","wm_qos.correlation_id":"test","wm_consumer.channel.type":"test","WM_SEC.ACCESS_TOKEN":"********","accept":"application/json"}}

The response would have this field in the body called nextOffset with a value of 380098 and would be used on the 2nd request below

{"url":"``https://marketplace.walmartapis.com/v3/report/reconreport/reconFileJson?reportDate=test&offset=380098&noOfRecords=1000","method":"GET","headers":{"authorization":"Basic``test==","wm_svc.name":"Walmart Marketplace","wm_qos.correlation_id":"test","wm_consumer.channel.type":"test","WM_SEC.ACCESS_TOKEN":"********","accept":"application/json"}}

When I reach the last page, the nextOffset field will contain a value of -1. However, even though I have specified the field and value in the “Path to paging complete field in HTTP response body and Paging complete values”. It still tries to make a call like below which would then error the flow step.

{"url":"``https://marketplace.walmartapis.com/v3/report/reconreport/reconFileJson?reportDate=test&offset=-1&noOfRecords=1000","method":"GET","headers":{"authorization":"Basic``test==","wm_svc.name":"Walmart Marketplace","wm_qos.correlation_id":"test","wm_consumer.channel.type":"test","WM_SEC.ACCESS_TOKEN":"********","accept":"application/json"}}

Hope my explanation makes sense. I appreciate any inputs

Thank you!

2

We do have a Path to total number of results field in HTTP response body on the "Skip number parameter" option, but Walmart is being different here and returning skip bytes instead of a skip number :confused:. So given that, you probably have a couple options until we can enhance our paging mechanisms.

Couple options:

  • Build a custom relative uri in the Override relative URI for subsequent page requests field. With this, you get access to the prior full response so you're able to do things like {{compare nextOffset == "-1"}}999999999999{{else}}{{nextOffset}}{{/compare}}. I'm assuming 999999999999 would return an empty array of data, but maybe it just errors. Maybe it also returns a -1, which makes this scenario go in a loop and not work.
  • Make a preSavePage script to ignore the error: https://docs.celigo.com/hc/en-us/articles/29207822102043-Handling-errors-in-lookups-with-PreSavePage-scripts
3

Thank you! @tylerlamparter I appreciate the help.

I’ll try out the suggested options