API query parameters are not available in downstream steps, and native paging can exceed the API response-size limitAPI query parameters are not available in downstream steps, and native paging can exceed the API response-size limit

[nuriensing]

I have built an API with the following structure:

1. An incoming GET /pricelist request.
2. A Microsoft Business Central lookup for Germany.
3. A Microsoft Business Central lookup for Iberia.
4. A Microsoft Business Central lookup for Italy.

The API request defines three optional query parameters:

GMBH_NextLink
IBERIA_NextLink
ITA_NextLink

The caller uses these parameters to send the continuation token for each Business Central environment.

Issue 1: Query parameters are not available in later steps

When I test a transformation on the initial API request step, I can see the query parameters:

{
  "record": {},
  "testMode": true,
  "queryParams": {
    "GMBH_NextLink": "sample-value",
    "IBERIA_NextLink": "sample-value",
    "ITA_NextLink": "sample-value"
  },
  "export": {
    "name": "Source API"
  }
}

However, when I test a transformation on the next Business Central lookup step, the queryParams object is no longer available:

{
  "record": {},
  "export": {
    "name": "Business Central lookup"
  },
  "testMode": true,
  "settings": {
    "connection": {},
    "iClient": {},
    "export": {}
  }
}

I also added a transformation to the initial request step to copy the query parameters into a field on the record. I then returned the complete JSON in the API response so I could inspect all available data.

However, the record remained empty and the queryParams object was still missing:

{
  "record": {},
  "settings": {
    "export": {}
  },
  "job": {
    "type": "export",
    "parentJob": {
      "type": "flow"
    }
  },
  "headers": {}
}

The same issue occurs when I use only one query parameter, so it does not appear to be caused by having multiple parameters.

The intended pagination flow is:

1. The caller requests the first page.
2. Celigo calls the three Business Central environments.
3. Celigo returns the records and the three continuation tokens.
4. The caller sends those tokens back as query parameters in the request URL.
5. Celigo uses each token to retrieve the next page from the corresponding Business Central environment.

The caller can send the query parameters, but I cannot access their values in the downstream Business Central lookup steps.

Is it correct that the caller should send these values as query parameters in the URL? If so, how can I access those query parameters in the downstream steps?

Issue 2: Native pagination can exceed the API response-size limit

The Business Central lookup step also includes Celigo’s native paging option under:

Does this API use paging?

This appears to make Celigo continue retrieving pages until all records have been collected.

However, this does not work well for large datasets. If Celigo retrieves too much data, the final API response can exceed the platform’s response-size limit. In our testing, this results in an HTTP 500 error.

For this use case, I need the API to return only one page at a time, together with the continuation token. The caller can then send the token back to request the next page.

Is there a supported way to return one page at a time instead of having Celigo retrieve all pages internally?

[sreevaniamara]

Hi @nuriensing ,

Issue 1:
I was not able to reproduce the issue.
A transformation needs to be added to make the query params available downstream. Without it, the query params will not be available in downstream steps.

I added a transformation to the initial request step to copy the query parameters into a field on the record which returned the complete JSON in the API response.

Post that in the downstream lookup step I was able to view the query params.

Can you please share more details so we can dig further into what might be different in your setup?

Issue 2:
We are working on an enhancement for this. In the meantime, we'll check if there's a workaround available and share it with you here.

[nuriensing]

Thanks! It was a bit unclear that you first need to add the queryParams back to the record using a transformation, but it worked.