Paginated data not being passed through remainder of flow

[joshg]

Really hoping someone here can help me out with this, as I can't figure out why this is happening, and the Celigo support represent I've been working with via a support ticket can't figure it out either.

Background:

I have a flow step that performs a Shopify GraphQL query to get a variety of product data. The amount of data returned requires me to perform pagination due to rate limits (I query 100 products per call as you'll see below). The queries work great, and in the preview pane, I see all records, which make me think my pagination is working. Below are some screenshots on how I have this flow step configured with the pagination:

And these are the queries I have set in the above screenshots:

HTTP request body:

{
  "query": "query ($numProducts: Int!, $sku_query: String, $cursor: String) { productVariants (first: $numProducts, query: $sku_query, after: $cursor) { pageInfo { hasNextPage endCursor } edges { node { id sku legacyResourceId inventoryItem { id legacyResourceId inventoryLevels(first: 1) { edges { node { id available location { id } } } } } } } } }",
  "variables" : {
    "numProducts": 100,
    "cursor": null,
    "sku_query": "{{#each record.inventory_data as |inv_record|}}{{#compare @index ">" 0}} OR {{/compare}}sku:{{inv_record.sku}}{{/each}}"
  }
}

Override HTTP request body for subsequent page requests:

{{#if previous_page.full_response.data.productVariants.pageInfo.hasNextPage}}
{
  "query": "query ($numProducts: Int!, $sku_query: String, $cursor: String) { productVariants (first: $numProducts, query: $sku_query, after: $cursor) { pageInfo { hasNextPage endCursor } edges { node { id sku legacyResourceId inventoryItem { id legacyResourceId inventoryLevels(first: 1) { edges { node { id available location { id } } } } } } } } }",
  "variables" : {
    "numProducts": 100,
    "cursor": "{{{previous_page.full_response.data.productVariants.pageInfo.endCursor}}}",
    "sku_query": "{{#each record.inventory_data as |inv_record|}}{{#compare @index ">" 0}} OR {{/compare}}sku:{{inv_record.sku}}{{/each}}"
  }
}
{{/if}}

When I preview this data, this is how the records are displayed in the "Parsed output" tab on the preview pane:

{ "page_of_records": [ 
 { "record": 
   { "node": { [Shopify product data here] } }
 },
 { "record": 
   { "node": { [Shopify product data here] } }
 },
 { "record": 
   { "node": { [Shopify product data here] } }
 },
 { "record": 
   { "node": { [Shopify product data here] } }
 },
 [...rest of product records]
]}

In addition, when I set the debug logs on this flow step, I am seeing 8 different logs appear when I run the flow, which also makes me think that my pagination is working.

Issue:

On the Results Mapping associated with this flow step, the Preview pane is only showing me the first record. Thinking this may be a bug with the Preview, I created a Javascript hook that intentionally breaks so I can review the actual data that appears in the "Edit retry data" window of the error display, but that also only shows me the first record. Through the remainder of my flow (which is only one other flow step), I am also only seeing the first record being passed through.

That said, how can I get all the records from this paginated flow step to pass through the remainder of my flow?

[basvanditzhuijzen]

Hi Josh, a nice way to test if you pagination is working correctly is to set the pagesize to a low number, so in this case set "numProducts" to 2 on both query body's. Then click the preview button on the export. If the pagination is working correctly Celigo will do 5 calls, show 10 records on the Parsed Output tab, and show the last call (2 records) on the HTTP Response tab.

Even if pagination is working incorrectly I would not expect only the first record to be passed but the whole page, do you have any Result Mappings on the export?

[joshg]

Hi Bas, thank you for that. Yes I have tried that previously, and I believe it is working. When I set both queries to have numProducts of 2, in the Preview pane it shows 1 Page with 10 Records (since that is what I have set in the Preview area), the Parsed Output tab shows the 10 records, and the HTTP Response tab shows the last two records. Below is a screenshot of the Parsed Output tab and the HTTP Response tab:

Parsed Output:

HTTP Response:
I can’t show the two records in a full screenshot, but here are the two records output from that display:

{ "data": { "productVariants": { "pageInfo": { "hasNextPage": true, "endCursor": "eyJsYXN0X2lkIjo0NDQ4MDg0NjIwMTE0OSwibGFzdF92YWx1ZSI6IjQ0NDgwODQ2MjAxMTQ5In0=" }, "edges": [ { "node": { "id": "gid://shopify/ProductVariant/44480846102845", "sku": "AC-220-WTRF-NA-130", "legacyResourceId": "44480846102845", "inventoryItem": { "id": "gid://shopify/InventoryItem/46528478511421", "legacyResourceId": "46528478511421", "inventoryLevels": { "edges": [ { "node": { "id": "gid://shopify/InventoryLevel/110539079997?inventory_item_id=46528478511421", "available": 72, "location": { "id": "gid://shopify/Location/74161586493" } } } ] } } } }, { "node": { "id": "gid://shopify/ProductVariant/44480846201149", "sku": "AC-220-GYRF-NA-125", "legacyResourceId": "44480846201149", "inventoryItem": { "id": "gid://shopify/InventoryItem/46528478609725", "legacyResourceId": "46528478609725", "inventoryLevels": { "edges": [ { "node": { "id": "gid://shopify/InventoryLevel/110539079997?inventory_item_id=46528478609725", "available": 41, "location": { "id": "gid://shopify/Location/74161586493" } } } ] } } } } ] } }, "extensions": { "cost": { "requestedQueryCost": 14, "actualQueryCost": 14, "throttleStatus": { "maximumAvailable": 10000, "currentlyAvailable": 9986, "restoreRate": 500 } } } }

The only thing I'm not sure how to tell from the Preview section is that Celigo did 5 calls - I am only assuming that is the case. But when I set up the debug logger and actually run the flow, I see all the log files appear with the relevant data.

[joshg]

Any other ideas what the problem could be, or is this a bug in the platform?

I also tried using the "Next page token" pagination option, but that either gives me errors or I don't get any data beyond the first page. Per your documentation at https://docs.celigo.com/hc/en-us/articles/360059620151-API-pagination#Tokenpagingmethod, it says "For all path values related to pagination, verify that you are using a path that matches the record structure displayed in the HTTP Response preview section".

My HTTP Response for the first page looks like the following:

{
  "data": {
    "productVariants": {
      "pageInfo": {
        "hasNextPage": true,
        "endCursor": "eyJsYXN0X2lkIjo0NDQ4MDg0NTY3Njg2MSwibGFzdF92YWx1ZSI6IjQ0NDgwODQ1Njc2ODYxIn0="
      },

Which would make me think the following "Next page token" configuration would be correct, but it is throwing an error as seen in the preview section:

Any ideas on what this issue is?

[scotthenderson]

Can you change the following?

"Path to next page token field in HTTP response body *" -> data.productVariants.pageInfo.endCursor

"Path to paging complete field in HTTP response body" -> data.productVariants.pageInfo.hasNextPage

"Paging complete values" -> false

[joshg]

Thanks for that info, I tried that but still get the same error:

This makes some sense to me though since my initial query passes in the "cursor" parameter of null, so I would assume it is trying to use null for both paginated calls. That said, I tried removing all cursor references from my initial graphQL query, but I still get the same "page_response_error" code shown in the Preview above.

[joshg]

Quick update - I believe I got the pagination to work via the "Next Page Token" Paging method, but I am still running into the original issue I mentioned on this ticket where the paginated results are not being passed down the flow.

For anyone else running into this pagination issue using the "Next Page Token" paging method, I made the updates Scott mentioned in the previous comment, and then updated my original graphQL query above to add some conditional logic to the "cursor" variable (which I discovered via some documentation). Instead of having "cursor": null, I changed it to:

"cursor": {{#if export.http.paging.token}}"{{export.http.paging.token}}"{{else}}null{{/if}}

Back to my original problem, I only have one other flow step that runs after this paginated data flow step. For reference, this is how the flow is structured (the flow step on the left is where I am paginating all the data, and the hook just performs some calculations and adds new data to be passed down to the final flow step):

The odd thing is when I look in that response mapping node, it only ever shows the first record when I click Preview. For reference, this is what the mapping looks like showing only one record being returned:

Evidence of the pagination appearing to work is in the following. These are the logs that are generated from the "Shopify (GraphQL) Get Product and Inventory Location Data by SKU" step:

And then there is only one log generated from the “Shopify (GraphQL) Update Inventory on Product Variant” step, which only seems to be processing a single record (even though the query loops through an array of objects, and only a single object is included in the array that is being processed in this step):

Any idea why all the paginated data is not being passed over to my final flow step?

[joshg]

Shoot, after I pasted my screenshot of the response mapping, I noticed I was using data[0] instead of data[*]. Changing that allowed all the results to be passed through the remainder of the flow.

Everything is now working as expected. Thank you all for your help with this.

[Maureen_Pozen]

Awesome notes on this @joshg, thank you, this was really helpful!
One addition (not sure if this has changed with more recent updates of Celigo)...
You've gotta escape the inner quotes in the JSON. Proper way to do that is \".

It took me a little hunting to figure this out. So, if you're following Josh's example, just be aware that your cursor line should look like this (add the back slashes)

"cursor": {{#if export.http.paging.token}}\"{{export.http.paging.token}}\"{{else}}null{{/if}}

[davidgollom]

I got some tips from this post too! Thanks to all.

[tylerlamparter]

I want to add to this post a bit since we're all being forced into GraphQL . I'm also curious where you all are having to escape quotes because non-escaped quotes are working fine for me.

GraphQL connector setup

If you're using the GraphQL connector we have, you would have a setup similar to what I have below in order to paginate on your export. Our branded Shopify connector will have this better GraphQL editor baked into it in the next release as well, so this will be more relevant.

Query:

query getProductVariants($first: Int, $after: String) {
  productVariants(first: $first, after: $after) {
    edges {
      node {
        id
        title
        sku
        price
        compareAtPrice
        inventoryQuantity
        availableForSale
        updatedAt
        product {
          id
          title
        }
        selectedOptions {
          name
          value
        }
        image {
          originalSrc
          altText
        }
      }
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

Variables:

{ 
  "first":200,
  "after": {{#if export.http.paging.token}}"{{export.http.paging.token}}"{{else}}null{{/if}}
}

GraphQL setup using HTTP connector or Shopify in HTTP form

If using the existing Shopify connector or the normal HTTP connector, your setup would be something more like this.

Post body

{
  "query": "query getProductVariants($first: Int, $after: String) { productVariants(first: $first, after: $after) { edges { node { id title sku price compareAtPrice inventoryQuantity availableForSale updatedAt product { id title } selectedOptions { name value } image { originalSrc altText } } } pageInfo { hasNextPage endCursor } } }",
  "variables": { "first": 2, "after": {{#if export.http.paging.token}}"{{export.http.paging.token}}"{{else}}null{{/if}} }
}

[Matthew_Lacey]

Thanks for this Tyler! I ran into problems on a custom Shopify flow today, due to the 250 variants limit and I have very little knowledge/experience when it comes to queries and API etc, so was a bit stuck when it came to setting up the pagination. I managed to eventually get it all working though, thanks to the instructions in this thread.