Using exports.runVirtualWithPaging(options) in a custom form script

Custom Forms are a powerful feature in integrator.io which can be used throughout many integration resources. With Manager Access, you can build forms within integrations, flows, connections, exports and imports by going to “Settings” (at the integration level) or “Custom settings” and selecting “Launch form builder”. From here, you can build a form defined by JSON objects or use a script. You can see many examples of the objects used in the form by going to Tools>Playgound and select Form builder: Field dictionary.

One of the most popular ways to use Custom Forms is creating dynamic mapping tables to define relationships from one endpoint to another (NetSuite Locations to Shopify Locations) or by using a dynamic dropdown list to define a default item to use if certain criteria is met. Both of these scenarios require the use of a Virtual Export within the form definition to dynamically populate the list. This is most commonly used in the “exportSelect” field type. Building a dynamic select list using the JSON object form definition is the most popular and recommended way, and will suffice for nearly all use cases. However, at the time of writing, a virtual export loaded into an “exportSelect” will not display more than 100 results, limiting the available options for that field to 100. In these cases, one must create the form using JavaScript.

integrator.io API supports JavaScript runtime objects and can use the runVirtual(options) function to run a virtual export from within JavaScript. This can be used to populate a “select” type form field in a custom form with values returned from the virtual export. However, if the API endpoint requires paging, you must use the runVirtualWithPaging(options) function.

exports.runVirtualWithPaging( options )

Runs a virtual export using an existing connection, and returns one page of data at a time. The function takes the following arguments:

  • options
    • “options” is an object with the following properties:
      • export
        • required
        • Type: Object
        • The virtual export definition
      • pagedExportState
        • Type: Object
        • the state object that was returned (as-is) by the last call to this function
      • startDate
      • endDate
        • Type: Date
        • end date for delta exports as a Date object
    • example:
options: {
  export: {
    _connectionId: "alias_connection_shopify_graph",
    asynchronous: true,
    http: {
      method: "POST",
      body: "{\"query\":\"{\\n  inventoryItems(first: 10, after: {{#if export.http.paging.token}}\\\"{{export.http.paging.token}}\\\"{{else}}null{{/if}},) {\\n    edges {\\n    cursor\\n      node {\\n        id\\n        sku\\n      }\\n    }\\n    pageInfo {\\n      hasNextPage\\n      startCursor\\n      endCursor\\n    }\\n  }\\n}\"}",
      isRest: false,
      formType: "graph_ql",
      paging: {
        method: "token",
        path: "data.inventoryItems.pageInfo.endCursor",
        pathLocation: "body",
        lastPagePath: "data.inventoryItems.pageInfo.hasNextPage",
        lastPageValues: [
          "false"
        ]
      }
    }
  },
  pagedExportState: { 
      pageSize: 1, 
      recordCount: 1, 
      pageIndex: 1, 
      nextPageURL: "https://base.uri.com/admin/api/2025-01/graphql.json", 
      nextPageBody: "{\"query\":\"{\\n inventoryItems(first: 10, after:\"cursor\",) {\\n edges {\\n cursor\\n node {\\n id\\n sku\\n }\\n }\\n pageInfo {\\n hasNextPage\\n startCursor\\n endCursor\\n }\\n }\\n}\"}", 
      done: false, 
      issuedAt: 1761155820, 
      sig: "0d37f339d62ae1803f046d24b251aa204cfb69b11acba8d7f038a41021a95f1a"
    
  }
}

The “pagedExportState” object contains the instructions on how to call the next page, and will be different depending on the paging method used in the virtual export object and the endpoint being called.

The function returns the following object:

  • data
    • Type: Array
    • the data that was exported
  • dataURIs
    • Type: Array
      • matching URI array
      • dependant on the virtual export configuration
  • pagedExportState
    • Type: Array
    • The state object to send (as-is) to the next call to the function
  • errors
    • Type: Array
    • any errors returned by the export

If the paging on the virtual export object is set up correctly, the pagedExportState object can be used to call the function as many times as needed (using a loop) until the pagedExportState.done value is “true”.

Creating the form

From the form builder screen, toggle to JavaScript, create a new script and insert the function stub.

/*
* formInit function stub:
*
* The name of the function can be changed to anything you like.
*
* The function will be passed one 'options' argument that has the following fields:
*   'resource' - the resource being viewed in the UI.
*   'parentResource' - the parent of the resource being viewed in the UI.
*   'grandparentResource': the grandparent of the resource being viewed in the UI.
*   'license' - integration apps only.  the license provisioned to the integration.
*   'parentLicense' - integration apps only. the parent of the license provisioned to the integration.
*   'sandbox' - boolean value indicating whether the script is invoked for sandbox.
*
* The function needs to return a form object for the UI to render.
* Throwing an exception will signal an error.
*/
function formInit (options) {
  return options.resource.settingsForm.form
}

At the top of the script, import the exports module from the integrator api.

import { exports } from 'integrator-api';

This will allow us to use the “Exports'“ JavaScript runtime Object in the script.

Next, we will update the formInit(options) function to directly create our forms as we would if using JSON definitions for brevity.

import { exports } from 'integrator-api';
/*
* formInit function stub:
*
* The name of the function can be changed to anything you like.
*
* The function will be passed one 'options' argument that has the following fields:
*   'resource' - the resource being viewed in the UI.
*   'parentResource' - the parent of the resource being viewed in the UI.
*   'grandparentResource': the grandparent of the resource being viewed in the UI.
*   'license' - integration apps only.  the license provisioned to the integration.
*   'parentLicense' - integration apps only. the parent of the license provisioned to the integration.
*   'sandbox' - boolean value indicating whether the script is invoked for sandbox.
*
* The function needs to return a form object for the UI to render.
* Throwing an exception will signal an error.
*/
function formInit(options) {
  options.resource.settingsForm.form = {
    fieldMap: {
      ShopifyItems: {
        id: "ShopifyItems",
        name: "shopifyItems",
        label: "Select Shopify Items",
        type: "select",
        options: createOptions(options)
      }
    },
    layout: {
      fields: ["ShopifyItems"]
    }
  }
  
  return options.resource.settingsForm.form
}

Notice that the type is not “exportSelect” but is now “select”. This is because we are going to run the call the virtual export, and update the “options” array for the field with the results returned using the “arrays of labels and values” option as explained here. To do this, we assign the “options” value the result of the funcion createOptions(options).

Next, create the createOptions function and its variables.

function createOptions(options) {

  // The array of items to return to the calling function
  let items = [];

 // The controller to use for the do while function       
  let loopControl = false;

 // an uninitialized variable which will be used to hold the request object
  let request;              
  
  return [{
    items: items
  }];
}

The “items” array will contain all of the concatenated data returned from the lookup, and will be what is returned from the function.

“loopControl” is a boolean, initialized to false, which will be used as our controller for the do-while loop, which is the main driver of our script. It is important to highlight that using a do-while loop can cause infinite looping if the condition is never met, and should only be used when necessary. Another method that can be used is a generator function with a yield.

“request” remains uninitialized and will be used to build our request object from the “pagedExportState” that is returned by the runVirtualWithPaging function.

do-while loop

The main driver will be our do-while loop. This will call the paging function, push the data to our items array, and check if the paging values contain instructions to call the next page, or if the paging is complete. If the paging is complete, set the “loopControl” variable to true, at which point the items object is returned to the form.

It is recommended to start these steps outside of the loop and use console.log() to see the data being returned at each step to better understand how to process it.

First, create a variable to hold the response from calling the paging function. What is sent to the function will depend on the current value of “request”. Remember, the first time this runs, “request” will have the value undefined in which case we want to send the “options” argument, which is the data sent to every entry function within integrator.io.

let response = request ? runVirtualExportWithPaging(request).body : runVirtualExportWithPaging(options).body;

We set the value of “response” to body returned from the function. The JavaScript Ternary Operator is used to check if “request” contains a value, and if it does, pass “request” to the function, otherwise pass “options”. We will go over the runVirtualExportWithPaging function later, but know that it returns an object with the following structure:

{
   statusCode: response.statusCode,
   headers: {},
   body: response.body
}

This is important for implementing error handling if needed.

Once the function returns and has a statusCode of 200, we need variables to hold the data. Add the following:

// destructure the response
let {
  data,
  pagedExportState
} = response;
// assign a value to 'request' which will be used as the new request object
request = {
  body: response
}

JavaScript Desctructuring is used to extract “data” and “pagedExportState” from the response object. This is the same as saying:

let data = response.data;
let pagedExportState = response.pagedExportState;

Here we also assign an object value to “request”. This object contains the entire response object, and will now notify our next run of the loop to send “request” instead of “options” to the function.

The value of “data” will be different depending on the endpoint and the virtual export object. It is important to review the value returned to better understand how to handle it in the next code block. Here, the data being returned from my export has this following structure:

[
  {
    "data": {
      "inventoryItems":{
        "edges":[
          {
            "cursor":"eyJsYXN0X2lkIjo0NzEwNjkyMDQ0ODE5NiwibGFzdF92YWx1ZSI6IjQ3MTA2OTIwNDQ4MTk2In0=",
            "node":{
              "id":"gid://shopify/InventoryItem/47106920448196",
              "sku":"testitem123"
            }
          },
          {
            "cursor":"eyJsYXN0X2lkIjo0NzEwNjkyMDQ4MDk2NCwibGFzdF92YWx1ZSI6IjQ3MTA2OTIwNDgwOTY0In0=",
            "node":{
              "id":"gid://shopify/InventoryItem/47106920480964",
              "sku":"testitem124"
            }
          },
        ],
        "pageInfo":{
          "hasNextPage":false,
          "startCursor":"eyJsYXN0X2lkIjo0NzEwNjkyMDQ0ODE5NiwibGFzdF92YWx1ZSI6IjQ3MTA2OTIwNDQ4MTk2In0=",
          "endCursor":"eyJsYXN0X2lkIjo0NzE2MTcxNzY4NjQ2OCwibGFzdF92YWx1ZSI6IjQ3MTYxNzE3Njg2NDY4In0="
        }
      }
    },
    "extensions":{
      "cost":{
        "requestedQueryCost":6,
        "actualQueryCost":4,
        "throttleStatus":{
          "maximumAvailable":2000,
          "currentlyAvailable":1996,
          "restoreRate":100
        }
      }
    }
  }
]

In this case, in order to extract the id and sku of each item, we want to loop through “data” and for each object, loop through the array at the path “data.inventoryItems.edges”. Add the following code:

data.forEach((d) => {
   // the path to the records
   let records = d.data.inventoryItems.edges;
   for(let item of records) {
     items.push({
       label: item.node.sku,
       value: item.node.id
     })
   }
 })

Again, the exact implementation of this code block with be dependent on the endpoint being called and the virtual export object configuration.

Next, we want to check the “pagedExportState” object to see if its property “done” is true or false but setting the corresponding value to “loopControl”.

loopControl = pagedExportState.done ? true : false;

Once thoroughly tested and complete understanding of the data being sent and returned has been established, put each of the code blocks inside a do-while loop within the function. The complete createOptions function now looks like this:

function createOptions(options) {
  
  let items = [];           // The array of items to return to the calling function
  let loopControl = false;  // The controller to use for the do while function
  let request;              // an uninitialized variable which will be used to hold the request object
  
  /**
   * Begin the loop.
   * At the start of the loop, 'request' contains no value, and the function 'runVirtualExportWithPaging' is run with the 'options' argument.
   * 
   */
  do {
    let response = request ? runVirtualExportWithPaging(request).body : runVirtualExportWithPaging(options).body;
     
    // destructure the response
    let {
      data,
      pagedExportState
    } = response;
    // assign a value to 'request' which will be used as the new request object
    request = {
      body: response
    }
    
    data.forEach((d) => {
      // the path to the records
      let records = d.data.inventoryItems.edges;
      for(let item of records) {
        items.push({
          label: item.node.sku,
          value: item.node.id
        })
      }
    })
    loopControl = pagedExportState.done ? true : false;
  } while(!loopControl);
  
  return [{
    items: items
  }];
}

runVirtualExportWithPaging(options)

This function is a wrapper for the runVirtualWithPaging(options) function, and can be called anything. It simply builds the export object, checks if the argument contains a “pagedExportState” object, and then calls the runVirtualWithPaging function, returning the value in the format specified earlier: an object with the properties “statusCode”, “headers”, and “body”.

Create the function and its variables:

function runVirtualExportWithPaging(options) {
  let exportObject; // the export object
  let invokeExportResponse; // the variable to hold the response from the function
  let response = {}; // the response that is used to send back to the calling function
  
  exportObject = {
          "_connectionId": "alias_connection_shopify_graph",
          "asynchronous": true,
          "http": {
            "method": "POST",
            "body": "{\"query\":\"{\\n  inventoryItems(first: 10, after: {{#if export.http.paging.token}}\\\"{{export.http.paging.token}}\\\"{{else}}null{{/if}},) {\\n    edges {\\n    cursor\\n      node {\\n        id\\n        sku\\n      }\\n    }\\n    pageInfo {\\n      hasNextPage\\n      startCursor\\n      endCursor\\n    }\\n  }\\n}\"}",
            "isRest": false,
            "formType": "graph_ql",
            "paging": {
              "method": "token",
              "path": "data.inventoryItems.pageInfo.endCursor",
              "pathLocation": "body",
              "lastPagePath": "data.inventoryItems.pageInfo.hasNextPage",
              "lastPageValues": [
                "false"
              ]
            }
          }
        }
}

“exportObject” is the virtual export object that eventually gets assigned the resource definition. For more information on how to create these objects, please see the Invoke a JavaScript API Virtual Export article, as well as the examples provided in Common Form Fields for Dynamic List, Refreshable dropdown list, and refreshable mapping.

It is important to notice that there must be a paging property in the object, and that there is no need to use a transform and a responsePath property as this can be handled in our code. Here, the responsePath property was left out so that the “pagedExportState” object is correctly able to handle the pagination. Also notice for the “_connectionId”, we are using an alias. Alias’s are very helpful when creating forms and scripts which reference integration resource Id’s as they allow for a single point of reference for that Id. For more information on creating and referencing an alias in this context, see here.

Next, we need to consider a small bit of error handling, as we want to be able to convey to the calling function if the export was successful. To do this, add a try…catch statement to the code, and inside the try block enter the code block shown.

try {
    const pagingObj = {
      export: exportObject
    }
    
    if(options.body && options.body.pagedExportState) {
      // check for the pagedExportState field in options
      pagingObj.pagedExportState = options.body.pagedExportState
    }
    invokeExportResponse = exports.runVirtualWithPaging(pagingObj);
    response.statusCode = 200;
  } catch(e) {
    invokeExportResponse = JSON.stringify(e);
    response.statusCode = 400;
  }

The “pagingObj” is created and used to hold the exportObject and is also given the “pagedExportState” object if it exists in the “options” argument. Remember, the “options” argument contains the “pagedExportState” paging information from the previous calls, and must be sent as-is to each subsequent call to runVirtualWithPaging. The only time “pagedExportState” will not be in “options” is when the code first executes and “request” is undefined in the calling function.

Call the function and assign its result to “invokeExportResponse”. This is where we use the “exports” module that was imported in the first line of code of this script, and allows us to finally call the runVirtualWithPaging function.

IMPORTANT: At this point, if you are building the script in the form builder, an error will be returned stating “Acess restricted”, or will be held in the results in “invokeExportResponse”. This is because this function cannot work in preview mode. When the script is saved and closed and you navigate to the form in the UI, this error should not be returned.

Should an error occur, the “catch” block will execute. The error (held in “e”) will be passed to “invokeExportResponse” and the response.statusCode will be set to 400.

Outside of the try…catch block, set response.body to “invokeExportResponse” and set the object to return like so:

// create body response
  response.body = invokeExportResponse;
  return {
    statusCode: response.statusCode,
    headers: {},
    body: response.body
  }

The runVirtualExportWithPaging function should now look like this:

function runVirtualExportWithPaging(options) {
  let exportObject; // the export object
  let invokeExportResponse; // the variable to hold the response from the function
  let response = {}; // the response that is used to send back to the calling function
  
  exportObject = {
          "_connectionId": "alias_connection_shopify_graph",
          "asynchronous": true,
          "http": {
            "method": "POST",
            "body": "{\"query\":\"{\\n  inventoryItems(first: 10, after: {{#if export.http.paging.token}}\\\"{{export.http.paging.token}}\\\"{{else}}null{{/if}},) {\\n    edges {\\n    cursor\\n      node {\\n        id\\n        sku\\n      }\\n    }\\n    pageInfo {\\n      hasNextPage\\n      startCursor\\n      endCursor\\n    }\\n  }\\n}\"}",
            "isRest": false,
            "formType": "graph_ql",
            "paging": {
              "method": "token",
              "path": "data.inventoryItems.pageInfo.endCursor",
              "pathLocation": "body",
              "lastPagePath": "data.inventoryItems.pageInfo.hasNextPage",
              "lastPageValues": [
                "false"
              ]
            }
          }
        }
  // execute the export
  try {
    const pagingObj = {
      export: exportObject
    }
    
    if(options.body && options.body.pagedExportState) {
      // check for the pagedExportState field in options
      pagingObj.pagedExportState = options.body.pagedExportState
    }
    invokeExportResponse = exports.runVirtualWithPaging(pagingObj);
    response.statusCode = 200;
  } catch(e) {
    invokeExportResponse = JSON.stringify(e);
    response.statusCode = 400;
  }
  
  // create body response
  response.body = invokeExportResponse;
  return {
    statusCode: response.statusCode,
    headers: {},
    body: response.body
  }
}

The data in the return body is then processed by the code block in createOptions, pushed to the items array, and returned to the “options” property in our form field. The entire script should look something like this:

import { exports } from 'integrator-api';

function formInit(options) {
  options.resource.settingsForm.form = {
    fieldMap: {
      ShopifyItems: {
        id: "ShopifyItems",
        name: "shopifyItems",
        label: "Select Shopify Items",
        type: "select",
        options: createOptions(options)
      }
    },
    layout: {
      fields: ["ShopifyItems"]
    }
  }
  
  return options.resource.settingsForm.form
}

function createOptions(options) {
  
  let items = [];           // The array of items to return to the calling function
  let loopControl = false;  // The controller to use for the do while function
  let request;              // an uninitialized variable which will be used to hold the request object
  
  /**
   * Begin the loop.
   * At the start of the loop, 'request' contains no value, and the function 'runVirtualExportWithPaging' is run with the 'options' argument.
   * 
   */
  do {
    let response = request ? runVirtualExportWithPaging(request).body : runVirtualExportWithPaging(options).body;
     
    // destructure the response
    let {
      data,
      pagedExportState
    } = response;
    // assign a value to 'request' which will be used as the new request object
    request = {
      body: response
    }
    
    data.forEach((d) => {
      // the path to the records
      let records = d.data.inventoryItems.edges;
      for(let item of records) {
        items.push({
          label: item.node.sku,
          value: item.node.id
        })
      }
    })
    loopControl = pagedExportState.done ? true : false;
  } while(!loopControl);
  
  return [{
    items: items
  }];
}
function runVirtualExportWithPaging(options) {
  let exportObject; // the export object
  let invokeExportResponse; // the variable to hold the response from the function
  let response = {}; // the response that is used to send back to the calling function
  
  exportObject = {
          "_connectionId": "alias_connection_shopify_graph",
          "asynchronous": true,
          "http": {
            "method": "POST",
            "body": "{\"query\":\"{\\n  inventoryItems(first: 10, after: {{#if export.http.paging.token}}\\\"{{export.http.paging.token}}\\\"{{else}}null{{/if}},) {\\n    edges {\\n    cursor\\n      node {\\n        id\\n        sku\\n      }\\n    }\\n    pageInfo {\\n      hasNextPage\\n      startCursor\\n      endCursor\\n    }\\n  }\\n}\"}",
            "isRest": false,
            "formType": "graph_ql",
            "paging": {
              "method": "token",
              "path": "data.inventoryItems.pageInfo.endCursor",
              "pathLocation": "body",
              "lastPagePath": "data.inventoryItems.pageInfo.hasNextPage",
              "lastPageValues": [
                "false"
              ]
            }
          }
        }
  // execute the export
  try {
    const pagingObj = {
      export: exportObject
    }
    
    if(options.body && options.body.pagedExportState) {
      // check for the pagedExportState field in options
      pagingObj.pagedExportState = options.body.pagedExportState
    }
    invokeExportResponse = exports.runVirtualWithPaging(pagingObj);
    response.statusCode = 200;
  } catch(e) {
    invokeExportResponse = JSON.stringify(e);
    response.statusCode = 400;
  }
  
  // create body response
  response.body = invokeExportResponse;
  return {
    statusCode: response.statusCode,
    headers: {},
    body: response.body
  }
}

3 Likes