Append new rows in bulk, missing some records

Integrator.io
,
1

I am using the Append new rows in bulk export from the Marketplace to write data to an Excel file in a SharePoint location.

I am exporting data from four different sources of an HR application. When checking the results, I have 857 of the 884 expected records. I am not sure why some of my records are not importing so I ran a test limiting to just one of the 27 records that did not write (using a branch filter) and, oddly enough, the single record wrote to the Excel file.

This is strange in that if I run the entire process in bulk, reading from the four different sources, 27 of the records are being left out. But if I run just one of the records left out, then it shows up.

It makes me wonder, could there be an issue with the paging features? I'm not 100% certain, but I thought I counted a total of 27 pages in the export - which could coincide with 27 missing records.

I'm having a hard time determining why these records would fail, but when isolated, could later write to the record as expected.

Any thoughts as to what might be happening here?

Thank you for your time,

-mitchel

2

Bumping this thread. Anyone have any thoughts? Any changes to the Marketplace flow, I should pull it down and try again?

-mitchel

3

Hey Mitchel, I'm not able to reproduce this issue. Here, I tested with 6,486 records and it all got in. I would suggest setting the page size on your export to a high number because it will result in fewer API calls to Excel. In my example, I set the page size to 10,000.

A Microsoft Excel spreadsheet displaying inventory data, including fields like record type, internal ID, name, and SKU. (Captioned by AI)
A Microsoft Excel spreadsheet displaying inventory data, including fields like record type, internal ID, name, and SKU. (Captioned by AI)1920×1153 207 KB
The image shows a workflow displaying successful data transfer from Oracle NetSuite to a spreadsheet, with 6,486 records processed without errors. (Captioned by AI)
The image shows a workflow displaying successful data transfer from Oracle NetSuite to a spreadsheet, with 6,486 records processed without errors. (Captioned by AI)1902×1238 169 KB

4

Thanks for the effort on this, Tyler. You've given me an idea...

I think the export is limited on the page size, so I might not be able to bump up the export so much. But what I can try is: lowering the limit (say by half). If my number of missing records doubles, then we know it has something to do with paging.

I'll try to run a test later today and post the result here. If it still seems wonky, I'll jump into Office Hours on Tuesday and share the details of the flow.

Thanks as always for the assistance. More to follow.

-mitchel

5

Closing the loop on this thread, Tyler.

After additional testing on our end, I believe we are pulling in all of the records from the export. All seems to be working at this moment.

Thanks again for all the help, it's always appreciated.

-mitchel