NetSuite Item Load Does Not Return All of the Items

Integrator.io
, , , ,
1

I have an export from NetSuite that is setup to pull all the items from the item database. In the sandbox, there are about 140k items in total. My "preSavePage" hook filters out some of the items such as UPC Codes with alpha-numeric characters and with a total length of over 14 positions. This reduces the total number of items to about 120k. I have taken the same script and downloaded the NetSuite saved search (same as in the export) csv file and run this exact same script and it returns around the 120k mark of items. I also have the export setup to batch size and page size set to 1,000 (and reduced it to 500 and it processed slightly more records). The script runs with no errors. But it stops or "completes" at about 33 pages at the 1,000 page size and 98 at the 500 page size. All of the items are correctly moved over to the import and processed properly.

I get no page size errors or any other errors when the flow runs. It just completes without getting all the records. Here is the javascript routine that "filters" the item records coming in:


const regex = /[a-zA-Z\'\s+]/;
let lastUPC=' ';
let items = [];
let coitems = [];
let zero = "0";
let upcremoved = 0;
let toobig = 0;
let dups = 0;
for (let i=0; i < options.data.length; i++){
// NaN test
if(regex.test(options.data[i]['UPC Code'])){
console.log(options.data[i]['UPC Code']);
options.data.splice(i,1);
upcremoved ++;
continue;
}
if (options.data[i]['UPC Code'] > 99999999999999){ //14 digits are under
options.data.splice(i,1);
toobig ++;
continue;
}
if(i>0){
if (options.data[i]['UPC Code'] === lastUPC){ // compare for duplicate
options.data.splice(i,1);
dups ++;
continue;
}
}
items.push(options.data[i]);
lastUPC = options.data[i]['UPC Code'];
}
console.log('Char UPC removed ' + upcremoved);
console.log('Big UPC removed ' + toobig);
console.log('Dupe UPC removed ' + dups);

This leaves an array of records that is handed off to the formatting for the outgoing API call in the import section. There are no errors in the import process and the final database has all the items that have been processed. I capture the incoming stream from NetSuite and it is pulling them correctly... it's like it just doesn't finish! Anyone with some suggestions would be greatly appreciated.

2

Hi PaulS

I see a small bug with your script, in the loop you're moving the array index from 0 to data.length, and in the loop you're splicing it.

Splice operation at index i (say 4 for eg) will change the length of the array, and bring a new element at i, i+1 moved to i, (5th element moves to 4), but your i++ will still move the i to i+1. So you will end up skipping one element each time you splice something.

You should i--, after splice at i.

That being said, I can not understand why the flow does not process all the exported records and not show any errors too. Please fix the script and re-run it on the NetSuite saved-search data to see what is the new filtered count.

I suggest you raise a support ticket as we'll need a closer look, so as soon as you've fixed the script and know the new filtered count, please run the flow and reach out to Celigo support if the issue still persists.

Thanks

3

Thanks for the catch - I'll make the change and try again. I had that decrement in there before but took it out as it didn't seem to act correctly for what I was doing.

4

I made the correction and reran the flow. Still only 50k out of 150k were moved over. I adjusted the page size and batch size to see if that would make a difference and it did not. There are no errors in the flow. The time for it to run varies between a little over 1 hour to almost 2 hours depending on the page & batch size. I am running this against the NetSuite sandbox, but I see no errors there either. Could there be some ceiling I'm hitting on NetSuite that stops the load? And I created another flow that used FTP to send the saved search with all the items in it to a FTP server... and it timed out and sent no file...

5

Hi PaulS,

Please raise a support ticket, as we'll need to take a closer look. Since this is not completing the export and not generates an error too, we'll have to look at the debug logs, and the jobs in question.

Please use support medium.