I am running a flow that is exporting csv files from a netsuite file cabinet. I have a filter that should be grouping records by Document ID, however the flow seems to be ignoring this and outputting two separate csv files with two different Document IDs into one singular JSON file. I cannot seem to configure around this. I just want to output one JSON file per csv file that is grabbed from the file cabinet.
On your export, set the page size to 1 as this will ensure that the records with the same document ID are grouped into a single "rows record," and then each page will have only one row record. From there, on your import, enable skip aggregation so that one file is made per page of data. This assumes each file only contains one document ID because if it includes multiple document IDs, then this will make a JSON file for each document ID, as opposed to one JSON file per CSV.
If you truly need one JSON file per CSV file, as opposed to one JSON file per document ID, then you’ll need to make an export that is a folder saved search that returns a list of all CSV files in the folder you want to export. From there, set your page size to 1. Then make a lookup to get the actual contents of the file and parse it as CSV. Response map the returned data to your main record. Then you get to your import step and again set skip aggregation to true. You should probably use mapper 2.0 as well to get it into the format you want. You can see this method here: Netsuite File Cabinet File .
By setting the page size to 1 on the import and then checking the “Skip Aggregation” on the export, it seems to now output separate json files. One issue I am having with this approach is that my input CSV files have multiple rows with the same Document ID and the output JSON only has the first row of each input CSV file. Is there a way to allow for the other rows to source through as well with this method?
I might be misunderstanding the usage, but it seems like I shouldn’t need to specify a page size as long as I am telling the import step to “group” by the values in the Document ID column. CSV file 1) has 3 rows all with the same Document ID that I want grouped together, CSV file 2) has 2 rows with the same Document ID’s but different from the value in file 1. If I am grouping by this value, shouldn’t the flow be outputting two pages?
This might be an issue with your setup. Are you using a mapper on your import? If so, what does it look like? I tested on my end and got all grouped items together in one file per document ID.
The default page size is 20. In your example, you grouped all the rows of file one into one record, and the second file also results in one record, so you have two records made: record: [{},{},{}] and record: [{},{}]. Since the page size is set to the default of 20, those two records are on the same page. By specifying a page size of one, they get on different pages.
The image below is all I have setup for parsing the input csv files. Besides that, I am not doing any mapping on this step in the flow. The preview data you see below contains one of the DocumentIDs:
This next image is from the Run History showing two separate CSV files were pulled. File 1 has the DocumentID you see in the preview above, while the second file has rows with a different DocumentID. As you can see, the flow consolidated the rows of both inputs into one output:
This output json file has the rows from both CSV files even though the DocumentIDs are different (DocumentID":"9928372" and "DocumentID":"838648M22")
Am I missing a param on the parser to ensure the DocumentIDs output to their own respective JSON file?
Do you have page size set to 1 on your export and skip aggregation set to true on your import?
I do have “Skip Aggregation” checked on my import. For the export “Page Size” if I set it to 1 then I get an output file per row on each of the input csv’s which is what I don’t want. If I set the page size to blank (default of 20) then the flow is consolidating all rows from both csv’s into one output despite the different DocumentID’s.
On your export you are sorting, not grouping. You should be grouping by document id and then page size to 1.