Strange Issue with 'Each' Loop in FTP Template

[nuriensing]

I'm running into a strange issue where an error keeps popping up, and I can't figure out why. It feels like it might be a bug. Here's a simplified version of the XML to show what I mean:

The preview works fine, but I keep seeing a 'data not found' error whenever it hits the 'each' loop in the template.

Hope someone can help me out here.

[tylerlamparter]

It's the validation for have an each loop specifically for the batch_of_records. As long as your page size is set to 1 and skip aggregation is set to true, then this will work. We're hoping to remove this validation in an upcoming release.

<?xml version="1.0" encoding="UTF-8"?>
{{#each batch_of_records}}
{{this.rows.0.id}}
{{#each this.rows}}
{{this.line}}
{{/each}}
{{/each}}

[nuriensing]

Thanks, this works! I assume that after the release, the current handlebar template setup which is now in place will still function properly following the release update, correct?

For anyone who encounters a similar situation, here are some screenshots to help fix it:

1. In the Export, set the page size to 1.
2. In the FTP transfer, set it to skip aggregation.
   
   

[tylerlamparter]

Yeah existing templates would work as they are.

[nuriensing]

Hi @tylerlamparter

I guess the validation isn’t in place yet, because I’m encountering the issue again.
This time, I set the Business Central export page size to 1 and I set “Skip Aggregation” to true on the FTP transfer, but I’m still getting the same error:

{{# each data}} not found in the template.
xml_template_parsing_error

[tylerlamparter]

An each loop of data or batch_of_records is still required, we never got around to removing this validation.

[nuriensing]

@tylerlamparter

<?xml version="1.0" encoding="UTF-8"?>
{{#each batch_of_records}}

Got it — so always one XML declaration at the top, and then start with {{#each batch_of_records}} (or data), regardless of whether we send one record or multiple.

To summarize for FTP:
By default, integrator․io will aggregate all pages of data generated by an export into one (possibly large) file. Check Skip aggregation to instead generate multiple files (one file per page of data).

Important

To generate one file per record, set the export page size to one record per page of data. However, if your page size is set to one record per page and you are using a one to many import, skip aggregation will generate one file per parent record, not one file per child record.

Example if you want to import 1 file:

• In the source step, set page size = 1
• In the FTP step, enable skip aggregation
• XML template: one XML declaration at the top, then the {{#each batch_of_records}} loop inside the body
  Example:

<?xml version="1.0" encoding="UTF-8"?>
<ZMATMAS03>
  {{#each batch_of_records}}

• If the FTP upload does not include a semicolon (:), it may be caused by the default filename format. When the flow step uses the standard file-{{timestamp}}.xml, update it to a format that explicitly defines the timestamp, for example:
  file-{{timestamp "YYYY-MM-DDTHHmmss" "Europe/Amsterdam"}}.xml
  I got this error when using the semicolon which the FTP did not allow: [Unknown message with code "Invalid path.".][Invalid path.]

[tylerlamparter]

If your goal is to make 1 file per record, then yes, you need to set skip aggregation to true and set page size to 1 on the export. If multiple source records can go into 1 file, then you don't need to use skip aggregation and don't need to set your page size to 1.

In any above case, {{#each data}} or {{#each batch_of_records}} is currently required (depending on if you are using AFE 1 or AFE 2).

[nuriensing]

I’ve updated my earlier reply slightly.
For importing a single file, all required conditions must be met; otherwise the file won’t upload.

[Wesley_Blackburn]

This is still a problem nearly a year later. Im also not seeing anything very specific about the skip aggregation enablement.

[Wesley_Blackburn]

I also have the page size set to 1 on my source step.
Here is what Ora is saying about the Skip aggregation:

[tylerlamparter]

Hey @Wesley_Blackburn the good news is this is actually targeted for our Nov release. Does the above thread help you or can you explain what you're trying to do?

[Wesley_Blackburn]

I have a json record im mapping into an xml file that is going into a directory in files.com.

[tylerlamparter]

Can you share a sample of the JSON and sample output you're trying to achieve?

[Wesley_Blackburn]

1. The source step picks up xml files from a vendor ftp, parses them to json
2. A lookup does a request to NetSuite to get an ID
3. Post Response mapping adds the ID to the record
4. Branching checks if the lookup ID is empty
   1. If empty, parse the json to xml and output the file on files.com
   2. If not empty go to step 5
5. Make a request to NetSuite API to adjust a purchase order

[tylerlamparter]

Thanks for the files @Wesley_Blackburn, that made this easy to pin down.

The file import checks that the template has a top-level {{#each batch_of_records}} block. Yours starts with {{#each batch_of_records.0.record as |value key|}}, which is valid handlebars but doesn't pass that check, so it fails at runtime even though preview renders fine (preview doesn't run that validation). Two edits and you're done:

Before:

<UpdateOrders><OrderInfo>{{#each batch_of_records.0.record as |value key|}} ... {{/each}}</OrderInfo></UpdateOrders>

After:

<UpdateOrders>{{#each batch_of_records}}<OrderInfo>{{#each this.record as |value key|}} ... {{/each}}</OrderInfo>{{/each}}</UpdateOrders>

Everything inside the loop stays as you have it. I rebuilt your flow in my account with the sample you sent and the file that comes out matches your preview output. Side benefit is that if you ever raise the page size, you get one <OrderInfo> per record in the file instead of an error.

On skip aggregation, the reason you couldn't find it is the checkbox got replaced. On the Files.com step it's now "How many files would you like to generate?" and you want "One file per page of records." With page size 1 on the source step, that's one file per record.

Removing the validation is still targeted for Nov. Your original template would work after that, but the above works today and keeps working after.

6ab2ef337b3d01a90883fc00 (1).zip (9.4 KB)

[Wesley_Blackburn]

Thank you for your quick review, and feedback.