Flow Data Previews - What am I doing wrong?

[steveklett]

I hope I'm doing something wrong and the state of the data previews isn't as bad as it appears (to me). I deal with this almost every day, whenever I need to create or edit a flow I struggle to some degree with the data that is, or ISN'T included in the previews. This morning as I'm working away and going through my usual distasteful set of techniques and tricks to make the previews work, I thought I'd post here and make sure that I'm not missing a setting somewhere or doing something incorrect. Again, I hope I am.

I didn't really state the problem yet: Data previews truncate arrays to a single element which not only is disorienting, it can make scripts fail which then breaks everything downstream.

My basic approach to workaround this is to copy preview data from wherever I can find it complete, modify it in a text editor, JSONlint it and paste into the preview of the following operation or step. After doing this several times I finally get to (what I think is) valid preview data and I can continue my mapping work (or whatever).

I'm hoping the screen shots will illustrate the pain and frustration of this process.
Here we go!

For this example the problem stems from a Lookup that returns an array. IIO will only show a SINGLE record from the lookup in the downstream previews.

First, I preview the lookup step (this pulls down up to 100 of the results):

Then I select and copy all the preview data and paste into notepad++ where I remove the "record" level from each record as well as the "page_of_records"

Then I copy that result and lint it, just to make sure it's valid JSON. After that I paste the result into the Lookup results mapping window because it FOR SOME REASON will only show a single record.

I click Preview and copy the Output. On we go to the script editor for that same Lookup step. I paste the output from the results mapping into the text editor. I remove the top level object and "record" property. I LINT it again, to make sure.

I then paste the single record (form the original export) with ALL the lookup records as an array property into the just-the-right-spot in the Input of the script editor:
Reminder: The script editor opens with Function Input that contains only ONE of the lookup records.

I copy all the Function output. The mapper dialog has no Input because the script on the lookup threw an exception because it couldn't find expected data in the lookup array. This is deliberate, scripts should be able to validate the presence of required data and conditions.

And finally my final paste:

So.... Am I doing it wrong?

Oh, and if I change the flow, maybe add a column in the lookup, for example, then I need to do this all over again.

Please tell me I'm doing it wrong.

[amanjotsingh]

Hello Steve,

Thank you for sharing the feedback. The workarounds you've listed above are what other users on the platform are using as well. The platform is designed to work with a single record for transformations, mapper, etc. That being said, we have identified these issues and are actively working on resolving them.

In short term, we are introducing a test mode, where you could test your flow using multiple records and can access those records within a step such as filters, transforms, or hooks. You won't have to take data from a previous step and paste it into the next. This functionality would only be limited to test mode.

Long term, we are exploring the idea (not prioritized for a release yet) to support the ability to process multiple records during design time (outside of test mode) so you can preview the complete data that gets out of export and not have any import step with no input data.

Hope this answers your query. I am happy to talk more about our plans for such issues over zoom, if necessary.

Regards