Issue with Logging Job Details During Script Execution in Flow

Integrator.io
,
1

Hi Team,

We are currently facing an issue with job detail logging during the execution of a flow.

Issue Summary:

We have added custom scripts (preMap, postMap, and postSubmit) to our flow to log the options.job object and help us trace job metadata during runtime. Here is the sample code used:

A script editor window displays JavaScript code defining a function, which logs job details and returns mapped data objects. (Captioned by AI)
A script editor window displays JavaScript code defining a function, which logs job details and returns mapped data objects. (Captioned by AI)420×366 18.3 KB

What’s Working:

  • When using the Preview feature while writing the script, the options.job object is populated correctly and visible in the logs.

(See Snapshot 2)

What’s Not Working:

  • During actual flow execution, the logs consistently show:

job : {"parentJob":{}} in all stages — preMap, postMap, and postSubmit. (See Snapshot 3)

  • The postMap function does not execute at all during real execution (only works during Preview).
  • When using similar logic in preSavePage, the logs appear blank initially, and after 7–20 minutes, job details start appearing in the console.

Additional Observation:

We did not run the flow twice but waited to observe the behavior of the parentJob field, which initially appeared as parentJob: {} at 12:25:08 PM on 05/16/2025, and by 12:30:33 PM - without any changes to the flow - it was fully populated with metadata including _id, type, startedAt, etc. (see Snapshot 1).

This suggests there may be a delay or internal lag in metadata propagation within the Celigo platform.

We would like to understand:

  1. Why are job details (especially parentJob) empty or delayed during actual flow executions?
  2. Why is the postMap function being skipped during live runs?
  3. Is the delayed logging in preSavePage a known issue?
  4. Are there best practices or configurations to ensure consistent availability of job context?

Snapshot References:

  • Snapshot 1: parentJob: {} during execution at 12:25:08 pm. Complete job details visible at 12:30:33 pm

The image shows a log file with entries detailing API posts, timestamps, and JSON records. (Captioned by AI)
The image shows a log file with entries detailing API posts, timestamps, and JSON records. (Captioned by AI)800×275 25.9 KB

  • Snapshot 2: Preview Screenshot: Showing full job object

A script editor interface displays sections for writing scripts, function input, function output, and console logs. (Captioned by AI)
A script editor interface displays sections for writing scripts, function input, function output, and console logs. (Captioned by AI)800×284 23.4 KB

  • Snapshot 3: Log Entries: Log Entries for (preMap, postMap, and postSubmit)

The image shows a dashboard interface for a JSON Post Data Job in Celigo's platform, displaying execution logs of data imported to an API. (Captioned by AI)
The image shows a dashboard interface for a JSON Post Data Job in Celigo's platform, displaying execution logs of data imported to an API. (Captioned by AI)800×277 25.2 KB

2

Does your flow have any branching? If not, can you add a branch anywhere on the flow and retest?

3

Currently, my flow does not have a branch. I tested by adding a branch in the flow & the job details are visible in the script logs. Could you provide the significance of adding a branch in the flow with respect to the visibility of job details?

4

I gave a short answer here:

Expanding on that answer, we are at the tail end of our platform rearchitecture, so this won’t be an issue when all customers and all flows are migrated to the new microservices we now have.

As for logging being delayed, we are working on a completely new logging feature and technology to consolidate and enhance everything about logging in Celigo.

5

Can you also share your flow zip here? The postMap shouldn’t be skipped and I’m not able to reproduce that.

6

Hi Tyler, below is the zip file of the test flow.
Display Job Details.zip (6.4 KB)

1 Like