Filtering out Duplicate data from source file

Integrator.io
,
1

Hi all, I'm working with a csv file which I selected as a source file, and I want to filter duplicate record from that source file before it reaches the destination. If I have to use output filter then what kind of filter I should use and also if there is any other way to filter out the duplicate records from the source CSV file?

2

Hi Dibyojyoti,

For an FTP export, you can group rows together.

Example:

5301043caa20740200000001-50a5cd907c0d4d449f55b65507d4616e.png
5301043caa20740200000001-50a5cd907c0d4d449f55b65507d4616e.png1431×569 32.1 KB

Before:

{
  "page_of_records": [
    {
      "record": {
        "Column0": "US",
        "Column1": "CA",
        "Column2": "2"
      }
    },
    {
      "record": {
        "Column0": "US",
        "Column1": "AZ",
        "Column2": "3"
      }
    },
    {
      "record": {
        "Column0": "US",
        "Column1": "AR",
        "Column2": "5"
      }
    },
    {
      "record": {
        "Column0": "IN",
        "Column1": "AR",
        "Column2": "5"
      }
    },
    {
      "record": {
        "Column0": "IN",
        "Column1": "TS",
        "Column2": "8"
      }
    }
  ]
}

After:

{
  "page_of_records": [
    {
      "rows": [
        {
          "Column0": "US",
          "Column1": "CA",
          "Column2": "2"
        },
        {
          "Column0": "US",
          "Column1": "AZ",
          "Column2": "3"
        },
        {
          "Column0": "US",
          "Column1": "AR",
          "Column2": "5"
        }
      ]
    },
    {
      "rows": [
        {
          "Column0": "IN",
          "Column1": "AR",
          "Column2": "5"
        },
        {
          "Column0": "IN",
          "Column1": "TS",
          "Column2": "8"
        }
      ]
    }
  ]
}

Let us know if this works for you. Feel free to revert back if you have any queries.

3

Thanks @dibyojyotisamanta for the post and @sachingoud for the prompt response!
(I've moved this post from "Welcome to our community" to "Troubleshoot custom flows")

4

Hi Sachin, Thanks for your valuable reply, It worked.