Data Visualizer - Import/Convert multiple date fields

Elastic Stack Kibana
1

All: Fought this all day and asking for help. I am running 7.4.2
I have a CSV file with two date fields. I want to keep both a keyword and date version of each field.
I have reviewed several posts but don't see one that addresses my problem.
I am attempting to use the Kibana Data Visualizer file import tools.

This is my mapping

{
  "ACType": {
    "type": "keyword"
  },
  "ASN": {
    "type": "keyword"
  },
  "BaseID": {
    "type": "keyword"
  },
  "Download_DT": {
    "type": "keyword"
  },
  "ESN": {
    "type": "keyword"
  },
  "FC3Caused": {
    "type": "long"
  },
  "FCTYPE": {
    "type": "keyword"
  },
  "FTIME": {
    "type": "double"
  },
  "FaultCode": {
    "type": "long"
  },
  "FaultText": {
    "type": "keyword"
  },
  "Fault_DT": {
    "type": "keyword"
  },
  "FltsAgo": {
    "type": "long"
  },
  "FreqFlt": {
    "type": "keyword"
  },
  "Location": {
    "type": "long"
  },
  "Remark": {
    "type": "text"
  },
  "Remark_DT": {
    "type": "keyword"
  },
  "DownloadDate": {
    "type": "date"
  },
  "FaultDate": {
    "type": "date"
  }
}

This is my input pipeline:

{
  "description": "Ingest pipeline created by file structure finder",
  "processors": [
   {
      "date": {
        "field": "Download_DT",
        "target_field": "DownloadDate",
        "timezone": "{{ beat.timezone }}",
        "formats": [
          "MM/dd/yy HH:mm"
        ]
      }
    },
    {
      "date": {
        "field": "Fault_DT",
        "target_field":"FaultDate",
        "timezone": "{{ beat.timezone }}",
        "formats": [
          "MM/dd/yy HH:mm"
        ]
      }
    }
  ]
}

I have tried several variations of this. In this case the import fails as follows:

0: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [12/21/17 17:53]

{"ACType":"F-15","ASN":"890500","BaseID":"SEYMOUR-JOHNSON","Download_DT":"12/21/17 16:50","ESN":"719321","Fault_DT":"12/21/17 17:53","FaultCode":72,"FaultText":"FTIT B SIGNL","FCTYPE":"FC","FltsAgo":0,"FreqFlt":"T","FTIME":3837.66,"Location":2}

1: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: unable to parse date [12/21/17 17:53]

{"ACType":"F-15","ASN":"890500","BaseID":"SEYMOUR-JOHNSON","Download_DT":"12/21/17 16:50","ESN":"719321","Fault_DT":"12/21/17 17:53","FaultCode":4070,"FaultText":"FTIT SYSTEM","FC3Caused":72,"FTIME":3837.66,"Location":2}

If I take out the extra {} around each date processor as suggested in another post, only the last one in the list is processed.

2

Hi

Do you need a ingest processing, would it be sufficient for you to solve it just by mapping? e.g. using the following:

{
  "date1": {
    "type": "date",
    "format": "MM/dd/yy HH:mm",
    "fields":{
      "raw": {
        "type": "keyword"
      }
    }
  },
  "date2": {
    "type": "date",
    "format": "MM/dd/yy HH:mm",
     "fields":{
      "raw": {
        "type": "keyword"
      }
    }
  }
}

Thx & Best,
Matthias

3

Thanks that did the trick.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.