Kibana won't parse field with date format from NIFI PutElastic Processor

Elastic Stack Kibana
1

Good day,

I failed to parse a field with string date in it in Kibana, below is the record sample taken from kibana dashboard

{date:2018/05/08 16:51:25 +0700 id:f0692c8a-72bd-46ed-8eba-df1aae8c8c20 msisdn:628528484800 body:Hello world!!!. infos.shortcode:9999 infos.latency:7,246 _id:3b08121d-1428-4c14-8105-625e6295e8ed _type:data _index:msg_idx-2018-05-08 _score:1}

As you can see, in NIFi i use format yyyy/MM/dd HH:mm:ss Z, i have been trying yyyyMMdd'T'HHmmss.SSSZ and yyyy-MM-dd HH:mm:ss.SSS; All failed to parse in Kibana (based on strict date format documentation), the only success one is when using yyyy/MM/dd value only, but i need to store the time also so this format is a no.

If i update the index using update API with curl, i got error below, even when i close the index first using curl -X POST 'http://localhost:9200/msg_idx-2018-05-08/_close'

error":{"root_cause":[{"type":"index_already_exists_exception","reason":"index [msg_idx-2018-05-08/IySLbAGjTa2DOdRIoLknzA] already exists","index_uuid":"IySLbAGjTa2DOdRIoLknzA","index":"msg_idx-2018-05-08"}],"type":"index_already_exists_exception","reason":"index [msg_idx-2018-05-08/IySLbAGjTa2DOdRIoLknzA] already exists","index_uuid":"IySLbAGjTa2DOdRIoLknzA","index":"msg_idx-2018-05-08"},"status":400}

I am using NIFI putElastic processor to store stream of records to my ES and intend to use Kibana to visualize data based on date histogram, but since Kibana can't parse my "date" field, it is not working.

My setup:

  • Apache Nifi 1.6.0
  • Elastic 5.6
  • Kibana 5.6

What am i missing here?

The last resort might be trying to separate the date and time into each field; i.e. date:2018/05/08 time:16:51:25,

2

yyyy/MM/dd HH:mm:ss Z is a valid format, as is.

Example mapping:

PUT sample_index
{
  "mappings": {
    "doc": {
      "properties": {
        "date": {
          "type": "date",
          "format": "yyyy/MM/dd HH:mm:ss Z"
        }
      }
    }
  }
}

Which will parse the date of this document correctly:

PUT sample_index/doc/1
{
  "date":"2018/05/08 16:51:25 +0700"
}
3

My fault, it seems like i need to change my date format in advanced settings to

YYYY-MM-DD HH:mm:ss.SSS

and using same pattern in my Nifi code as well..

Thank you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.