Date parsing error

Elastic Stack Elasticsearch
1

Hello, I'm beginner of ELK.
I got a problem while mapping date of my log file.
Does anybody know how to map this kind of date format ? (20170802-022130.497289)

My mapping source -->

"column1": {
"type": "date",
"format" : "yyyyMMdd-HHmmss.SSSSSS"
},

mapper_parsing_exception -->

"reason"=>"failed to parse [column1]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "20170802-022130.497289" is malformed at "289""}}}}}

2

I am using 5.1.1. For me its working fine. FYI adding all the mappings

PUT test-date
{
  "mappings": {
    "logs":{
      "properties": {
        "column1":{
          "type": "date",
          "format": "yyyyMMdd-HHmmss.SSSSSS"
        }
      }
    }
  }
}

Inserted one document

POST test-date/logs/1
{
  "column1": "20170802-022130.497289"
}

Search using range query

GET test-date/_search
{
  "query": {
    "range": {
      "column1": {
        "format": "yyyy-MM-dd", 
        "gte": "2017-08-01",
        "lte": "2017-08-03"
      }
    }
  }
}
1 Like
3

Oh! Thanks :slight_smile:
I think "logs" parameter was missed.
It's working well now!

4

Thanks, but I got one more problem.
This is what I saw in Kibana.

test
test.JPG1041×445 76.3 KB

when I do mapping with this format, time(hh) value has been changed.

Why does it happen?

Original Value : 20170803-050214.627322
Mapping Value : 20170803-140214.627

5

ES always stores the date in GMT format.

6

Yes, I can understand that GMT format was used.
But I cannot understand why hour is changed from 05 to 14

Original Value : 20170803-050214.627322
Mapping Value : 20170803-140214.627

7

In what time zone you written your log???

8

Thanks for your really fast support!

Oh! now I understand what you mean.
I was using (UTC+01:00) Amsterdam for log file, and (UTC+09:00) Seoul for ELK Server.

There are 8h of time difference . Not exactly matching with time difference but anyway, Is there any method to ignore time zone then?

9

Try this mapping

PUT test-date
{
  "mappings": {
    "logs":{
      "properties": {
        "column1":{
          "type": "date",
          "format": "yyyyMMdd-HHmmss.SSSSSS Z"
        }
      }
    }
  }
}

And index the data with time zones

POST test-date/logs/1
{
  "column1": "20170802-022130.497289 +0100"
}
10

Sorry what you understand here is wrong.

If you don't pass time zone ES thinks that you are passing GMT time. Means UTC +00:00. So you have nine hours gap there

11

Yes, you were right!
It was not the problem of data.

I corrected time zone of Kibana which was browser default.
Management -> Advanced Setting -> dateFormat:tz to Etc:GMT
and it displayed perfectly!

Anyway thank you very much Ravi :slight_smile:

Best Regards
SG

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.