My timestamp can't get converted to a date type

Elastic Stack Logstash
1

Hello,

I have some working timestamp detected as date in elasticsearch.
But can't make this one works. Grok is matching well, and logstash is parsing well the line:
2017-05-29 10:27:41,756 | soft-1 | user1 | | 10.x.x.x | INFO | intranet | Order 9979a662-b42a-414d-9d5c-5cxxxxxxxx creation TESTING request is finished

filter {
      match => {
        "message" => [
          "^%{TIMESTAMP_ISO8601:[log][timestamp]}\s*\|\s*%{GREEDYDATA:[log][message]}$"
        ]
      }
    }
      mutate {
        convert => { "[log][timestamp]" => "date" }
      }
      date {
        match => [ "[log][timestamp]", "yyyy-MM-dd HH:mm:ss,SSS" ]
      }
}

For some reason, [log][timestamp] appears as a string, even after deleting everything, or forcing to mutate to date.

How to troubleshoot that? Using logstash debug doesn't show me the resulting type of data (I tried in debug/verbose mode):
output { stdout { codec => rubydebug } }

Thank you for any help!
Greg.

2

Please show what stdout { codec => rubydebug } produces. What do the mappings of your ES index look like?

3

Thank you for taking time helping me!
Here is the output of stdout:

logstash.png1786×261 45.5 KB

From this output, is there a way to see the format of the field log.timestamp? Only test I can do is to send to elastic and check the field type over there:

{
  "filebeat-2017.05.29": {
    "aliases": {},
    "mappings": {
      "log": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
...
          "log": {
            "properties": {
              "timestamp": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
...
}
  • Seems like I am having problem (detected as a string) with this timestamp specifically "2017-05-29 10:27:41,756" and date { match => [ "[log][timestamp]", "yyyy-MM-dd HH:mm:ss,SSS" ] }
  • "2017-05-29 10:27:41Z" is detecting fine as a date withdate { match => [ "[log][timestamp]", "ISO8601" ] }

Thank you!

4

You didn't include the mapping of the timestamp field when you redacted the mappings.

The date filter worked just fine and was able to update the @timestamp field. If you expect it to update [log][timestamp] you need to configure it accordingly.

Why keep the fields under log? They appear to be duplicates of the corresponding root-level fields.

5

Sorry, I updated/edited the mapping on top.
I didn't write in advance the [log][timestamp] mapping, it was automatic.
I will remove this [log][timestamp] you are right, no need for duplication.

6

Trying this pattern:

filter {
    grok {
      match => {
        "message" => [
          "^%{MYTIME:mytimestamp:date}\s*\|\s*%{GREEDYDATA:msg}$"
        ]
      }
    }
      date {
        match => [ "mytimestamp", "yyyy-MM-dd HH:mm:ss" ]
      }
  }
}

# log example: 2017-05-29 10:26:41 | test1
# Added patterns : MYTIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s%{HOUR}:?%{MINUTE}(?::?%{SECOND})

Still, mytimestamp is recognized as a string. I can't see what is missing...

7 
%{MYTIME:mytimestamp:date}

"date" isn't a valid type here. See the docs.

Why do you need mytimestamp to be a date? Your date filter (as configured) stores the result in @timestamp, a field you can't delete. Why not just use @timestamp and delete mytimestamp after the date filter?

8

Thx Magnus for your answer!

I got 2 sources of mytimestamp:

  • 2017-05-23 17:14:16,125 --> recognized as string
  • 2017-05-23T17:13:16.925Z --> recognized as date
    These makes conflict in ES so I wanted both as date.

I managed to fix my issue with forcing target => "mytimestamp", but it is a bit ugly.

filter {
      match => {
        "message" => [
          "^%{TIMESTAMP_ISO8601:mytimestamp:date}\s*\|\s*%{GREEDYDATA:msg}$"
        ]
      }
    }
      date {
        match => [ "mytimestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
        target => "@timestamp"
        timezone => "Europe/Zurich"
      }
      date {
        match => [ "mytimestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
        target => "mytimestamp"
        timezone => "Europe/Zurich"
      }
  }
}

You are right, for now I will remove this temp field (mytimestamp) as @timestamp is right.

Thank you for your help!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.