Converting date to ISO8601

Elastic Stack Logstash
1

Hi guys. I have a date converting problem. I have logs in this format:

2015.11.10 03:02:23.832: Some text.

And I parse logs by grok:

match => [ "message", "%{YEAR:year}.%{MONTHNUM:month}.%{MONTHDAY:day}%{SPACE}%{TIME:time}:%{SPACE}%{GREEDYDATA:text}" ]

While not adding date filter it works perfectly: log parses and field adds. But with date filter some parts of messages disappears

I'm using the following config:

input {
tcp {
type => "somelogs"
port => 8889
}
}

filter {
if [type] == "somelogs"
{
grok
{
match => [ "message", "%{YEAR:year}.%{MONTHNUM:month}.%{MONTHDAY:day}%{SPACE}%{TIME:time}:%{SPACE}%{GREEDYDATA:text}" ]
}
}
mutate
{
add_field => { "timestamp" => "%{year}-%{month}-%{day} %{time}" }
gsub => [ "timestamp", " ", "T" ]
}
date
{
match => [ "timestamp", "ISO8601" ]
timezone => "Asia/Novosibirsk"
remove_field => [ "timestamp" ]
}
}

output {
elasticsearch
{
protocol => "http"
host => "localhost"
index => "%{type}-%{+YYYY.MM}"
}
}

2

Works fine for me. If you can be a bit more explicit than "some parts of messages disappears" it'll probably be easier to help.

$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  grok { 
    match => [
      "message",
      "%{YEAR:year}.%{MONTHNUM:month}.%{MONTHDAY:day}%{SPACE}%{TIME:time}\:%{SPACE}%{GREEDYDATA:text}"
    ]
  }
  mutate {
    add_field => { "timestamp" => "%{year}-%{month}-%{day} %{time}" }
    gsub => [ "timestamp", " ", "T" ]
  }
  date {
    match => [ "timestamp", "ISO8601" ]
    timezone => "Asia/Novosibirsk"
    remove_field => [ "logTimestampString" ]
  }
}
$ echo '2015.11.10 03:02:23.832: Some text.' | /opt/logstash/bin/logstash -f test.config
Logstash startup completed
{
       "message" => "2015.11.10 03:02:23.832: Some text.",
      "@version" => "1",
    "@timestamp" => "2015-11-09T21:02:23.832Z",
          "host" => "lnxolofon",
          "year" => "2015",
         "month" => "11",
           "day" => "10",
          "time" => "03:02:23.832",
          "text" => "Some text.",
     "timestamp" => "2015-11-10 03:02:23.832"
}
Logstash shutdown completed
3

@magnusbaeck

For example if I create simple " * " index, Kibana shows all 1000+ logs.
For example:

message:2015.11.10 03:02:34.988: Some text @version:1 @timestamp:2015-11-09T21:02:34.988Z host:127.0.0.1 type:common-logs year:2015 month:11 day:10 time:03:02:34.988 text:Some text _id:AVE4WAr88rd3B6Hi6wjO _type:common-logs _index:common-logs-2015.11

As you can see in message date = 2015.11.10 03:02:34.988. But in @timestamp I have "2015-11-09T21:02:34.988Z". Why it is not equal?

But if I create " common-logs-* " index, I have only 4 logs from 1000+. For example:

1314×373 24.3 KB

Message field has been corrupted(( Full message may look loke: "Event was changed for Id 194. Old value was null, new value is 67"

4

As you can see in message date = 2015.11.10 03:02:34.988. But in @timestamp I have "2015-11-09T21:02:34.988Z". Why it is not equal?

The @timestamp field is UTC while the time displayed in the browser is adjusted to the browser's local time. The six-hour difference between your two timestamps are explained by the UTC+6 timezone in Novosibirsk.

Message field has been corrupted(( Full message may look loke: "Event was changed for Id 194. Old value was null, new value is 67"

It looks like the original message actually was " at b.a()", seemingly a line from a multiline stacktrace.

5

@magnusbaeck Can I send you a file (privately) with sample logs and logstash conf for testing on your machine?

6

I'd rather not. Please prove that there are no " at b.a()" lines in the log file, e.g. using grep.

7

My logs have this record: "2015.11.10 03:07:57.523: Thread was being aborted. , at System.Threading.Thread.SleepInternal(Int32 millisecondsTimeout)
at System.Threading.Thread.Sleep(Int32 millisecondsTimeout)
at b.a()"

When I create simple " * " index, Kibana shows all 1000+ logs normally including the above record.
Problems starts when I trying create . common-logs-* index. In that case it only shows 4 logs from 1000.
If I don't use a date plugin - Kibana shows everything correctly/

8

I strongly doubt the date filter has anything to do with this. I don't know what you mean by "* index". Lines without a timestamp in the beginning will be stamped with the current date and time, so if you're processing old logs they will be found in the wrong place.

Instead of focusing on thousands of events, feed Logstash with a single line and make sure you get the expected results.

9

@magnusbaeck You are definetely right. I'm sorry, it was my fault. Problem was in multiline record. Problem is solved.

10

Hi Forgand.

Can you please suggest what was the problem with multi-line records and how you solved it.

I am also facing the same issue.

11

I also have this problem

please can you show your multi-line case in config file