Filter issue with logstash beta 2.0

Elastic Stack Logstash
1

Hi, I've just installed elaticsearch 2.0.0-rc1, kibana 4.2.0-beta2 and logstash 2.0.0-beta2.

I have a logstash config that picks up data from at txt file:

Textfile example data:
1;127.0.0.1;2014-06-22T12:32:35

Logstash config file:
input { file { path => "C:/dashboard/datafiles/users.txt"}}
filter {grok {match => ["message","%{NUMBER:siteid};%{IP:usersip};%{TIMESTAMP_ISO8601:signdate}"]} geoip {source => "usersip"}}
output { elasticsearch { hosts => "127.0.0.1:9202"}}

Now when I config a pattern in Kibana, it doesn't recognize my data correctly. It recognizes the field siteid correct, but then "usersip" is read as "1;127.0.0" and "signdate" as "1;127.0.0.1;20"

I hope someone can help out with this one as I am running out of ideas.

Kind regards, Bo

2

I can't reproduce what you're seeing.

$ cat test.config 
input { stdin {} }
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => [
      "message",
      "%{NUMBER:siteid};%{IP:usersip};%{TIMESTAMP_ISO8601:signdate}"
    ]
  }
}
$ cat data 
1;127.0.0.1;2014-06-22T12:32:35
$ /opt/logstash/bin/logstash -f test.config < data
Logstash startup completed
{
       "message" => "1;127.0.0.1;2014-06-22T12:32:35",
      "@version" => "1",
    "@timestamp" => "2015-10-21T11:07:36.109Z",
          "host" => "lnxolofon",
        "siteid" => "1",
       "usersip" => "127.0.0.1",
      "signdate" => "2014-06-22T12:32:35"
}
Logstash shutdown completed
3

Could it be an issue with kibana? This is how kibana filters the data. It doesn't recognize signdate as a time-field name when create an index pattern.

Udklip.PNG1486×57 8.79 KB

4

Kibana shows what's in Elasticsearch and Elasticsearch stores what Logstash gives to it. Logstash creates indexes based on the @timestamp field which, absent a date filter, will contain the time when the message entered Logstash.

I don't think the document shown in the Kibana screenshot was created by a Logstash instance with the configuration you showed us. I've given you an example that works as expected. Do you get a different result?

5

Excuse me for my ignorance but do you know a way to test like you do using data in the command prompt on a windows machine?

6

My suspicion is that it is related to the txt-file. The same setup though works fine using logstash-2.0.0-beta1 and elasticsearch 2.0.0-beta1

7

..Or of course related to the newer versions of elasticsearch and logstash :slight_smile:

8

Excuse me for my ignorance but do you know a way to test like you do using data in the command prompt on a windows machine?

It works the same way. You won't have the "cat" command and the path to the Logstash executable will be different, obviously.

9

Hi again Magnus. I removed logstash 2.0.0-beta2 and installed current production version 1.5.4 and now it reads my data correctly. So guess that there is a bug in the current beta version of logstash regarding the parsing of txt files.

Thanks for your help

/bo