How to add logstash gork filter and field

Elastic Stack Logstash
#1

I'm using Filebeat -> Logstash -> Elasticsearch -> Kibana to have an overview of my glassfish log file.

My log message look like this.

[#|2016-11-   01T11:29:33.347+0100|INFO|glassfish3.1.2|com.MachineProxy|_ThreadID=226;_ThreadName=Thread-2;|proxy started|#]

I'm using Filebeat -> Logstash -> Elasticsearch -> Kibana to have an overview of my glassfish log file.

My log message look like this.

[#|2016-11-01T11:29:33.347+0100|INFO|glassfish3.1.2|com.MachineProxy|_ThreadID=226;_ThreadName=Thread-2;|proxy started|#]

My current pattern look like this. I have tested with http://grokdebug.herokuapp.com/

(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL}\|%{DATA:server_version}\|%{JAVACLASS}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]

I want to have a fields in Kibana for "DateTime", "Log Level", "Server version", "Java Class Name", "Thread" and "Message detail"

Many thanks in advance.

Thomas

#2

I want to have a fields in Kibana for "DateTime",

Use the date filter to parse the timestamp field you're extracting from the log entry. That'll populate the @timestamp field that you can use in Kibana.

"Log Level", "Server version", "Java Class Name", "Thread" and "Message detail"

What you have almost works. Just make sure you extract the fields you're interested in in your grok expression instead of just matching the strings, i.e. use %{LOGLEVEL:Log Level} instead of %{LOGLEVEL}.

1 Like
#3

Hi @magnusbaeck

Thanks.

I have another problems with my filebeat.yml

when I start my "sudo ./filebeat -e -c filebeat-logstash.yml -d "publish", I always got the following error.

anb041:filebeat-1.3.1 mmlug$ sudo ./filebeat -e -c filebeat-logstash.yml -d "publish"
2016/11/08 13:36:28.292774 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/11/08 13:36:28.294921 publish.go:269: INFO No outputs are defined. Please define one under the output     section.
Error Initialising publisher: No outputs are defined. Please define one under the output section.
2016/11/08 13:36:28.294943 beat.go:161: CRIT No outputs are defined. Please define one under the output section.

My filebeat configuration look like here.

http://pastie.org/10958177

Many Thanks in advance.

Thomas

#4

You have so much commented out noise in your config file that you're not seeing that you've commented the

  output:

line so the subsequent

    logstash:

line ends up in the wrong configuration section.

Please post any follow-up questions about Filebeat in the Filebeat category.

1 Like
#5

Thanks.!!! Yes, I just found it too :slight_smile:

#6