# Trying to parse Hadoop log with logstash **URL:** \ **Category:** Logstash\ **Created:** [June 24, 2016, 2:13pm UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894 "2016-06-24T14:13:16Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![siasic](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@siasic](https://discuss.elastic.co/u/siasic)\ **Post date:** [June 24, 2016, 2:13pm UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/1 "2016-06-24T14:13:16Z")
hello, I have a Hadoop cluster (Hortonworks) and i try to send hdfs/yarn/.... logs to logstash for parsing them. The sent is made by filebeat. My logs look like that (i think it is log4j type) : ``` 2016-06-12 03:00:20,432 INFO ipc.Client (Client.java:handleConnectionFailure(869)) - Retrying connect to server: d1hdpslave01.ouest-france.fr/128.1.228.46:8020. A lready tried 25 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS) 2016-06-12 03:00:20,660 ERROR datanode.DataNode (DataXceiver.java:run(278)) - d1hdpslave03.ouest-france.fr:50010:DataXceiver error processing unknown operation sr c: /128.1.228.49:51734 dst: /128.1.228.49:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) ``` Here is my logstash config at the moment : ``` filter{ grok { match => ["message", "(?m)%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:severity} {1,2}%{JAVACLASS:class} \(%{USERNAME:java_family}\.%{USERNAME:error_type}\:%{USERNAME:java_sub_family}\(%{INT:java_num}\)\) \- %{GREEDYDATA:message}"] overwrite => ["message"] } mutate { remove_field => ["[beat]","input_type","offset" ] } } ``` Did anyone succeed to parse entirely this kind of log ? At least a little bit more the java error message. Thanks, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 29, 2016, 8:16pm UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/2 "2016-06-29T20:16:49Z")
This should work. What do you get? Please show an example event produced by a `stdout { codec => rubydebug }` codec. ---
**Author:** ![siasic](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@siasic](https://discuss.elastic.co/u/siasic)\ **Post date:** [June 30, 2016, 7:07am UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/3 "2016-06-30T07:07:07Z")
Here is what i get : ``` 2016-06-30T07:02:24.329Z dxxdpslavexx dXXdpslave0X.ouest-france.fr:50010:DataXceiver error processing unknown operation src: /128.1.22x.xx:49066 dst: /128.1.22x.xx:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) { "message" => "dxxdpslavxxx.ouest-france.fr:50010:DataXceiver error processing unknown operation src: /128.1.22x.xx:43851 dst: /128.1.22x.xx:50010\njava.io.EOFException\n\tat java.io.DataInputStream.readShort(DataInputStream.java:315)\n\tat org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)\n\tat org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)\n\tat java.lang.Thread.run(Thread.java:745)", "@version" => "1", "@timestamp" => "2016-06-30T07:02:32.405Z", "count" => 1, "source" => "/var/log/hadoop/hdfs/hadoop-hdfs-datanode-d1hdpslave02.log", "type" => "hdfs_datanode", "fields" => nil, "host" => "dxxdpslavexx", "tags" => [ [0] "beats_input_codec_plain_applied" ], "timestamp" => "2016-06-30 09:02:23,269", "severity" => "ERROR", "class" => "datanode.DataNode", "java_family" => "DataXceiver", "error_type" => "java", "java_sub_family" => "run", "java_num" => "278" } ``` Is there a way to parse a little bit more the "message" part? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 30, 2016, 7:29am UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/4 "2016-06-30T07:29:32Z")
Sure, just continue the grok expression in the same manner as you started it. I'm not sure I understand the difficulty. How, exactly, would you like to have the remainder of the message parsed? ---
**Author:** ![siasic](https://avatars.discourse-cdn.com/v4/letter/s/5daacb/32.png) [@siasic](https://discuss.elastic.co/u/siasic)\ **Post date:** [June 30, 2016, 7:56am UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/5 "2016-06-30T07:56:52Z")
The problem is, messages can be different: ``` 2016-06-30T07:52:08.822Z d1hdpslave01 Get corrupt file blocks returned error: Operation category READ is not supported in state standby { "message" => "Get corrupt file blocks returned error: Operation category READ is not supported in state standby", "@version" => "1", "@timestamp" => "2016-06-30T07:52:08.822Z", "source" => "/var/log/hadoop/hdfs/hadoop-hdfs-namenode-d1hdpslave01.log", "type" => "hdfs_namenode", "count" => 1, "fields" => nil, "host" => "d1hdpslave01", "tags" => [ [0] "beats_input_codec_plain_applied" ], "timestamp" => "2016-06-30 09:52:08,774", "severity" => "WARN", "class" => "namenode.FSNamesystem", "java_family" => "FSNamesystem", "error_type" => "java", "java_sub_family" => "getCorruptFiles", "java_num" => "7324" } ``` I don't know the full possibility of logstash. Maybe i can't go any further. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 30, 2016, 8:54am UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/6 "2016-06-30T08:54:46Z")
A single grok filter can try to match the message against multiple expressions (see example in the grok filter documentation), so you don't have to write a single expression that matches every string imaginable. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:50am UTC](https://discuss.elastic.co/t/trying-to-parse-hadoop-log-with-logstash/53894/7 "2017-07-06T04:50:11Z")