Graylog2 didn't recognize gelf output from logstash,but try in nc test is ok, Thanks for any suggestion


(Hong Zhou) #1

graylog2 didn't recognize gelf output from logstash,
and still cann't find the way to fix it 。

but when i try nc test direct to graylog2 ,everything is ok。 just not work when get from logstash gelf output

when i trying tcpdump compare nc and logstash, the message show from the tcp stream totally not recognize but it's ok for nc test!


here is the some detail,really appreciate for any suggestion

a: my setup

logstash(gelf output)--------->--------graylog2(gelf udp input)

b: my log

55.3.244.1 GET /index.html 15824 0.043

c: logstash conf

input {
file {
path => "/etc/logstash/ces"
}
}

filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"}
}
}

output {
gelf {
host => "192.168.88.109"
codec => json
custom_fields => ["environment", "uat"]
}
stdout { codec => json }
}

d: logstash debug message (Version : 2.1.1)

https://groups.google.com/forum/?fromgroups=#!topic/graylog2/uq__WIOxI5s

tcpdump message when test different way

1: use nc test (ok)

echo -e '{"message":"55.3.244.1 GET /index.html 15824 0.043","@version":"1","@timestamp":"2014-08-17T19:36:45.825Z","host":"gz3.test","path":"/etc/logstash/ces","client":"55.3.244.1","method":"GET","request":"/index.html","bytes":"15824","duration":"0.043"}' | nc -u 192.168.88.109 12201

0.000000 192.168.88.64 192.168.88.109 UDP 291 Source port: 34983 Destination port: 12201

{"message":"55.3.244.1 GET /index.html 15824 0.043","@version":"1","@timestamp":"2014-08-17T19:36:45.825Z","host":"gz3.test","path":"/etc/logstash/ces","client":"55.3.244.1","method":"GET","request":"/index.html","bytes":"15824","duration":"0.043"}

2: use logstash test (not ok)
[root@gz3 logstash]# cat c
55.3.244.1 GET /index.html 15824 0.043
[root@gz3 logstash]# cat c >> ces
[root@gz3 logstash]#

32.343310 192.168.88.64 192.168.88.109 UDP 261 Source port: 34000 Destination port: 12201

x..P.N.0.......k....?.{.&.......(..gM%.Wn.3;3..`;..Rd#....T...]q..[.a.....-fr......C.......W6J-.%.~V|.t_l.x&...H.....\6./..sq.7n1.F..Q7....s.....|5hI..N.O+.Ul7A....D.cW..h.L0Y.=.m...~.T.r.BjO5....

..y...r.....d._..i.

3: use nc test (not ok)

echo -e '{"message"=>"55.3.244.1 GET /index.html 15824 0.043", "@version"=>"1", "@timestamp"=>"2014-08-17T22:11:13.806Z", "host"=>"gz3.test", "path"=>"/etc/logstash/ces", "client"=>"55.3.244.1", "method"=>"GET", "request"=>"/index.html", "bytes"=>"15824", "duration"=>"0.043"}' | nc -u 192.168.88.109 12201

208.101006 192.168.88.64 192.168.88.109 UDP 310 Source port: 41607 Destination port: 12201
{"message"=>"55.3.244.1 GET /index.html 15824 0.043", "@version"=>"1", "@timestamp"=>"2014-08-17T22:11:13.806Z", "host"=>"gz3.test", "path"=>"/etc/logstash/ces", "client"=>"55.3.244.1", "method"=>"GET", "request"=>"/index.html", "bytes"=>"15824", "duration"=>"0.043"}


(system) #2