User_agent and geoip filters does not work


(jonasv) #1

Hey,

I'm got a field which looks like this:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 OPR/42.0.2393.94
or this:
curl/7.47.0

filter {

mutate {
  remove_tag => ["beats_input_codec_plain_applied"]
}

if[type] == "nginx-access" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {"message" => "%{NGINXACCESS}"}
}

useragent {
    source => "useragent"
    prefix => "browser"
}
geoip {
    source => "request_ip"
}

}

mutate {
remove_field => ["message", "fields", "beat.hostname", "beat.name", "host"]
gsub => ["request_arrival_time", "(.{6}$)", ""]
}

}

Once I send new data - nothing changes in output, I see same fields and data just like if there were user_agent and geoip filters. What I'm doing wrong here ?

thanks


(Magnus Bäck) #2

Please show the result of a stdout { codec => rubydebug } output so we can see what the event really looks like.


(jonasv) #3

Magnus, could you please be more specific where to put that line ?


(jonasv) #4

I get PluginLoadingError: Couldn't find any filter plugin named 'stdout'.


(Magnus Bäck) #5

Magnus, could you please be more specific where to put that line ?

Anywhere inside an output { } block.

I get PluginLoadingError: Couldn't find any filter plugin named 'stdout'.

I was sure that plugin was shipped with Logstash. But if it isn't, install it with the usual method.


(jonasv) #6

Thanks, I've put it into output block, restarted logstash, sent the data, but now what ? What should I see and most importantly - where ? Because I see no changes (not even in logstash/elk log files).


(jonasv) #7

This is my output.conf

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["10.1.2.129:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}


(Magnus Bäck) #8

You should start seeing events in /var/log/logstash/logstash.stdout or wherever stdout ends up for you. If you run Logstash interactively it's going to end up in your terminal.


(jonasv) #9

Thank you, you are right, here is the output you requested:

Sending logstash logs to /var/log/logstash/logstash.log.
{
"@version" => "1",
"@timestamp" => "2017-02-07T13:05:16.579Z",
"input_type" => "log",
"tags" => [
[0] "project-X",
[1] "web-server",
[2] "beats_input_codec_plain_applied"
],
"count" => 1,
"beat" => {
"hostname" => "deb.int.team.com",
"name" => "deb"
},
"source" => "/var/log/nginx/elastic_access.log",
"offset" => 4672,
"type" => "nginx-access",
"request_arrival_time" => "07/Feb/2017:13:05:14",
"request_ip" => "10.1.2.129",
"our_server" => "deb.int.team.com:8181",
"request" => "GET /test_dev/151 HTTP/1.1",
"request_response_code" => "404",
"request_body_bytes_sent" => "233",
"user_agent" => "curl/7.47.0",
"referer" => "-",
"request_processing_time" => "0.009",
"upstream_response_time" => "0.009",
"request_length" => "108"
}


(jonasv) #10

One more:

{
"@version" => "1",
"@timestamp" => "2017-02-07T13:16:21.649Z",
"input_type" => "log",
"count" => 1,
"source" => "/var/log/nginx/elastic_access.log",
"offset" => 4830,
"type" => "nginx-access",
"beat" => {
"hostname" => "deb.int.team.com",
"name" => "deb"
},
"tags" => [
[0] "project-X",
[1] "web-server",
[2] "beats_input_codec_plain_applied"
],
"request_arrival_time" => "07/Feb/2017:13:16:20",
"request_ip" => "10.100.2.2",
"our_server" => "deb.int.team.com:8181",
"request" => "GET /test_dev/160 HTTP/1.1",
"request_response_code" => "404",
"request_body_bytes_sent" => "209",
"user_agent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 OPR/42.0.2393.94",
"referer" => "-",
"request_processing_time" => "0.006",
"upstream_response_time" => "0.006",
"request_length" => "432"
}


(Magnus Bäck) #11

Your useragent filter doesn't work since it's trying to parse a field that doesn't exist (user_agent vs. useragent).

Your geoip filter doesn't work because you can't geolocate RFC1918 addresses like 10.100.2.2.


(jonasv) #12

Thank you very much Magnus for helping me solving this!


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.