User_agent and geoip filters does not work

Elastic Stack Logstash
1

Hey,

I'm got a field which looks like this:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 OPR/42.0.2393.94
or this:
curl/7.47.0

filter {

mutate {
  remove_tag => ["beats_input_codec_plain_applied"]
}

if[type] == "nginx-access" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => {"message" => "%{NGINXACCESS}"}
}

useragent {
    source => "useragent"
    prefix => "browser"
}
geoip {
    source => "request_ip"
}

}

mutate {
remove_field => ["message", "fields", "beat.hostname", "beat.name", "host"]
gsub => ["request_arrival_time", "(.{6}$)", ""]
}

}

Once I send new data - nothing changes in output, I see same fields and data just like if there were user_agent and geoip filters. What I'm doing wrong here ?

thanks

2

Please show the result of a stdout { codec => rubydebug } output so we can see what the event really looks like.

3

Magnus, could you please be more specific where to put that line ?

4

I get PluginLoadingError: Couldn't find any filter plugin named 'stdout'.

5

Magnus, could you please be more specific where to put that line ?

Anywhere inside an output { } block.

I get PluginLoadingError: Couldn't find any filter plugin named 'stdout'.

I was sure that plugin was shipped with Logstash. But if it isn't, install it with the usual method.

6

Thanks, I've put it into output block, restarted logstash, sent the data, but now what ? What should I see and most importantly - where ? Because I see no changes (not even in logstash/elk log files).

7

This is my output.conf

output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["10.1.2.129:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

8

You should start seeing events in /var/log/logstash/logstash.stdout or wherever stdout ends up for you. If you run Logstash interactively it's going to end up in your terminal.

9

Thank you, you are right, here is the output you requested:

Sending logstash logs to /var/log/logstash/logstash.log.
{
"@version" => "1",
"@timestamp" => "2017-02-07T13:05:16.579Z",
"input_type" => "log",
"tags" => [
[0] "project-X",
[1] "web-server",
[2] "beats_input_codec_plain_applied"
],
"count" => 1,
"beat" => {
"hostname" => "deb.int.team.com",
"name" => "deb"
},
"source" => "/var/log/nginx/elastic_access.log",
"offset" => 4672,
"type" => "nginx-access",
"request_arrival_time" => "07/Feb/2017:13:05:14",
"request_ip" => "10.1.2.129",
"our_server" => "deb.int.team.com:8181",
"request" => "GET /test_dev/151 HTTP/1.1",
"request_response_code" => "404",
"request_body_bytes_sent" => "233",
"user_agent" => "curl/7.47.0",
"referer" => "-",
"request_processing_time" => "0.009",
"upstream_response_time" => "0.009",
"request_length" => "108"
}

10

One more:

{
"@version" => "1",
"@timestamp" => "2017-02-07T13:16:21.649Z",
"input_type" => "log",
"count" => 1,
"source" => "/var/log/nginx/elastic_access.log",
"offset" => 4830,
"type" => "nginx-access",
"beat" => {
"hostname" => "deb.int.team.com",
"name" => "deb"
},
"tags" => [
[0] "project-X",
[1] "web-server",
[2] "beats_input_codec_plain_applied"
],
"request_arrival_time" => "07/Feb/2017:13:16:20",
"request_ip" => "10.100.2.2",
"our_server" => "deb.int.team.com:8181",
"request" => "GET /test_dev/160 HTTP/1.1",
"request_response_code" => "404",
"request_body_bytes_sent" => "209",
"user_agent" => "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 OPR/42.0.2393.94",
"referer" => "-",
"request_processing_time" => "0.006",
"upstream_response_time" => "0.006",
"request_length" => "432"
}

11

Your useragent filter doesn't work since it's trying to parse a field that doesn't exist (user_agent vs. useragent).

Your geoip filter doesn't work because you can't geolocate RFC1918 addresses like 10.100.2.2.

12

Thank you very much Magnus for helping me solving this!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.