Nginx Grok expressions do not match field value


#1

I have two questions I really need help with. According to https://grokdebug.herokuapp.com/, my grok expression matches my log line. But Kibana shows an error message saying that my Grok expression doesn't match the field value and then shows the line that doesn't match. Here's my pipeline config file including the grok filter, an example from the nginx log and the error that is showing on Kibana under the "Discover" option for the entry:

Pipeline config:

input {
  beats {
    port => 5044
  }
}
filter {
  if [fileset][module] == "nginx" {
    if [fileset][name] == "access" {
      grok {
        match => { "message" => ["%{IPORHOST:[nginx][access][host]} %{IP:[nginx][access][clientip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} %{NUMBER:[nginx][access][request_time]} %{DATA:[nginx][access][upstream_time]} %{DATA:[nginx][access][pipe]} \"%{DATA:[nginx][access][agent]}\" \"%{DATA:[nginx][access][forwarded_for]}\" \"%{DATA:[nginx][access][referrer]}\""] }
        remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
          target => "[nginx][access][user_agent]"
          remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][clientip]"
        target => "[nginx][access][geoip]"
      }
    }
  }
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Example from nginx log:

samplehost.com 174.111.222.211 - - [09/Dec/2017:12:11:50 -0500] "GET /selector.php?selector_part=css HTTP/1.0" 200 437 0.057 0.056 . "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" "-" "https://samplehost.com/prompt.php"

And the error on the Kibana Discover "error.message"

Provided Grok expressions do not match field value: [samplehost.com 174.111.222.211 - - [09/Dec/2017:12:11:50 -0500] "GET /selector.php?selector_part=css HTTP/1.0" 200 437 0.057 0.056 . "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12" "-" "https://samplehost.com/prompt.php"]

Why am I getting this Grok error?

In addition to this, my Nginx Access and Error Logs Dashboard that filebeats configured is not showing me any data like the geoip, Top pages, response codes over time, etc. Is this due to the grok filter not matching or is this a different issue?


#2

So I solved my problem with the Grok filter. I had to disable elasticsearch output and host in the filebeats.yml and then enable logstash output and host:

#output.elasticsearch:
#  hosts: ["localhost:9200"]

output.logstash:
  hosts: ["localhost:5044"]

Once I did that, I was able to see logstash outputting the correct data and formats in stdout. But I still didn't see those fields on Kibana's "Discover" tab for the data after deleting the filebeat data registry and sending a curl request to delete the elasticsearch index for the data. I had to go to Kibana -> Management -> Indexes Patterns and then click on the refresh button to refresh the indexes. Then on the Discover tab, I clicked on the settings gear next to filters and unchecked "Hide unavailable filters" and the new indexes were there. However, my data is still not populating these indexes correctly. Any idea why?


#3

Ok I ironed this out myself and found out what was going on. I had to make sure the logs that needed to be parsed were in both filebeats.yml AND in modules.d/nginx.yml for filebeats. Once I did that, I deleted the indexes, removed the filebeats data registry again and started up filebeats. I refreshed the indexes on Kibana to be safe and then my data was showing correctly.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.