Logstash config for Haproxy and Nginx not working together

Elastic Stack Logstash
1

I am having a ELK setup for processing haproxy and nginx logs, for this i have used separate config files for logstash, the main data which i want from logs are the "content url" and the "response time", in haproxy the responsetime is in milliseconds like 1345 and in nginx the response time is in seconds like 1.23. In order to bring the response time in same format i changed the haproxy response time to seconds using ruby plugin in logstash. And i m getting the desired results from both when ran individually, in kibana also i changed the response time field to duration on which input is in seconds and output also in seconds. But when i run both configs together the response time for ngnix logs returns 0.000 value and i can see tag of "_grokparsefailure" in json response, but when i run the ngnix config individually to debug it everything works fine, in kibana dashboard i can see proper response time values.

Below is the config for my Nginx logstash Config:

input {
  beats {
    port => 5045
  }
 }


filter {
 grok {
        match => { "message" => "%{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:content} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} %{NUMBER:response_bytes:int} \"-\" \"%{GREEDYDATA:junk}\" %{NUMBER:response_time}"}
 }
        date {
                match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
        }

}
output {
 elasticsearch {
   hosts => ["localhost:9200"]
  }
}

Below is the config of my Haproxy logstash config:

input {
  beats {
    port => 5044
  }
 }

filter {
  grok {
    match => { "message" => "%{MONTH:month}  %{MONTHDAY:date} %{TIME:time} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[timestamp]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{NUMBER:[response_bytes]:int} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{URIPATHPARAM:[content]} HTTP/%{NUMBER:[http_version]}" }

  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  ruby {
  code => "event.set('response_time', event.get('response_time').to_f / 1000)"
 }
 }

output {
  elasticsearch { hosts => ["localhost:9200"] }
        stdout {
                codec => rubydebug
        }
  }

Below is the log format of Nginx:

104.225.246.210 - - [09/Mar/2020:19:06:07 +0000] "GET /felaapp/hd_videos/transformers.mp4 HTTP/1.1" 206 111101 "https://aws.adcontentamtsolutions.net/felaapp/hd_videos/transformers.mp4" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Mobile/15E148 Safari/604.1" 0.241 - .

Below is the log format of my haproxy:

Feb 22 21:17:32 ap haproxy[1235]: 10.172.80.45:32071 10.31.33.34:44541 10.31.33.34:32772 13.127.229.72:443 [22/Feb/2020:21:17:32.006] this_machine~ backend_test-ron/test-ron_32772 40/0/5/1/836 200 701381 - - ---- 0/0/0/0/0 0/0 {testprod.net} {cache_hit} "GET /ob/720/output00007.ts HTTP/1.1"

I m suspecting the response_time pattern ie %{NUMBER:[response_time]} in haproxy and nginx is creating problem. Don't know what is causing this issue tried every possible thing.

2

If you are having problems getting the grok pattern to match then read this.

3

My gork pattern are getting matched when i run individual nginx config file by stopping the logstash service first, but when i run the logstash the response time for nginx shows value of 0.000 and shows "_grokparsefailure" tag, dont know what is getting missed when ran the config file individually and running all config files together.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.