Elb logs grok inside grok

[bob-bza]

This was my filter looks like for ELB log.
filter {
grok {

** match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} "(?:%{GREEDYDATA:elb_request}|-)" "(?:%{GREEDYDATA:userAgent}|-)" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}" ]**
}
}

From this i was trying to get only if elb_status_code = 400 then only i want to copy my message to ES and only with few fields. i don't want all fields also.
Ex:
filter {
grok {

** match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} "(?:%{GREEDYDATA:elb_request}|-)" "(?:%{GREEDYDATA:userAgent}|-)" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}" ]**
}
if [elb_status_code] =~ "400" {

grok { match => [ "elb_status_code", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_sslprotocol}" ]
** }**
}
}
is anything wrong in this, this was not working and i was getting my full log to ES.???????

[jkuang (Jimmy Kuang)]

Can you provide a few lines of the log?

[bob-bza]

@jkuang

Can you provide a few lines of the log?

2017-03-18T10:14:32.584347Z tf-lb-06d292ba3da94e1d307 64.3.9.38:40979 96.1.1.183:8080 0.00002 0.001577 0.000017 404 404 0 0 "GET https://96.1.2.0:443/index.htm HTTP/1.1" "-" ECDHE-RSA-AES12HA256 TLSv1.2

[jkuang (Jimmy Kuang)]

It looks like the ** are the problem. Please use the grok below:

filter {

grok {
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} \"(?:%{GREEDYDATA:elb_request}|-)\" \"(?:%{GREEDYDATA:userAgent}|-)\" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}"]  }

if [elb_status_code] =~ "400" {

grok { match => [ "elb_status_code", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_sslprotocol}" ]}

}
}

[bob-bza]

@jkuang

filter {

grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} "(?:%{GREEDYDATA:elb_request}|-)" "(?:%{GREEDYDATA:userAgent}|-)" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}"] }

if [elb_status_code] =~ "400" {

grok { match => [ "elb_status_code", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_sslprotocol}" ]}

}
}

I was using same it dint worked as expected as i shown above. these ** i was trying to bold those wile posting in elastic discussions . so starts was added.

[jkuang (Jimmy Kuang)]

What is not working? Please elaborate. Since I only have one line of logging, and it passes through the filter into ES I don't see where the problem is.