Logstash grok plugin is considering number as text

Elastic Stack Logstash
1

Dear Team,

We are using logstash that will receive logs from filebeat for sending tomcat access logs. We have configured grok to map message's column data to respective fields in elasticsearch. We have configured response as NUMBER in grok but the field is appearing as string in index mappings.

Even we tried mutate but no luck. Can someone please help?

Here is the configuration -

input {
beats {
port => "5044"
tags => ["logstash-access-log"]
}
}
filter {

mutate {
convert => { "response" => "integer" }
}

fingerprint {
source => "message"
target => "[@metadata][fingerprint]"
method => "MURMUR3"
}

grok {
    match => { "message" => "%{IP:client} %{USER:IDENT} %{USER:AUTH} \[%{HTTPDATE:DATEANDTIME}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status} %{NUMBER:bytes} %{WORD:verb} %{NUMBER:response}"}
}

}
output {
if "logstash-access-log" in [tags]{
elasticsearch {
hosts => ["MYPOCHOST:9200"]
index => "index-accesslogs-%{+YYYY.MM.dd}"
document_id => "%{[@metadata][fingerprint]}"
user => XXXXXX
password => XXXXXXX
}
}
#stdout { codec => rubydebug }
}

2

Can we get some sample of your logs to test the grok?

3

dear vikas, thanks for looking into this. Here is the sample lines of log file for your reference -

100.120.77.247 - - [13/Jun/2019:06:25:09 +0000] "GET /specialcase/v2/experiment/balance?lid=989 HTTP/1.1" 200 246 GET 32
100.120.77.177 - - [13/Jun/2019:06:25:09 +0000] "PUT /specialcase/v2/experiment/preflight?uatype=Android&uaversion=4.12.3 HTTP/1.1" 200 46 PUT 4
100.120.77.241 - - [13/Jun/2019:06:25:09 +0000] "GET /specialcase/v2/sudogroup/l7R0i2toK62qS5t6Mmglzv HTTP/1.1" 200 1095 GET 22
100.120.77.141 - - [13/Jun/2019:06:25:09 +0000] "PUT /specialcase/v2/experiment/lockAmount/1pk1srora8u9nuvz9d999cv7wk7ic49h HTTP/1.1" 202 - PUT 0
100.120.77.161 - - [13/Jun/2019:06:25:10 +0000] "GET /specialcase/v2/analytics/sudogroup/http://admin.specialcase.com:3000/admin/sudogroups/uwaUt0EUifkEy3y4Iy0RaT/threshold HTTP/1.1" 500 1866 GET 2
100.120.77.221 - - [13/Jun/2019:06:25:10 +0000] "GET /specialcase/v2/sudogroup/l7R0i2toK62qS5t6Mmglzv/trays?details=true HTTP/1.1" 200 16154 GET 105
100.120.77.189 - - [13/Jun/2019:06:25:11 +0000] "GET /specialcase/v2/sudogroup/SACIG7dh7idzUPFVFC3yiZ/trays?details=true HTTP/1.1" 200 15385 GET 107
100.120.77.188 - - [13/Jun/2019:06:25:12 +0000] "GET /specialcase/v2/sudogroup/V8LUCL9QNSkPSHSgTv9UHW/trays?details=true HTTP/1.1" 200 8239 GET 61
100.120.77.156 - - [13/Jun/2019:06:25:12 +0000] "GET /specialcase/v2/experiment/balance?lid=2&showdeactivated=true HTTP/1.1" 200 262 GET 6
100.120.77.192 - - [13/Jun/2019:06:25:12 +0000] "GET /specialcase/v3/experiment/offers?locationId=1313&type=activation_offer HTTP/1.1" 200 189 GET 4
100.120.77.245 - - [13/Jun/2019:06:25:12 +0000] "GET /specialcase/v2/app/forceupgrade?appname=Android&appversion=4120300 HTTP/1.1" 200 33 GET 1
100.120.77.140 - - [13/Jun/2019:06:25:12 +0000] "GET /specialcase/v2/sudogroup/n8jbBXblv16H2SJ881qONX/trays?details=true HTTP/1.1" 200 15303 GET 110

4

You grok is working absolutely fine..
100.120.77.247 - - [13/Jun/2019:06:25:09 +0000] "GET /specialcase/v2/experiment/balance?lid=989 HTTP/1.1" 200 246 GET 32

%{IP:client_ip} %{USER:ident} %{USER:auth} [%{HTTPDATE:apache_timestamp}] "%{WORD:method} /%{NOTSPACE:request_page} HTTP/%{NUMBER:http_version}" %{NUMBER:server_response}

Output:
{
"request_page": "specialcase/v2/experiment/balance?lid=989",
"method": "GET",
"auth": "-",
"ident": "-",
"http_version": "1.1",
"client_ip": "100.120.77.247",
"server_response": "200",
"apache_timestamp": "13/Jun/2019:06:25:09 +0000"
}

5

Your mutate+convert is before your grok, so at that point the response field does not exist, so the mutate+convert does not do anything.

6

fantastic Mr. Badger. Moved the mutate next to grok and issue fixed.

Many thanks for your support.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.