Hello, everyone!
I logstash, i use dissect plugin to split nginx log, then use convert_datatype to convert filed data type. But in my nginx log, the request_time and upstream_response_time fields will be '-' sometimes, not only the float type. So when i use convert_datatype to convert these to fields the logstash will throw the following exception, and then logstash will give the string value(the original string '-') to the request_time and upstream_response_time fields, what i want is that when convert data type failed, give these two fields a default number value such as 0.000 not '-'(a string value):
[2017-08-05T22:28:29,551][WARN ][logstash.filters.dissect ] Dissector datatype conversion, value cannot be coerced, key: request_time, value: -
[2017-08-05T22:28:29,551][WARN ][logstash.filters.dissect ] Dissector datatype conversion, value cannot be coerced, key: upstream_response_time, value: -
My nginx log format:
%{time_local}|%{server_ip}|%{request}|%{status_code}|%{remote_user}|%{remote_addr}|%{http_user_agent}|%{http_referer}|%{host}|%{bytes_sent}|%{request_time}|%{upstream_response_time}|%{upstream_addr}|%{connection}|%{connection_requests}|%{uuid}
An example log(Take care the request_time and upstream_response_time fields are '-'):
05/Aug/2017:22:22:33 -0700|54.153.101.30|GET /listing/detail/1864252/MO/St-Louis/www HTTP/1.1|200|-|5.255.250.132|Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)|-|env1-2.chime.me|76627|-|-|127.0.0.1:10300|4483145|2|38873227-aa0e-412b-b93f-13a7d0e26fb7
My logstash config(just debug):
input {
beats {
port => 5044
}
}
filter {
ruby {
code => "
event.timestamp.time.localtime
tstamp = event.get('@timestamp').to_i
Time.at(tstamp).strftime('%Y-%m-%d')
"
}
dissect {
mapping => {
"message" => "%{time_local}|%{server_ip}|%{request}|%{status_code}|%{remote_user}|%{remote_addr}|%{http_user_agent}|%{http_referer}|%{host}|%{bytes_sent}|%{request_time}|%{upstream_response_time}|%{upstream_addr}|%{connection}|%{connection_requests}|%{uuid}"
}
convert_datatype => {
status_code => "int"
bytes_sent => "int"
request_time => "float"
upstream_response_time => "float"
}
}
}
output {
if [business] == "nginx" and [type] == "access" {
stdout { codec => rubydebug }
}
}
So, when i run logstash with above config, the result i get as following(the request_time and upstream_response_time fields are string value not number):
{
"request" => "GET /listing/detail/1864252/MO/St-Louis/www HTTP/1.1",
"status_code" => 200,
"upstream_addr" => "127.0.0.1:10300",
"connection_requests" => "2",
"source" => "/home/ec2-user/nginx/logs/access.log",
"type" => "access",
"uuid" => "38873227-aa0e-412b-b93f-13a7d0e26fb7",
"http_user_agent" => "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)",
"remote_user" => "-",
"request_time" => "-",
"@version" => "1",
"beat" => {
"hostname" => "awsuw7-50.opi.com",
"name" => "awsuw7-50.opi.com",
"version" => "5.5.1"
},
"host" => "env1-2.chime.me",
"server_ip" => "54.153.101.30",
"connection" => "4483145",
"remote_addr" => "5.255.250.132",
"offset" => 270,
"business" => "nginx",
"input_type" => "log",
"time_local" => "05/Aug/2017:22:22:33 -0700",
"message" => "05/Aug/2017:22:22:33 -0700|54.153.101.30|GET /listing/detail/1864252/MO/St-Louis/www HTTP/1.1|200|-|5.255.250.132|Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)|-|env1-2.chime.me|76627|-|-|127.0.0.1:10300|4483145|2|38873227-aa0e-412b-b93f-13a7d0e26fb7",
"bytes_sent" => 76627,
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_dataconversionuncoercible_request_time_float",
[2] "_dataconversionuncoercible_upstream_response_time_float"
],
"@timestamp" => 2017-08-06T05:28:24.508Z,
"http_referer" => "-",
"upstream_response_time" => "-"