Hi,
I have been trying to turn a JSON blob keys, which I receive from input into data fields but I have been unsuccessful for some hours.
JSON Blob fed to input
{
"timestamp": "[16/Feb/2018:19:19:03 +0000]",
"@version": "1",
"clientip": "127.0.0.1",
"url": "/test",
"code": "200",
"method": "GET",
"size": "12",
"bytes_rcvd": "135",
"bytes_sent": "395",
"country": "test",
"domain": "test.com",
"customer": "john_beast",
"user": "beast",
"cache": "HIT",
"ttfb": "0.002457",
"referer": "-",
"user-agent": "curl/7.47.0",
"x-forwarded-for": "x.x.x.x",
"error": "-"
}
Logstash 1
Filter
filter {
json {
source => "message"
}
mutate {
remove_field => [ "message", "path", "timestamp"]
}
date {
match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
}
}
Output plugin
tcp {
id => "es-output"
host => "x.x.x.x"
port => 80
mode => "client"
codec=> "json_lines"
}
Logstash 2
input {
tcp {
mode => "server"
port => 31311
}
}
output {
elasticsearch {
hosts => ["http:/x.x.x.x1:9200"]
}
}
ES Output
{
"_index": "logstash-2018.02.16",
"_type": "doc",
"_id": "wQcAoGEBkMC_ERzMJgWb",
"_version": 1,
"_score": null,
"_source": {
"message": "{\"country\":\"test\",\"referer\":\"NULL\",\"code\":\"200\",\"user\":\"beast\",\"ttfb\":\"0.002468\",\"error\":\"NULL\",\"clientip\":\"127.0.0.1\",\"@version\":\"1\",\"host\":\"test-server\",\"cache\":\"hit\",\"method\":\"GET\",\"x-forwarded-for\":\"x.x.x.x\",\"bytes_sent\":\"397\",\"url\":\"/test\",\"@timestamp\":\"2018-02-16T19:03:39.820Z\",\"bytes_rcvd\":\"135\",\"size\":\"12\",\"domain\":\"test.com\",\"customer\":\"john_beast\",\"user-agent\":\"curl/7.47.0\"}",
"@version": "1",
"host": "gcloud-123df2tr43g34g3",
"port": 1264,
"@timestamp": "2018-02-16T19:03:39.829Z"
},
"fields": {
"@timestamp": [
"2018-02-16T19:03:39.829Z"
]
},
"sort": [
1518807819829
]
}
Could you help me turn each message key into field data? I have tried many solutions from forum but no success yet.
Thanks very much.
Kind regards,
John