I've been trying for quite some time to get this to work with no luck - I'm using the Packetbeat plugin to record http data (and headers).
At present it's getting them currently - however the host is nested within the JSON. I'm trying to pull out the hostname from http.request.headers using the following filter.
filter {
if [type] == "http" {
mutate {
add_field => { "hostname" => "%{[http][request][headers][host]}" }
}
}
}
I'm trying to get the information from
"http": {
"request": {
"headers": {
"content-length": 0,
"host": "www.google.com",
"connection": "Keep-Alive",
"accept-encoding": "identity",
"user-agent": "Wget/1.17.1 (linux-gnu)",
"accept": "*/*"
},
How would i go about filtering this? Currently seemed to have tried every combination i can find
I'm getting the error below - I'm wondering if it's not working as the field it's trying to split isn't being split by Packetbeat correctly?
[2016-11-21T17:37:09,607][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"packetbeat-2016.11.21", :_type=>"http", :_routing=>nil}, 2016-11-21T17:37:08.115Z ip-10-0-193-193 %{message}], :response=>{"index"=>{"_index"=>"packetbeat-2016.11.21", "_type"=>"http", "_id"=>"AViH9oPTs5hNkRVWqo3X", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [hostname] tried to parse field [hostname] as object, but found a concrete value"}}}}
Full JSON below in case that helps
{
"_index": "packetbeat-2016.11.21",
"_type": "http",
"_id": "AViH0R2Is5hNkRVWqnMP",
"_score": null,
"source": {
"server": "",
"proc": "",
"method": "GET",
"bytes_in": 141,
"query": "GET /",
"ip": "XXX",
"type": "http",
"client_proc": "",
"tags": [
"beats_input_raw_event"
],
"path": "/",
"client_port": 58206,
"client_server": "",
"@timestamp": "2016-11-21T16:56:17.116Z",
"bytes_out": 10927,
"port": 80,
"beat": {
"hostname": "ip-10-0-193-193",
"name": "xxx",
"version": "5.0.1"
},
"@version": "1",
"host": "ip-10-0-193-193",
"http": {
"request": {
"headers": {
"content-length": 0,
"host": "www.google.com",
"connection": "Keep-Alive",
"accept-encoding": "identity",
"user-agent": "Wget/1.17.1 (linux-gnu)",
"accept": "/"
},
"params": ""
},
"response": {
"headers": {
"date": "Mon, 21 Nov 2016 16:56:17 GMT",
"content-length": 10261,
"server": "gws",
"expires": "-1",
"transfer-encoding": "chunked",
"vary": "Accept-Encoding",
"x-frame-options": "SAMEORIGIN",
"p3p": "CP="This is not a P3P policy! See P3P and Google's cookies - Google Account Help for more info."",
"set-cookie": "NID=91=GjjAcot9wLIWyrPiLTSGi8GSQYX1XOiAlwBpge-0VpmJDJFB-lk5HpjhacImtvCSS89SlTJyAk-nCmoioCaRLOg6it6eXiuKYcsj5CVeMQ895XAlzz-QjPrN9ZhaVXjgoDHMoMy1c1gamk; expires=Tue, 23-May-2017 16:56:17 GMT; path=/; domain=.google.com; HttpOnly",
"x-xss-protection": "1; mode=block",
"content-type": "text/html; charset=ISO-8859-1",
"cache-control": "private, max-age=0",
"accept-ranges": "none"
},
"code": 200,
"phrase": "OK"
}
},
"responsetime": 38,
"client_ip": "10.0.193.193",
"status": "OK",
"direction": "out"
},
"fields": {
"@timestamp": [
1479747377116
]
},
"highlight": {
"type": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@"
]
},
"sort": [
1479747377116
]
}