How can i create a template to handle data inbound from my logs for an Array of IP addresses. I just can't seem to quite figure this out. The rest i have got down. I have all my numbers at numbers, float, integer, string, all working great.
But we have NGINX and APACHE logs coming in from websites that are fronted by PROXY and by CACHE services like Verizon and Cloudflare. So the client IP is always going to be the proxy ip.
The true client ip is in a field called http_x_forwarded_for, but for each proxy or ELB it may pass through, that header field gets appended. So i and up with an array of ips in that filed.
EXAMPLE: [ "1.2.3.4", "111.222.333.444", "123.456.789.001" ] sometimes there are two , sometimes there are 3, sometimes there is only 1. but it is always formated as a json array as it comes into elasticsearch.
In logstash, i can access them and mutate, etc with [@fields][http_x_forwarded_for][0] to get the first element value and so on so the json array is valid and seems to be parsed out correctly. I send the data to Elasticsearch as JSON already since my logs are already in JSON format natively. I have a template all setup with @fileds, and the properties under that, and then each field with a type, (string, integer, float, etc) and if i want analyzed or not, taht is all great.
When i get to the property for http_x_forwarded_for, i assume i just go down a level but is that type OBJECT or type NESTED, or is it just ARRAY and how can i have that be elements of type IP so that in kibana i can search and aggregate and find IP addresses , counts, and such and use these values. ?? i just can't seem to find the right format for the template to make that work.
any help to point me in the right direction would be appreciated.
Example: here is one row from our log file from the web server. logstash is picking up this row from the log and sending it to elasticsearch. all the fields are working except http_x_forwarded_for where i want to capture the IP addresses somehow to use in kibana.
{
"@timestamp": "2016-01-04T04:18:53-05:00",
"eventTime": "2016-01-04T04:18:53-05:00",
"@fields": {
"remote_addr": "173.245.50.146",
"remote_user": "-",
"request": "GET /product/test HTTP/1.1",
"request_method": "GET",
"status": "200",
"body_bytes_sent": "35110",
"request_time": "0.183",
"http_referrer": "-",
"http_user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:41.0) Gecko/20100101 Firefox/41.0",
"http_x_forwarded_for": ["104.245.34.88", "173.245.50.146"],
"http_cfray": "25f5dbd050cb095c-DFW",
"host": "www.host.com",
"hostname": "farm1.www.host.com"
}
}