AWS ELBv2 extract latitude, longitude and requests subparts

[jensolsson.se]

Hi

I just for the first time set up kibana, elasticsearch and logstash to index logs from ELBv2 on AWS. It looks like this in the configuration and works quite well:

input {
    file {
        path => "/home/jensolsson/aws/**/*.gz"
        type => "elb"
        mode => "read"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        max_open_files => 15000
    }
}
filter {
    if [type] == "elb" {
        grok {
            match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int} %{IP:backend_ip}:%{NUMBER:backend_port:int} %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{NUMBER:elb_status_code:int} %{NUMBER:backend_status_code:int} %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} %{QS:request}" ]
        }
        date {
            match => [ "timestamp", "ISO8601" ]
        }
    }
}
output {
    elasticsearch { hosts => ["localhost:9200"] }
}

Now I have two issues that I am trying to solve.

1. The request is stored in a big string. I would really like to have the HTTP verb, the path, etc stored in different strings.
2. I have a lot of URLs with latitude and longitude parameters and I would like to extract these so that I can see them in Maps in Kibana. I have googled a lot for this and found some examples but I cannot understand how I can do this. Lets say the url contains latitude=63.123&longitude=13.123 and I would like to extract these to make a Point object so that it will be plotted in Kibana, I still want to log the request as usual but in addition to this I would like to plot it on the map.

Any help appreciated
Jens

[Badger]

1. Use a second grok to parse the [request] field

2. Once you have parsed the query parameters out of the request, use a kv filter to parse them

3. Make certain that your index template sets the field type as geo_point. The default template includes an example of how to do that.