Logstash influxdb output with geoip feature

Elastic Stack Logstash
1

I'm trying to use logstash geoip filter to send data about location of IP address to InfluxDB via InfluxDB output plugin.

My logstash conf file is:

input {
  file {
    path => "/root/geoip_test.txt"
    start_position => "beginning"
  }
}

filter {
  geoip {
    source => "message"
    fields => ["latitude", "longitude"]
  }
}

output {
  stdout {codec => rubydebug}
  influxdb {
    host => "localhost"
    port => 8086
    db => "metrics"
    measurement => "geoip_test"
    codec => "json"
    use_event_fields_for_data_points => true
  }
}

geoip_test.txt file contains only one IP address:

14.143.35.10

Output with error I receive is:

[2020-09-07T12:26:26,696][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
         "geoip" => {
         "latitude" => 12.9771,
        "longitude" => 77.5871
    },
       "message" => "14.143.35.10",
          "path" => "/root/geoip_test.txt",
    "@timestamp" => 2020-09-07T10:26:33.963Z,
          "host" => "test",
      "@version" => "1"
}
[2020-09-07T12:26:34,942][WARN ][logstash.outputs.influxdb][main][941178b6897abb80f9a5f7654e5e62ba752d5e20b68781bc62b466e489c2ce56] Non recoverable exception while writing to InfluxDB {:exception=>#<InfluxDB::Error: {"error":"unable to parse 'geoip_test,host=test geoip={\"latitude\"=\u003e12.9771, \"longitude\"=\u003e77.5871},path=\"/root/geoip_test.txt\" 1599474393963': invalid boolean"}
>}

I think geoip filter generates some boolean field which InfluxDB is not able to work with.

Does anyone have any idea what to do with that? Is it possible to set up geoip filter in some way so it wouldn't generate anything but lon and lat fields?

Any help is really appreciated!

2

That looks like a valid geoip output, from an Elastic Stack point of view. If Influx isn't accepting it, you may need to adjust the format that is being sent so that it matches.

What that might look like I can't say sorry, I don't know Influx.

3

@warkolm Thanks for reply! I'm still examining the issue and what I figured out is that there might be a problem with way logstash insert the geoip information into InfluxDB.

As we can see from the correct output:

"domain3" => "***",
   "protocol" => "***",
   "response" => "200",
"domain1" => "***",
   "clientip" => "***",
   "port" => "443",
"request" => "***",
 "@timestamp" => 2020-09-08T12:34:17.186Z,
  "timestamp" => "03/Sep/2020:21:34:17 +0200",
  "bytes" => "301",
   "referrer" => "***",
"domain2" => "***",
   "verb" => "GET",
  "ident" => "-",
   "auth" => "-",
"httpversion" => "1.1"

And if we compare the output with the invalid boolean error:

      "ident" => "-",
   "referrer" => "***",
    "domain2" => "***",
   "protocol" => "https",
  "timestamp" => "03/Sep/2020:21:34:17 +0200",
    "domain1" => "***",
"httpversion" => "1.1",
       "port" => "443",
      "bytes" => "301",
   "clientip" => "***",
   "response" => "200",
 "@timestamp" => 2020-09-08T12:40:45.042Z,
        "lon" => {
    "longitude" => 15.775
},
    "request" => "***",
       "verb" => "GET",
       "auth" => "-",
    "domain3" => "***"

Look at the lon (which stands for geoip) it is writted like an "array". Isn't there any way to change that format? So it could be inserted as something like longitude => 15.775

4

The first event doesn't have geo info that I can see? The second one only has a longitude, which is odd.

It might be worth raising an issue on the output plugin's GitHub repo.

5

Can I suggest you remove these two and try something like

data_points => {
    "lon" => "%{[geoip][longitude]}"
    "lat" => "%{[geoip][latitude]}"
}

Add "path" => "%{path}" to the hash if you want that field in the db.

2 Likes
6

Yeah I limited the geoip only for longitude so I can see what it's doing...

The very first solution which helped me to tackle the problem was add_filed with the information I got from the geoip plugin and then remove the information with mutate filter feature.

  geoip {
    source => "clientip"
    target => "lat"
    fields => "latitude"
  }

  geoip {
    source => "clientip"
    fields => ["longitude", "latitude"]
    add_field => {"longitude" => "%{[geoip][longitude]}"}
  }

  mutate {
  convert => {
   "lon" => "string"
  }
  remove_field => ["message", "host", "@version", "path", "lat", "geoip"]

  }

}

I would like to make it a tag though so I can visualize it with Grafana...

8

I'm sharing output of my "final" .conf file with geoip in filter part in case someone would find it useful :slight_smile:

  geoip {
    source => "clientip"
    add_field => {"longitude" => "%{[geoip][longitude]}"
                  "latitude"  => "%{[geoip][latitude]}"
                  "timezone"  => "%{[geoip][timezone]}"
                  "ip"  => "%{[geoip][ip]}"
                  "continent_code"  => "%{[geoip][continent_code]}"
                  "region_name"  => "%{[geoip][region_name]}"
                  "postal_code"  => "%{[geoip][postal_code]}"
                  "country_name"  => "%{[geoip][country_name]}"
                  "region_code"  => "%{[geoip][region_code]}"
                  "country_code3"  => "%{[geoip][country_code3]}"
    }
  }


}

output {
  stdout {codec => rubydebug}
  influxdb {
    host => "localhost"
    port => 8086
    db => "metrics"
    measurement => "test"
    codec => "json"
    use_event_fields_for_data_points => true
    send_as_tags => ["response", "host", "domain3", "domain2", "domain1", "referrer", "port", "protocol"]
    exclude_fields => ["@timestamp","timestamp","timestamp_object","timestamp_local","message","@version","geoip"]
  }
}
1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.