GeoIP - Elasticsearch

Elastic Stack Elasticsearch
1

Hi: I have an Elasticsearch installed in Ubuntu. What are the proper steps to setup Geoip so that every time I ingest data it is enriched with geolocation data?

Thanks

2

Hi @DFIR_Cap , welcome to our community.

How are you ingesting you data? from query, logstash, beats...

3

For now Logstash, but it will eventually be both logstash and beats.

4

I notice there is no ingest pipeline

 curl -XGET http://localhost:9200/_ingest/pipeline/geoip?pretty
{ }

But there is the cluster setting for GEOIP Download. How do I create the pipeline? Or am I missing a step?

 curl -XGET http://localhost:9200/_cluster/settings?pretty
{
  "persistent" : {
    "ingest" : {
      "geoip" : {
        "downloader" : {
          "eager" : {
            "download" : "true"
          }
        }
      }
    }
  },
5

Where are you parsing your data?

Since you are using Logstash, are you parsing your data in logstash?

6

My idea is the full ELK Stack. Elasticsearch, Logstash, Kibana, Filebeat. At the moment, I only have Logstash and Elasticsearch installed. I have been just testing some ingests using logstash directly to elasticsearch.

Yes, parsing in logstash.

7

In logstash you can use the geoip filter to get geoip information from public ip address.

The documentation can be found here.

You would need to have your ip field and use the filter this way:

	geoip {
		source => "source_ip_field"
		target => "destination_field"
	}

Keep in mind that for this to work without any issues in Elasticsearch and Kibana you will need a template for your index and to map the destination field correctly as geo point as explained in this documentation.

1 Like
8

To illustrate, I just ran this example and got this response:

input {
  generator {
    message => "34.107.161.234"
    count => 1
  }
  stdin {}
}

filter {
  geoip {
    database => "/Users/alexsalgado/Desktop/elastic/observability/logstash/logstash-8.14.3/data/geoip_database_management/1720790685/GeoLite2-City.mmdb"
    source => "message"
    target => "source"  # Change target to "source" or another valid option
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

result


{
       "message" => "34.107.161.234",
      "@version" => "1",
         "event" => {
        "sequence" => 0,
        "original" => "34.107.161.234"
    },
    "@timestamp" => 2024-07-12T13:59:54.780095Z,
          "host" => {
        "name" => "alexs-MacBook-Pro.local"
    },
        "source" => {
          "ip" => "34.107.161.234",
        "mmdb" => {
            "dma_code" => 616
        },
         "geo" => {
                 "region_name" => "Missouri",
                    "location" => {
                "lat" => 39.1027,
                "lon" => -94.5778
            },
                   "city_name" => "Kansas City",
             "region_iso_code" => "US-MO",
                    "timezone" => "America/Chicago",
              "continent_code" => "NA",
                "country_name" => "United States",
                 "postal_code" => "64184",
            "country_iso_code" => "US"
        }
    }
}
9

nice ok. and if I am going to let Elastic do the parsing? How would I set geoip up?

10

Unless you are using an Elastic Agent integration or Filebeat module, elasticsearch will not parse anything by default.

I would need to create an ingest pipeline in Elasticsearch, parse your data in it and then use the geoip processor to add geoip information.

For example, this documentation explains a little more about ingest pipelines and this is the documentation about the geoip processor.

It is pretty similar to what you have in a Logstash pipeline, but it is executed in Elasticsearch.

1 Like
11

Thanks very much for the info.