# Getting geohash / geoip data from apache

**URL:** <https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455>\
**Category:** Elasticsearch\
**Created:** [December 12, 2018, 1:10am UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455 "2018-12-12T01:10:02Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 1:10am UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/1 "2018-12-12T01:10:03Z")

</div>

Im mining apache2 logs using filebeat. Data is being put into indices 'apache2-' via logstash.

If I look at the data using the 'discover' function I can see 'geoip.location.lat/lon' and 'location.lat/lon' but attempts to use this data in kibana are fruitless. Creating a map visualization gives me the error message "No Compatible Fields: The "apache2\*" index pattern does not contain any of the following field types: geo\_point".

What do I need to do with my data to use it this way? Have I made a configuration error along the line somewhere?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 12, 2018, 1:16am UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/2 "2018-12-12T01:16:33Z")

</div>

What is the mapping for the index in question?

---

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 2:09am UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/3 "2018-12-12T02:09:09Z")

</div>

Here's one:

```
      "geoip" : {
        "location": {
          "properties": {
            "lat": {
              "type": "float"
            },
            "lon": {
              "type": "float"
            }
          }
        },
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 12, 2018, 2:50am UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/4 "2018-12-12T02:50:06Z")

</div>

Ok, you need those combined and mapped as a single `geopoint`.

Are you using the geoip filter in Logstash? What does the config look like?

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 12, 2018, 3:09am UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/5 "2018-12-12T03:09:02Z")

</div>

That mapping is incorrect. location would be a property of geoip. Not only that, but as @warkolm said...that needs to be collapsed into a geo\_point type.

```
      "geoip" : {
        "dynamic" : true,
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "location" : {
            "type" : "geo_point"
          },
          "latitude" : {
            "type" : "half_float"
          },
          "longitude" : {
            "type" : "half_float"
          }
        }
      } 

```

if you're using the geoip filter in logstash, this will work. What may be the easiest thing to do is just copy the logstash index template and create a new template that matches your index pattern. i.e.

Using the Dev console

> GET \_template/logstash

copy the json output. Edit the index\_patterns setting to be something like "apache2-\*" to match your new index, then

> PUT \_template/logstash-apache

paste the copied json with new index pattern directly below the PUT and run it.

Now when a new apache2-\* index is created, it should use this mapping and be ready to go if you're using the geoip filter in Logstash

---

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 12:26pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/6 "2018-12-12T12:26:28Z")

</div>

Ok! I followed your (great) instructions and created a new template. (BTW --\> I looked high and low in the online documentation and didn't find this information.. hint). Once it was created I took the further step of deleting the existing apache2 logs from elastic and kibana.

A new log entry for apache2 was promptly created but it doesn't seem to have the appropriate data type:

```
      "geoip" : {
        "properties" : {
          "city_name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "continent_code" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "country_code2" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "country_code3" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "country_name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "dma_code" : {
            "type" : "long"
          },
          "ip" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "latitude" : {
            "type" : "float"
          },
          "location" : {
            "properties" : {
              "lat" : {
                "type" : "float"
              },
              "lon" : {
                "type" : "float"
              }
            }
          },
          "longitude" : {
            "type" : "float"
          },
          "postal_code" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "region_code" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "region_name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "timezone" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
```

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 12, 2018, 2:58pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/7 "2018-12-12T14:58:42Z")

</div>

What is that json from? The index mapping or the template? If its the mapping maybe you have multiple templates and one is incorrect? Or your original logstash template you copied is not correct? The geoip.location field in your original logstash template should look like what I pasted unless you had previously modified it. Double check the templates.

---

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 3:20pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/8 "2018-12-12T15:20:04Z")

</div>

The JSON is from **GET /apache2-2018.12.12/\_mapping**.

The only modifications i made when creating the index template were to rename it and remove the "logstash" property so it would be accepted.

Im sure this is a UFU - just need to figure out what step I missed.

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 12, 2018, 3:51pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/9 "2018-12-12T15:51:41Z")

</div>

Are you using the Apache module in Filebeat or just pointing Filebeat to the Apache logs and then parsing with Logstash? Unless there is additional event processing that is only available in Logstash, its easier to just use the Filebeat Apache module in Filebeat. The work is already all done for you via the pre-built ingest pipelines. Did you double check the template to make sure its correct?

> GET \_template/logstash\*

Assuming your new template begins with logstash.

---

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 3:58pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/10 "2018-12-12T15:58:32Z")

</div>

I checked the template before I copied it over. Indeed it does include the geoip data type in question.

Am using the apache2 module in filebeat.

In the logstash config - do I need to specify anything in particular so the indexes will use the template?

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 12, 2018, 4:14pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/11 "2018-12-12T16:14:46Z")

</div>

Interesting...well, the Apache module in Filebeat should already be taking care of the geoip information for you...it's built into the ingest pipleline. You need to make sure the ingest-geoip and ingest-user-agent plugins are installed on your Elasticsearch instances. Just use the default index names at first to see if you can get the data in correctly. It's pointless to send the data to a Logstash instance unless you need additional event processing that's only available via Logstash. The ingest node in Elastic has quite a few processors already and if using one of the built-in modules, all of the work is done for you already! Set your Filebeat output to poing to an Elastic host instead, using the default index names and lets see if it works.

---

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 4:25pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/12 "2018-12-12T16:25:16Z")

</div>

Is there a way to determine whether or not the template is being used?

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 12, 2018, 4:37pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/13 "2018-12-12T16:37:05Z")

</div>

At index creation time, any templates that match your index pattern will be loaded (in order from lowest to highest order number).

---

<div class="post-metadata">

**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\
**Post date:** [December 12, 2018, 4:52pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/14 "2018-12-12T16:52:47Z")

</div>

Solved!

I updated the logstash config to include '"template\_name" =\> "apache2"' in the output section. Looking at the data confirms that Im getting geoip points now.

Thank you for sticking with me on this. Your help was invaluable.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2019, 4:52pm UTC](https://discuss.elastic.co/t/getting-geohash-geoip-data-from-apache/160455/15 "2019-01-09T16:52:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.