# Adding GeoIP pipeline to APM data

**URL:** <https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035>\
**Category:** APM\
**Tags:** java\
**Created:** [May 1, 2023, 5:36pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035 "2023-05-01T17:36:16Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 5:36pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/1 "2023-05-01T17:36:16Z")

</div>

Hi, We're essentially on ES 8.7 and I've interested on adding Geo Location data to our APM data from our Java application.

I did see what I think is an [old post](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline/162622) that talked about it, but I suspect that a new mechanism was introduced with `@custom` processors?

What is the recommended mechanism for this? FWIW we're not using RUM.

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 8:08pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/2 "2023-05-01T20:08:02Z")

</div>

OK, I actually made some progress myself. I created a pipeline called `metrics-apm@custom` and had it search for a custom label that I had to add for the user's public IP address. However, then I test the pipeline against a sample document I see a tag on the result of `_geoip_database_unavailable_GeoLite2-City.mmdb`.

I did the example test [here](https://www.elastic.co/guide/en/elasticsearch/reference/8.7/geoip-processor.html) and that all seemed to work just fine.

It looks to me like the processor config is exactly the same, but the sample works, but my custom one does not?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 1, 2023, 8:11pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/3 "2023-05-01T20:11:41Z")

</div>

There is a little edge case where the DBs are lazy loaded the first time... check the APM data again.

Is it still saying the same thing?

Otherwise can you share your exact custom pipeline?

And a sample document

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 8:37pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/4 "2023-05-01T20:37:25Z")

</div>

So here's my doc:

```auto
[
  {
    "_id": "Something",
    "_index": "whatever",
    "_source": {
      "labels": {
        "IP": "100.20.56.87"
      }
    }
  }
]

```

Processor:

```auto
[
  {
    "geoip": {
      "field": "labels.IP",
      "ignore_missing": true
    }
  }
]

```

Output looks like:

```auto
{
  "docs": [
    {
      "doc": {
        "_index": "whatever",
        "_id": "Something",
        "_version": "-3",
        "_source": {
          "labels": {
            "IP": "100.20.56.87"
          },
          "tags": [
            "_geoip_database_unavailable_GeoLite2-City.mmdb"
          ]
        },
        "_ingest": {
          "timestamp": "2023-05-01T20:34:45.576782612Z"
        }
      }
    }
  ]
}

```

So I did just notice that I get SOME results occasionally when I do the test with the above document. But maybe just 25%

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 1, 2023, 8:47pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/5 "2023-05-01T20:47:58Z")

</div>

What does _"We're essentially on ES 8.7"_ mean 🙂

There was a change in some behavior...  
What type of cluster are you running? Self Managed Node, Elastic Cloud etc? how many nodes?...

Perhaps the GeoIP db settings are not consistent on all the nodes, which can result in intermittent results.

Is that output from the pipeline simulate? How are you testing, hard to tell without the actual commands... or are you doing that in the Ingest Pipeline Tester in Kibana?

Try this from [this](https://discuss.elastic.co/t/geoip-database-unavailable-geolite2-asn-mmdb/330772/22) post

Disable the geoip databases

```auto
PUT _cluster/settings
{
  "persistent": {
    "ingest.geoip.downloader.enabled" : false
  }
}

GET _cat/indices/.ge*?v

GET _ingest/geoip/stats

```

Wait about 2 mins then re-enable

```auto
PUT _cluster/settings
{
  "persistent": {
    "ingest.geoip.downloader.enabled" : true
  }
}

GET _cat/indices/.ge*?v

GET _ingest/geoip/stats

```

Also how are you directing these request... only ingest nodes have the geoip database so you if you direct the request to a data only node (not and ingest) it will fail.

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 9:01pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/6 "2023-05-01T21:01:13Z")

</div>

Sorry, 8.6.2 (doesn't round up! 😉 ). Self-managed node.  
I am running the test in Kibana.  
The stats look good:

```auto
{
    "stats": {
        "successful_downloads": 3,
        "failed_downloads": 0,
        "total_download_time": 14420,
        "databases_count": 3,
        "skipped_updates": 0,
        "expired_databases": 0
    },
    "nodes": {
        "m1JBAEpIQASeJp4ktyEaFQ": {
            "databases": [
                {
                    "name": "GeoLite2-Country.mmdb"
                },
                {
                    "name": "GeoLite2-ASN.mmdb"
                },
                {
                    "name": "GeoLite2-City.mmdb"
                }
            ],
            "files_in_temp": [
                "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
                "GeoLite2-ASN.mmdb_LICENSE.txt",
                "GeoLite2-City.mmdb_LICENSE.txt",
                "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
                "GeoLite2-ASN.mmdb",
                "GeoLite2-City.mmdb_COPYRIGHT.txt",
                "GeoLite2-City.mmdb",
                "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
                "GeoLite2-Country.mmdb_LICENSE.txt",
                "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
                "GeoLite2-Country.mmdb",
                "GeoLite2-Country.mmdb_COPYRIGHT.txt",
                "GeoLite2-City.mmdb_README.txt"
            ]
        },
        "yYLvtvJ0Qgav4mzrU4e7sQ": {
            "databases": [
                {
                    "name": "GeoLite2-Country.mmdb"
                },
                {
                    "name": "GeoLite2-ASN.mmdb"
                },
                {
                    "name": "GeoLite2-City.mmdb"
                }
            ],
            "files_in_temp": [
                "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
                "GeoLite2-ASN.mmdb_LICENSE.txt",
                "GeoLite2-City.mmdb_LICENSE.txt",
                "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
                "GeoLite2-ASN.mmdb",
                "GeoLite2-City.mmdb_COPYRIGHT.txt",
                "GeoLite2-City.mmdb",
                "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
                "GeoLite2-Country.mmdb_LICENSE.txt",
                "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
                "GeoLite2-Country.mmdb",
                "GeoLite2-Country.mmdb_COPYRIGHT.txt",
                "GeoLite2-City.mmdb_README.txt"
            ]
        }
    }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 1, 2023, 9:12pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/7 "2023-05-01T21:12:17Z")

</div>

2 Nodes? Exactly? are there more nodes?

"I am running the test in Kibana."

Multiple ways of running the tests ... are you running them in Dev Tools or the Ingest Pipeline Constructor?

Did you try my disable and enable settings the geoip settings above and test again?

Apologies when things are intermittent every detail counts ...

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 9:20pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/8 "2023-05-01T21:20:57Z")

</div>

Exactly 2 ingestion nodes.  
I am using the `Test Pipeline` in the pipeline editor in Kibana.  
I did disable, and re-enable the setting.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 1, 2023, 9:28pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/9 "2023-05-01T21:28:31Z")

</div>

> [@Scott\_Chapman1](#):
>
> Exactly 2 ingestion nodes.

So there are other nodes? Are There data only nodes?

> [@Scott\_Chapman1](#):
>
> I did disable, and re-enable the setting.

Good ... And the tests now?..... are they still intermittent?

Personally, I would use Dev Tools for Testing using [Pipeline Simulate](https://www.elastic.co/guide/en/elasticsearch/reference/current/simulate-pipeline-api.html)

Is Kibana pointed to an Ingest Node?

Is the APM Server only pointed ONLY to the ingestion nodes?

I suspect it's possible you are directing requests to data only nodes ... I would think the coordinator part should re-direct but for some reason... I am not completely clear on that ... and don't have an easy way to test right now...

If the behavior is still inconsistent I would check that all the requests that need GeoIP are being directed to the ingest nodes.

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 9:41pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/10 "2023-05-01T21:41:32Z")

</div>

Thanks. Maybe I should (for now) ignore my testing strategy.  
I did create a VERY simple pipeline to add a new field to my APM App data named `metrics-apm@custom`. But I am not seeing the new field get added to that data.

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 1, 2023, 9:43pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/11 "2023-05-01T21:43:44Z")

</div>

I have 3 master nodes, 2 data-hot nodes, 2 ingest nodes, 1 APM server

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 1, 2023, 9:44pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/12 "2023-05-01T21:44:43Z")

</div>

> [@Scott\_Chapman1](#):
>
> I did create a VERY simple pipeline to add a new field to my APM App data named `metrics-apm@custom`. But I am not seeing the new field get added to that data.

Yup that is the very first thing I do whenever I integrate a new pipeline ... I use a set processor and add a field `pipeline_run: true`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [May 1, 2023, 9:52pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/13 "2023-05-01T21:52:53Z")

</div>

If you are trying to add the geoip to the APM transactions that is not the correct data stream...

Think you want to look at the following

Data Stream : traces-apm-default

```auto
data_stream.dataset: apm
data_stream.namespace: default
data_stream.type: traces

```

The datastream you are looking at is for APM Metrics (which is aggregated data)  
How familiar are you with Sample Rate etc...

Only Sampled Transactions will / should have IP details etc that can be geoip'd

This may help you understand it a bit more

> **[Transactions | APM User Guide \[8.7\] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/data-model-transactions.html)**

> <https://github.com/elastic/kibana/blob/main/x-pack/plugins/apm/dev_docs/apm_queries.md>

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 2, 2023, 1:08am UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/14 "2023-05-02T01:08:10Z")

</div>

Yea, my pipeline was for `metrics-apm` and I was looking at the documents in the `data_stream.type: metrics` and weren't seeing the additional field.

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 2, 2023, 2:25pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/15 "2023-05-02T14:25:04Z")

</div>

FWIW I did also add a simple pipeline for traces as well, also just appends a new field/value and I am not seeing any evidence that is getting leveraged either.

I am clearly missing something...

---

<div class="post-metadata">

**Author:** ![Scott\_Chapman1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/scott_chapman1/32/118292_2.png) [@Scott\_Chapman1](https://discuss.elastic.co/u/Scott_Chapman1)\
**Post date:** [May 2, 2023, 7:02pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/16 "2023-05-02T19:02:27Z")

</div>

OK, I think I got it working. Not sure what the problem was, but restarting my nodes appeared to resync some data. Looks good now. Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2023, 3:02pm UTC](https://discuss.elastic.co/t/adding-geoip-pipeline-to-apm-data/332035/17 "2023-05-23T15:02:29Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.