# Elastic APM GeoIP Pipeline on cloud service **URL:** \ **Category:** APM\ **Tags:** rum, server\ **Created:** [May 6, 2019, 7:34pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838 "2019-05-06T19:34:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ariel\_k](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@ariel\_k](https://discuss.elastic.co/u/ariel_k)\ **Post date:** [May 6, 2019, 7:34pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/1 "2019-05-06T19:34:52Z")
**Kibana version** : 7 **Elasticsearch version** :7 **APM Server version** :7 **APM Agent language and version** :RUM/js **Browser version** : **Original install method (e.g. download page, yum, deb, from source, etc.) and version**☁ \*\*Fresh install or upgraded from other version?\*\*deployment created by elastic cloud Hi , i am using the cloud deployment for elastic stack, i have 2 pipelines used for APM data 1. user\_agent 2. apm\_user\_geoip the first pipeline added was the user\_agent and it seems to be working. the second pipeline i am not sure 3. the pipeline : > "apm\_user\_geoip" : { > "description" : "Resolve GeoIP information for APM events", > "processors" : [ > { > "geoip" : { > "field" : "client.ip", > "target\_field" : "qageo", > "ignore\_missing" : true > } > } > ] > } 1. the test : > GET /\_ingest/pipeline/apm\_user\_geoip/\_simulate > { > "docs": [ > { > "\_source": { > "client": { > "ip": "108.2.12.80" > } > } > } > ] > } 1. test response : > { > "docs" : [ > { > "doc" : { > "\_index" : "\_index", > "\_type" : "\_doc", > "\_id" : "\_id", > "\_source" : { > "client" : { > "ip" : "108.2.12.80" > }, > "qageo" : { > "continent\_name" : "North America", > "region\_iso\_code" : "US-PA", > "city\_name" : "Philadelphia", > "region\_name" : "Pennsylvania", > "location" : { > "lon" : -75.1968, > "lat" : 39.9597 > }, > "country\_iso\_code" : "US" > } > }, > "\_ingest" : { > "timestamp" : "2019-05-06T19:31:19.219Z" > } > } > } > ] > } 4.the pipelines in the APM > # Note that the syntax for user settings can change between major versions. > > # You might need to update these user settings before performing a major version upgrade. > > # > > # To learn more, see the documentation. > > apm-server.register.ingest.pipeline.enabled: true > output.elasticsearch.pipelines: > > - pipeline: "apm\_user\_agent" > - pipeline: "apm\_user\_geoip" the problem : i can see the client.ip field , i can not the the "qageo" target being created for it. i am sure i have missed something . any ideas ? thanks , Ariel. ---
**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\ **Post date:** [May 6, 2019, 9:32pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/2 "2019-05-06T21:32:11Z")
Hey @ariel_k, thanks for the detailed description of the issue. Your setup looks good to me so we'll need more inforrmation to figure this out. How are you checking for the `qageo` field? Would post whatever you're able to share from this query: ```auto GET apm-*/_search { "query": { "exists": { "field": "client.ip" } } } ``` `qageo` won't be indexed by default so I'd like to rule that the chance the data is present but just not queryable yet - you'll want to update your index template and recreate the index if that's the case. ---
**Author:** ![ariel\_k](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@ariel\_k](https://discuss.elastic.co/u/ariel_k)\ **Post date:** [May 6, 2019, 9:51pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/3 "2019-05-06T21:51:43Z")
Hey @gil , about the search i can see results and i can see the client.ip field in documents. this is part of the response to the search you suggested. any more info from this search is needed , the complete response is huge. { "took" : 8, "timed\_out" : false, "\_shards" : { "total" : 19, "successful" : 19, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 1127, "relation" : "eq" }, regarding the index, i was able to refresh it. i am not sure how to re-create the index. thanks, Ariel ---
**Author:** ![ariel\_k](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@ariel\_k](https://discuss.elastic.co/u/ariel_k)\ **Post date:** [May 7, 2019, 4:29pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/4 "2019-05-07T16:29:31Z")
Hey @gil is there any information you need that i have not added ? thanks Ariel ---
**Author:** ![gil](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gil/32/41911_2.png) [@gil](https://discuss.elastic.co/u/gil)\ **Post date:** [May 7, 2019, 11:24pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/5 "2019-05-07T23:24:01Z")
I initially overlooked your configuration, the problem is here: ```auto output.elasticsearch.pipelines: - pipeline: "apm_user_agent" - pipeline: "apm_user_geoip" ``` Only the [first matching pipeline](https://www.elastic.co/guide/en/apm/server/current/elasticsearch-output.html#pipelines-option-es) is used in the indexing query, in this case there are no conditions so only the first pipeline is ever used. I'd suggest combining the pipelines into a single one with multiple processors like: ```json [ { "id": "apm_user_info", "body": { "description": "Add user agent information for APM events", "processors": [ { "user_agent": { "field": "user_agent.original", "target_field": "user_agent", "ignore_missing": true } }, { "geoip": { "field": "client.ip", "target_field": "qageo", "ignore_missing": true } } ] } } ] ``` And then using it: ```auto GET /_ingest/pipeline/apm_user_info/_simulate { "docs": [ { "_source": { "user_agent": { "original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" }, "client": { "ip": "108.2.12.80" } } } ] } ``` yields the expected result: ```json { "docs" : [ { "doc" : { "_index" : "_index", "_type" : "_doc", "_id" : "_id", "_source" : { "client" : { "ip" : "108.2.12.80" }, "user_agent" : { "name" : "Chrome", "original" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36", "os" : { "name" : "Mac OS X", "version" : "10.13.6", "full" : "Mac OS X 10.13.6" }, "device" : { "name" : "Other" }, "version" : "73.0.3683" }, "qageo" : { "continent_name" : "North America", "region_iso_code" : "US-PA", "city_name" : "Philadelphia", "region_name" : "Pennsylvania", "location" : { "lon" : -75.1968, "lat" : 39.9597 }, "country_iso_code" : "US" } }, "_ingest" : { "timestamp" : "2019-05-07T23:20:16.598455Z" } } } ] } ``` We plan to add something similar as a default in the near future, you can follow [https://github.com/elastic/apm-server/issues/1283](https://github.com/elastic/apm-server/issues/1283) for updates on that effort. ---
**Author:** ![ariel\_k](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@ariel\_k](https://discuss.elastic.co/u/ariel_k)\ **Post date:** [May 8, 2019, 3:49am UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/6 "2019-05-08T03:49:15Z")
Hey @gil , thank you , its working 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 28, 2019, 11:49pm UTC](https://discuss.elastic.co/t/elastic-apm-geoip-pipeline-on-cloud-service/179838/7 "2019-05-28T23:49:15Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.