# Dec 23rd, 2021: \[en\] Enriching documents with the geo\_match policy **URL:** \ **Category:** Advent Calendar\ **Created:** [December 23, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-23rd-2021-en-enriching-documents-with-the-geo-match-policy/289302 "2021-12-23T08:00:39Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![jsanz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsanz/32/53734_2.png) [@jsanz](https://discuss.elastic.co/u/jsanz)\ **Post date:** [December 23, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-23rd-2021-en-enriching-documents-with-the-geo-match-policy/289302/1 "2021-12-23T08:00:39Z")
> 🗣 This article is also available in [Spanish](https://discuss.elastic.co/t/dec-23rd-2021-es-aumentando-documentos-con-la-politica-geo-match/292462) As you may know, on September 19th of 2021, the [Cumbre Vieja volcano eruption](https://en.wikipedia.org/wiki/2021_Cumbre_Vieja_volcanic_eruption) started in La Palma, Canary Islands, Spain. The island's government almost immediately began to publish drone surveys of the lava flow footprints in an Open Data Portal ([example](https://www.opendatalapalma.es/datasets/perimetro-dron-211124/explore)). I decided to upload them to the Elastic Stack to visualize and understand the evolution of the flows. Apart from the footprints, there is also a dataset in the Open Data portal with the island [buildings](https://www.opendatalapalma.es/datasets/edificaciones/explore?location=28.657424%2C-17.915888%2C18.57). I saw an excellent opportunity to conflate the lava flows and the buildings to see how the lava affects dwellings, factories, etc. > ## Presenting the geo\_match policy for the enrich processor The [enrich processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-processor.html) can be used in an ingest pipeline to add data from a reference index to an ingested document based on a target field or by a geospatial relationship. With a [`geo_match` policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-enrich-policy-api.html), given a reference index with a polygon field, we can transfer data from those polygons into the ingested document based on a defined spatial relationship. Typically we want to use the intersection relationship. So, for example, given a reference index with world countries, we can transfer the country name to an ingested document with a geometry. You can learn more about this policy on [this example](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-match-enrich-policy-type.html) from Elasticsearch documentation and a longer tutorial on this [blog post](https://www.elastic.co/blog/how-to-map-custom-boundaries-in-kibana-with-reverse-geocoding) on how to use the policy for reverse geocoding. Our reference index will be the **lava footprints** , and the **buildings** will be the enriched documents. ## Setting up the lava footprints The first thing to address was uploading the footprints to be used with the `geo_match` policy. As the eruption advances, the area affected by the lava flows only grows. Thus, in any given location, several footprints would fall under it, but we only want to transfer the first surveyed footprint. The data upload needed to be automated to adapt the format and fix some issues. Inside this [process](https://github.com/jsanz/cumbre-vieja/blob/main/src/footprints.py), I added a step to compute the difference between a footprint and its predecessor, adding this new geometry (like a ribbon) to the document uploaded as a new geo\_shape field named `diff_geometry`. This way, any location can only intersect with one document of the index. ![drawing](https://us1.discourse-cdn.com/elastic/original/3X/c/a/ca2cc07b39a4a486365f6dfa85d5d17f732223d9.png) This is the snippet that concatenates the [shapely](https://shapely.readthedocs.io/en/stable/) library `difference` and `simplify` methods to produce a geometry difference: > ## Uploading the buildings The data processing script [uploads the buildings](https://github.com/jsanz/cumbre-vieja/blob/d9a5eb6e11c739bb6b6236ce0e0186856de22e21/src/buildings.py#L137-L166) if they are not present in the cluster. After downloading the dataset and ensuring the geometries are valid, the [`bulk` helper](https://elasticsearch-py.readthedocs.io/en/v7.15.2/helpers.html#bulk-helpers) from the Elastirsearch python client pushes the data to the cluster. ## Defining the enrich policy and ingest pipeline We store the footprints in an index called `lapalma`, and we want to transfer to the buildings the fields `id` and `timestamp' so the policy to define would be: ```auto PUT /_enrich/policy/lapalma_lookup { "geo_match": { "indices": "lapalma", "match_field": "diff_geometry", "enrich_fields": ["id", "timestamp"], } } ``` The next step will be to execute the policy: ```auto POST /_enrich/policy/lapalma_lookup/_execute ``` > ⚠ **Important!** ⚠: We need to execute the policy every time we update the index with new footprints. With the policy ready, we can add it to an ingest pipeline that will expect a `geometry` field to intersect with the geometries from the policy. The policy will create a new object field called `footprints` to store the identifier and the timestamp from the reference index. The ingest pipeline will remove the geometry that is also, by default, transferred to the enriched document. ```auto PUT _ingest/pipeline/buildings_footprints { "description": "Enrich buildings with Cumbre Vieja footprints.", "processors": [ { "enrich": { "field": "geometry", "policy_name": "lapalma_lookup", "target_field": "footprints", "shape_relation": "INTERSECTS", "ignore_missing": True, "ignore_failure": True, } }, { "remove": { "field": "footprints.diff_geometry", "ignore_missing": True, "ignore_failure": True, "description": "Remove the shape field", } }, ], } ``` ## Running the ingest pipeline Now that we have our ingest pipeline, we are going to update the buildings in place with a [`_update_by_query`](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html) request. We won't run it against the full index, instead we will search for the documents that satisfy two conditions and pass them through the pipeline 1. They are inside a bounding box of the affected area 2. They have no footprint assigned yet ```auto POST buildings/_update_by_query?pipeline=buildings_footprints { "query": { "bool": { "must_not": [{ "exists": { "field": "footprints.id" } }], "filter": { "geo_bounding_box": { "geometry": { "top_left": { "lat": 28.647, "lon": -17.95 }, "bottom_right": { "lat": 28.58, "lon": -17.83 }, } } }, } } } ``` ## Putting all together I used Kibana DevTools syntax on this post for readability, but all these requests run from a [python script](https://github.com/jsanz/cumbre-vieja/blob/main/src/app.py) in a [Github Action](https://github.com/jsanz/cumbre-vieja/blob/main/.github/workflows/python-app.yml) on every push that detects a change in the [footprints identifiers script](https://github.com/jsanz/cumbre-vieja/blob/main/src/data.py). Unfortunately, there is no way to automatically detect the publishing of the new datasets in the Open Data portal, so these identifiers need to be added manually to trigger the data update ([example commit](https://github.com/jsanz/cumbre-vieja/commit/915d7ebd376d5e56c610b59c9787a2c57f1ee6ca)). With the datasets in the cluster, we can explore them with [Elastic Maps Time Slider](https://ela.st/cumbre-vieja-eruption-map) and with a [Kibana Dashboard](https://ela.st/cumbre-vieja-eruption). ![05-animation-buildings](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bf83679b36a9ad38ecfc49c61aca560852392ee0.gif) _[https://ela.st/cumbre-vieja-eruption-map](https://ela.st/cumbre-vieja-eruption-map)_ ![06-dashboard](https://us1.discourse-cdn.com/elastic/original/3X/5/3/531fe15c457af21fbb2be4a99b093c7700325889.jpeg) _[https://ela.st/cumbre-vieja-eruption](https://ela.st/cumbre-vieja-eruption)_ Have fun! 👋 * * * **Update** : a new post has been published at Elastic's blog regarding this disaster tracking project, focusing in the usage of Elastic Maps and time slider to visualize events that evolve over time. Check it out! > **[How Elastic Maps can be used to track the devastating eruption of a volcano,...](https://www.elastic.co/blog/understanding-evolution-volcano-eruption-elastic-maps)** > > La Palma, in the Canary Islands (Spain), faced a major natural disaster with the eruption of the Cumbre Vieja volcano. Learn how Elastic Maps and Kibana were used to display and animate the evolution ... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 20, 2022, 8:01am UTC](https://discuss.elastic.co/t/dec-23rd-2021-en-enriching-documents-with-the-geo-match-policy/289302/2 "2022-01-20T08:01:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.