# Dec 19th, 2019 \[EN\]\[Elasticsearch\] Simplifying Ingest Pipelines with the new Enrich Processor
**URL:** https://discuss.elastic.co/t/dec-19th-2019-en-elasticsearch-simplifying-ingest-pipelines-with-the-new-enrich-processor/211026
**Category:** Advent Calendar
**Created:** [December 19, 2019, 8:00am UTC](https://discuss.elastic.co/t/dec-19th-2019-en-elasticsearch-simplifying-ingest-pipelines-with-the-new-enrich-processor/211026 "2019-12-19T08:00:03Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@Imma](https://discuss.elastic.co/u/Imma)
#### Post date: [December 19, 2019, 8:00am UTC](https://discuss.elastic.co/t/dec-19th-2019-en-elasticsearch-simplifying-ingest-pipelines-with-the-new-enrich-processor/211026/1 "2019-12-19T08:00:03Z")
When ingesting data through a regular Elasticsearch [ingest pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/ingest.html) (e.g. with dissect, rename, or remove [processors](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/ingest-processors.html)) we can now add an [Enrich Processor](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/enrich-processor.html).
This allows us to do lookups on other Elasticsearch’s indices and enrich the incoming document before sending it to its own index.
[Enrich Policies](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/enrich-policy-definition.html) are what powers this new functionality in **Elasticsearch version 7.5.** , and at this point, we can enrich data based on [geolocation](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/geo-match-enrich-policy-type.html) or through [matching](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/match-enrich-policy-type.html) exact values using a term query.
Let’s use an example to demonstrate this new capability.
I’m an avid reader at my local library. They have ingested their catalog in Elasticsearch, and each book is searchable in the index **book-catalog** with book _title_, _author_ and _ISBN_ (the International Standard Book Number). This is a sample of the catalog books:
```json
POST book-catalog/_doc/
{
"isbn-10": "0571276547",
"author": "Paul Auster",
"title": "The Brooklyn Follies"
}
POST book-catalog/_doc/
{
"isbn-10": "0571283209",
"author": "Paul Auster",
"title": "Winter Journal"
}
POST book-catalog/_doc/
{
"isbn-10": "2298068968",
"author": "Marc Levy",
"title": "Un sentiment plus fort que la peur"
}
POST book-catalog/_doc/
{
"isbn-10": "086068511",
"author": "Maya Angelou",
"title": "I Know Why The Caged Bird Sings"
}
POST book-catalog/_doc/
{
"isbn-10": "163286696",
"author": "James Rhodes",
"title": "Instrumental"
}
```
Each time someone borrows a book, the library creates a document in another index, let’s call it **book-lending** , with the _user_ id, which we will imagine it’s the user’s e-mail, and the book _ISBN_. For example:
```json
POST book-lending/_doc
{
"isbn-10": "163286696",
"user": "mim@mail.net"
}
```
Later on, they would like to explore what are the user’s favorite authors. Or, whenever a new book is acquired from an author, inform users who have borrowed books from the same author in the past.
What will help my librarians get that information in a single query, involves enriching the book lending data at ingest time in a way that the book-lending document holds not only the ISBN code for a book but also the name of the book and author.
This can be achieved as follows.
1. Create an [Enrich Policy](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/enrich-policy-definition.html) that [matches](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/match-enrich-policy-type.html) the isbn-10 field from the incoming document to the book-catalog index.
2. [Execute](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/execute-enrich-policy-api.html) the previous policy.
3. Create an ingest pipeline that makes use of that enrich policy. In the example, we are using a [matching](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/match-enrich-policy-type.html) that will add a field called book-details to the incoming book lending documents, which will hold the book’s author and title based on its isbn-10 code.
4. Ingest the documents in the book-lending using the pipeline created in the step above.
If we now have a look at the documents ingested, we will see the book information was added under the field book-details:
```json
GET book-lending/_search?filter_path=hits.hits._source
{
"hits" : {
"hits" : [
{
"_source" : {
"isbn-10" : "163286696",
"book-details" : {
"isbn-10" : "163286696",
"author" : "James Rhodes"
"title" : "Instrumental"
},
"user" : "mim@mail.net"
}
},
…
```
And thus, my librarians could find out what users like to read “Paul Auster” and send them an e-mail when a new book comes in:
```json
GET book-lending/_search?filter_path=hits.hits._source.user
{
"query": {
"bool": {
"filter": [
{ "term": { "book-details.author.keyword": "Paul Auster" }} ]
}
},
"collapse": {
"field": "user.keyword"
}
}
__________________
{
"hits" : {
"hits" : [
{
"_source" : {
"user" : "mim@mail.net"
}
}
]
}
}
```
Or what is the favorite author for each user:
```json
GET book-lending/_search?filter_path=aggregations.top_author.buckets
{
"size": 0,
"query": {
"bool": {
"filter":
{ "term": { "user.keyword": "mim@mail.net" }}
}
},
"aggs": {
"top_author": {
"terms": {
"field": "book-details.author.keyword",
"size": 1
}
}
}
}
____________________________
{
"aggregations" : {
"top_author" : {
"buckets" : [
{
"key" : "Paul Auster",
"doc_count" : 2
}
]
}
}
}
```
Enjoy reading books and take the enrich processor for a spin!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 7:36am UTC](https://discuss.elastic.co/t/dec-19th-2019-en-elasticsearch-simplifying-ingest-pipelines-with-the-new-enrich-processor/211026/2 "2022-11-04T07:36:33Z")