# Enrich fields in new index from other index present in kibana using logstash
**URL:** https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397
**Category:** Logstash
**Created:** [September 16, 2021, 12:29pm UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397 "2021-09-16T12:29:29Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)
#### Post date: [September 16, 2021, 12:29pm UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/1 "2021-09-16T12:29:29Z")
Hi ,
I am using logstash to create a new index from csv file which is having the index name-"therapy"
Another index is "participation" already present in kibana with other fields.
Both participation and therapy index is haiving '"id" field in common.
I am able to built a conf file to ingest therapy data to kibana.
I want some ideas/hints on how to add fields from the participation index when therapy and participation field "id" got matched. and enriched therapy docs with participation fields to the therapy index.
Thank you in advance.
Current conf file-
```auto
input {
file {
path => "/home/data.csv"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
skip_header => "true"
columns => ["user_id","DeviceSerialNumber"]
}
elasticsearch {
hosts => "<>"
index => "participation"
query => "id:\"%{[id]}\"
#Fields to add from participation index to therapy
fields => {
"Month" => "Month",
"cost" => "cost"
}
}
}
output{
stdout {}
#output {
# elasticsearch {
# hosts => "<>"
# index => "therapy"
# }
}
```
---
### Author:  [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)
#### Post date: [September 20, 2021, 11:10am UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/2 "2021-09-20T11:10:06Z")
Hi,
I checked with enrich processor to enrich fields from one index "participation" to another index "therapy" and stored consolidated fields into 3 rd index '"data-participation-therapy1"
"user\_id" is the common field between two indexes.
create an enrich policy-
```auto
PUT /_enrich/policy/therapy-policy
{
"match": {
"indices": "therapy",
"match_field": "user_id",
"enrich_fields": [<.....>]
}
}
```
execute that policy in order to create an enrichment index
```auto
POST /_enrich/policy/therapy-policy/_execute
{
"status" : {
"phase" : "COMPLETE"
}
}
```
```auto
PUT /_ingest/pipeline/therapy_lookup
{
"description" : "Enriching data-participation with therapy data",
"processors" : [
{
"enrich" : {
"policy_name": "therapy-policy",
"field" : "user_id",
"target_field": "tmp",
"max_matches": "1"
}
},
{
"script": {
"if": "ctx.tmp != null",
"source": "ctx.putAll(ctx.tmp); ctx.remove('tmp');"
}
}
]
}
*************
{
"acknowledged" : true
}
***************
```
Getting error while reindex the cosolidated data into "data-participation-therapy1" index
```auto
POST _reindex
{
"source": {
"index": "-participation"
},
"dest": {
"index": "data-participation-therapy1",
"pipeline": "therapy_lookup"
}
}
output-
{"ok":false,"message":"backend closed connection"}
```
Ref link-[https://stackoverflow.com/questions/64008094/combine-two-index-into-third-index-in-elastic-search-using-logstash](https://stackoverflow.com/questions/64008094/combine-two-index-into-third-index-in-elastic-search-using-logstash)
@val
---
### Author:  [@val](https://discuss.elastic.co/u/val)
#### Post date: [September 22, 2021, 9:41am UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/3 "2021-09-22T09:41:25Z")
`backend closed connection` simply means that the client (i.e. Kibana Dev Tools in your browser) timed out. But the reindex process is still ongoing in the background.
If your source index is big, odds are high that it will take longer than the timeout for the operation to terminate. So you should start your reindex to run asynchronously using
```
POST _reindex?wait_for_completion=false
```
The call will return immediately and give you a task ID which you can use to check the task status as it progresses using
```
GET _tasks/
```
---
### Author:  [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)
#### Post date: [September 23, 2021, 10:47am UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/4 "2021-09-23T10:47:43Z")
Thanks, Val for the reply.
participation index consist of 80k docs and therapy index contains 9k docs with ''user\_id" field in comman.
I want to know if enrichment processor scans docs 1 to 1 for user id in both indexes.
OR Enrichment processor will scan all docs present in participation index for particular "user id" in therapy index and it will enrich stored docs in new index.
If docs get 1 to 1 that is the user id of 1st doc in therapy is check with user id participation in 1 st then hardly any docs will get enriched.
Enriching process described here-[Enrichment processor working fine on demo index](https://discuss.elastic.co/t/getting-error-502-ok-false-message-backend-closed-connection-for-enriching-fields-to-another-index/284637)
---
### Author:  [@val](https://discuss.elastic.co/u/val)
#### Post date: [September 23, 2021, 11:04am UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/5 "2021-09-23T11:04:10Z")
The reindex API will iterate over all docs of the source index (i.e. participation) and for each will retrieve the doc with the corresponding user\_id from the enrichment index (i.e. therapy) and will add what it finds to the source document and store the result in the destination index (i.e. data-participation-therapy1).
---
### Author:  [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)
#### Post date: [September 23, 2021, 2:05pm UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/6 "2021-09-23T14:05:40Z")
Thanks Val, It's a Success.
Index pattern showing 49 fields which is a combination of participation and therapy fields.
Two Queries-
1. But In Discover section I am seeing only 21 fields which is from participation index.Therapy index fields is not present in data-participation-therapy1 index.
I am able to query all fields in dev tools dsl query and mapping also showing all 49 fields.
21 fields in discovery tab of data-participation-therapy1 index.
Index pattern showing 49 fields-
2,Field "DayDateStorage" is date field in "therapy" index but in "data-participation-therapy1" it's showing text , have to changed it to date.
**Update-**
**Both Queries are solved. I place mapping of data-participation-therapy1 index before reindexing data-participation-therapy1. Discover field showing all fields and "DayDateStorage" is converted to date type due to changes in mapping of index.**
Thanks.
---
### Author:  [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)
#### Post date: [September 27, 2021, 1:16pm UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/7 "2021-09-27T13:16:48Z")
One Query-
As the participation and therapy index is getting updated once in a day.(live pipelines)
How to enrich the new docs into the enriched index " **data-participation-therapy1**" index.
Ref link-[What is the best way to enrich near real-time data in ElasticSearch with batch data that may come in later? - Stack Overflow](https://stackoverflow.com/questions/66632049/what-is-the-best-way-to-enrich-near-real-time-data-in-elasticsearch-with-batch-d)
I got this link, but still not got the clarity.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 25, 2021, 1:17pm UTC](https://discuss.elastic.co/t/enrich-fields-in-new-index-from-other-index-present-in-kibana-using-logstash/284397/8 "2021-10-25T13:17:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.