# Send parts of a message to different Elasticsearch indices from a single Logstash instance
**URL:** https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950
**Category:** Logstash
**Created:** [November 15, 2022, 12:06pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950 "2022-11-15T12:06:42Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@powerful\_clouds](https://discuss.elastic.co/u/powerful_clouds)
#### Post date: [November 15, 2022, 12:06pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950/1 "2022-11-15T12:06:42Z")
Suppose my message has fields `A` and `B`. I want to separately send each of the fields to the same ES instance, but to different indices. Is that possible?
E.g. if this is the message: `{"A": some text, "B": some more text}`, I want "some text" to end up in `Index_1`, and "some more text" in `Index_2`. Both indices are on the same host. I don't know if this is important, but the `B` field only gets added to the message in the `filter` section.
If only the `elasticsearch` output plugin had the ability to parse the message like other plugins do, I could just extract the desired field in each of the `elasticsearch` output blocks. However, that doesn't seem to be possible.
Of course, what I'm describing can be achieved using two Logstash instances, but can it be achieved using only one?
Note: I'm looking for a solution that can be put into a Logstash pipeline.conf file. 🙂
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 15, 2022, 12:24pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950/2 "2022-11-15T12:24:47Z")
You can use a [clone filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html) to create two events that you can then format separately and send to different indices.
---
### Author:  [@Krishna\_Teja](https://discuss.elastic.co/u/Krishna_Teja)
#### Post date: [November 15, 2022, 12:34pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950/3 "2022-11-15T12:34:56Z")
Hello powerful\_clouds
Have you tried `_reindex` with filter?
Something like
```auto
POST /_reindex
{
"source": {
"index": "SOURCE INDEX",
"query": {
"match": {
"A": "some text"
}
}
},
"dest": {
"index": "DESTINATION_INDEX"
}
}
POST /_reindex
{
"source": {
"index": "SOURCE INDEX",
"query": {
"match": {
"B": "some more text"
}
}
},
"dest": {
"index": "DESTINATION_INDEX"
}
}
```
Regrads
Krishna
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 15, 2022, 1:50pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950/4 "2022-11-15T13:50:18Z")
As @Christian_Dahlqvist already said, this can be done using the `clone` filter.
You would need something like this.
```auto
filter {
clone {
clones => ["index-field-b"]
}
if [type] == "index-field-b" {
mutate {
remove_field => ["A"]
}
mutate {
... logic that you use to add field B ...
}
}
}
output {
if [type] == "index-field-b" {
elasticsearch {
... output to the index for the field B
}
} else {
elasticsearch {
... output to the index for the field A
}
}
}
```
You just need to check if you have `pipeline.ecs_compatibility` enabled for your pipeline or not as this will change the behavior of the filter, you can check the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html#plugins-filters-clone-ecs_compatibility) for more examples, but basically with `pipeline.ecs_compatibility` set to `v1` or `v8`, you will need to use the `tags` field in the conditional.
Something like:
```auto
if "index-field-b" in [tags]
```
---
### Author:  [@powerful\_clouds](https://discuss.elastic.co/u/powerful_clouds)
#### Post date: [November 15, 2022, 4:14pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950/5 "2022-11-15T16:14:06Z")
@Christian_Dahlqvist @leandrojmp Thank you very much for your help. 🙂 I've managed to get it working.
There's one thing I wanted to note: I noticed that the number of docs in each of the ES indices was double what I was expecting. I looked at the documents in the `B` index and saw that half of them had the desired field (e.g. a field of type `string` called `desired_b_field`), whereas half of them didn't. After some googling, I realized that the simplest solution is to add an `and` part to the `if` in the `output` section in order to check that the desired field is not null. E.g. for messages of type `B`:
```auto
if "index-field-b" in [tags] and ("" in [desired_b_field]) {
...
}
```
If you have any additional info on why that duplication happened, feel free to share it here. I'm sure people will find it useful. 🙂
Also, @Krishna_Teja, thanks for your proposed solution. Although it's not what I was looking for, I'm sure others will find it useful.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 13, 2022, 4:14pm UTC](https://discuss.elastic.co/t/send-parts-of-a-message-to-different-elasticsearch-indices-from-a-single-logstash-instance/318950/6 "2022-12-13T16:14:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.