# How we can create two index in logstash
**URL:** https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082
**Category:** Logstash
**Created:** [May 23, 2023, 7:07am UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082 "2023-05-23T07:07:08Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [May 23, 2023, 7:07am UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/1 "2023-05-23T07:07:08Z")
Hello,
Anyone came across below scenario,
I have a json as input and am filtering the data later creating index in output block to push it into elastic
Here i want to split the data into two set and want them to send to two respective indices. like below example.
created query\_details event object by combination of few fields from json input.
i want to send this quer\_details columns alone to first index rest of the columns which is part of input json will be send to second index.
All my transaction has query\_details so i can't use If else condition in my output field.
//Code is here
````auto
input {
tcp {
codec => json_lines { charset => "UTF-8" }
port =>
}
}
filter {
json {
source => "payload_raw"
target => "payload"
}
ruby {
code => "
removed_keys = ['id','operator']
query_details = {}
removed_keys.each do |key|
if event.get('msg').include?(key)
query_details[key] = event.get('msg')[key]
event.remove('[msg][#{key}]')
end
end
event.set('[query_details]', query_details)
"
}
mutate {copy => { "[msg][metadata]" => "metadata" }
remove_field => ["[msg][metadata]"]
}
}
output {
if [query_details] {
elasticsearch {
hosts => ["hello.net:9200","hello2.net:9200"]
index => "query-details-%{+YYYY.MM.dd}"
user => "elas"
password => "hai"
}
}
else {
elasticsearch {
hosts => ["hello.net:9200","hello2.net:9200"]
index => "query-default-%{+YYYY.MM.dd}"
user => "ela"
password => "hai"
}
stdout { codec => rubydebug }
}
}```
````
---
### Author:  [@eMitch](https://discuss.elastic.co/u/eMitch)
#### Post date: [May 24, 2023, 1:38pm UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/2 "2023-05-24T13:38:52Z")
Hi @subash_k and welcome to the community!
One way to accomplish this would be to utilize [Ingest Pipelines](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#ingest) along with two separate elasticsearch outputs to send the same document to two different indices.
Something like this:
```auto
output {
elasticsearch {
index => "query-details-%{+YYYY.MM.dd}"
pipeline => "query_details_pipeline"
hosts=> "${ELASTIC_HOSTS}"
user=> "${ELASTIC_USER}"
password=> "${ELASTIC_PASSWORD}"
}
elasticsearch {
index => "query-default-%{+YYYY.MM.dd}"
pipeline => "query_default_pipeline"
hosts=> "${ELASTIC_HOSTS}"
user=> "${ELASTIC_USER}"
password=> "${ELASTIC_PASSWORD}"
}
}
```
Then in your [Ingest Pipelines - you can drop the `[query_details]`](https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html#remove-processor) or any other fields that aren't required in the final document to be put in the index:
```auto
PUT _ingest/pipeline/query_default_pipeline
{
"description": "pipeline for processing query_default documents",
"processors": [
{
"remove": {
"description": "removing query_details",
"field": "query_details"
}
}
]
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 24, 2023, 1:55pm UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/3 "2023-05-24T13:55:30Z")
You can use the clone filter to [clone](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html#_description_123) your event and a conditional to only apply the ruby filter to the cloned event and use the same conditional in the output.
Something like this:
```auto
filter {
json {
source => "payload_raw"
target => "payload"
}
clone {
clones => ["details"]
}
if [type] == "details" {
ruby {
code => "
removed_keys = ['id','operator']
query_details = {}
removed_keys.each do |key|
if event.get('msg').include?(key)
query_details[key] = event.get('msg')[key]
event.remove('[msg][#{key}]')
end
end
event.set('[query_details]', query_details)
"
}
}
mutate {
copy => {
"[msg][metadata]" => "metadata"
}
remove_field => ["[msg][metadata]"]
}
}
output {
if [type] == "details" {
elasticsearch {
hosts => ["hello.net:9200","hello2.net:9200"]
index => "query-details-%{+YYYY.MM.dd}"
user => "elas"
password => "hai"
}
} else {
elasticsearch {
hosts => ["hello.net:9200","hello2.net:9200"]
index => "query-default-%{+YYYY.MM.dd}"
user => "elas"
password => "hai"
}
}
}
```
You just need to check if you have `pipeline.ecs_compatibility` enabled or not as this will change the behavior of the `clone` filter as explained in the documentation.
---
### Author:  [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [May 26, 2023, 9:01am UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/4 "2023-05-26T09:01:52Z")
Thanks @leandrojmp this will fail to insert the deafult set data. pervious solution (pipeline logic ) worked
---
### Author:  [@subash\_k](https://discuss.elastic.co/u/subash_k)
#### Post date: [May 26, 2023, 9:03am UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/5 "2023-05-26T09:03:49Z")
Hey Thanks @eMitch .
It worked 😃
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [May 26, 2023, 12:10pm UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/6 "2023-05-26T12:10:38Z")
> [@subash\_k](#):
>
> Thanks @leandrojmp this will fail to insert the deafult set data.
I don't think so, if the conditional is correct it will index the cloned documents, with `type` equals to `details` on one index, and the other documents on the another index.
But since the other solution worked for you, there is no need to troubleshoot this further.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 23, 2023, 12:10pm UTC](https://discuss.elastic.co/t/how-we-can-create-two-index-in-logstash/334082/7 "2023-06-23T12:10:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.