# Logstash file input - output to different indices
**URL:** https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906
**Category:** Logstash
**Created:** [July 28, 2021, 7:44pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906 "2021-07-28T19:44:24Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [July 28, 2021, 7:44pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906/1 "2021-07-28T19:44:24Z")
Hello All,
I am running ELK 7.6.2 stack.
In my current set up, a single file gets ingested in logstash and creates an index successfully as below (only relevant part shown):
```auto
input {
file {
path => "/opt/gtal/ital/elasticsearch/app/logstash/stage/MQA_PRD_STATS-*.txt"
- - - - - - - - - - -
- - - - - - - - - - -
- - - - - - - - - - -
- - - - - - - - - - -
output {
file {
path => "/opt/gtal/ital/elasticsearch/logs/mqa/rubydebug.txt"
codec => rubydebug
}
elasticsearch {
hosts => ["xxxxxxx:43045","xxxxxxxx:43045","xxxxxxxx:43045","xxxxxxxx:43045"]
user => "elastic"
password => "xxxxxxxx"
index => "demo-csv-%{+YYYY.MM.dd}"
doc_as_upsert => true
action => "update"
document_id => "%{my_fingerprint}"
}
}
```
Now I want to ingest 2 files instead of one in the same logstash file and direct the output to two different indices. The resultant logstash config file would looks something like below:
```auto
input {
file {
path => [
"/opt/gtal/ital/elasticsearch/app/logstash/stage/MQA_PRD_STATS-*.txt",
"/opt/gtal/ital/elasticsearch/app/logstash/stage/MQA_DR_STATS-*.txt"
]
- - - - - - - - - - -
- - - - - - - - - - -
- - - - - - - - - - -
- - - - - - - - - - -
```
How do I modify the output part below which should direct output of "MQA\_PRD\_STATS-_.txt" and "MQA\_DR\_STATS-_.txt" separately in two indices?
```auto
output {
file {
path => "/opt/gtal/ital/elasticsearch/logs/mqa/rubydebug.txt"
codec => rubydebug
}
elasticsearch {
hosts => ["xxxxxxx:43045","xxxxxxxx:43045","xxxxxxxx:43045","xxxxxxxx:43045"]
user => "elastic"
password => "xxxxxxxx"
index => "demo-csv-%{+YYYY.MM.dd}"
doc_as_upsert => true
action => "update"
document_id => "%{my_fingerprint}"
}
}
```
Please guide.
Thanks
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [July 28, 2021, 9:27pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906/2 "2021-07-28T21:27:14Z")
Hi,
I think, the easiest way to achive what you want is to create a new field who is gonna take the name of the file you are currently reading and put that file name in the index.
```auto
input {
...
}
filter {
grok {
match => {
# Take the value between a slash and the extention
# and put this value in the field filename
"path" => "^%{GREEDYDATA}/{DATA:filename}[.]{WORD}$"
# In grok, greedydata take all the values until the last value that follow it
# So here, it take all the values until the last /
}
}
}
output {
file {
path => "/opt/gtal/ital/elasticsearch/logs/mqa/rubydebug.txt"
codec => rubydebug
}
elasticsearch {
hosts => ["xxxxxxx:43045","xxxxxxxx:43045","xxxxxxxx:43045","xxxxxxxx:43045"]
user => "elastic"
password => "xxxxxxxx"
#Adding the filename to the index
index => "demo-csv-%{[filename]}-%{+YYYY.MM.dd}"
doc_as_upsert => true
action => "update"
document_id => "%{my_fingerprint}"
}
}
```
Cad.
---
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [July 29, 2021, 3:28pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906/3 "2021-07-29T15:28:25Z")
Thanks. I did some research too which assigns a type to the file. Based on the file type requests go to the relevant index. Would the following work?
```auto
input {
file {
type => "PRD"
path => "/opt/gtal/ital/elasticsearch/app/logstash/stage/MQA_PRD_STATS-*.txt"
}
file {
type => "DR"
path => "/opt/gtal/ital/elasticsearch/app/logstash/stage/MQA_DR_STATS-*.txt"
}
}
-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
if [type] == "PRD" {
elasticsearch {
hosts => ["xx-xxx-xxxx:43045","xx-xxx-xxxx:43045","xx-xxx-xxxx:43045","xx-xxx-xxxx:43045","xx-xxx-xxxx:43045"]
user => "xxxxxx"
password => "xxxxxxxxxxxxxxxxxxxxxxx"
index => "prd-csv-%{+YYYY.MM.dd}"
doc_as_upsert => true
action => "update"
document_id => "%{my_fingerprint}"
}
}
if [type] == "DR" {
elasticsearch {
hosts => ["xx-xxx-xxxx:43045","xx-xxx-xxxx:43045","xx-xxx-xxxx:43045","xx-xxx-xxxx:43045","xx-xxx-xxxx:43045"]
user => "xxxx"
password => "xxxxxxxxxxxxxxxxxxxxxxx"
index => "dr-csv-%{+YYYY.MM.dd}"
doc_as_upsert => true
action => "update"
document_id => "%{my_fingerprint}"
}
}
}
```
The resultant would be two indices prd-csv-%{+YYYY.MM.dd} and dr-csv-%{+YYYY.MM.dd}
Please guide.
Thanks
---
### Author:  [@grumo35](https://discuss.elastic.co/u/grumo35)
#### Post date: [July 29, 2021, 3:34pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906/4 "2021-07-29T15:34:02Z")
if you want to do it before the output something like this would actually work
```auto
mutate{
add_field => {"[@metadata][target_index] => "index-%{[filename]}-%{+YYYY.MM.dd}"}}
}
```
---
### Author:  [@zaeemmasood](https://discuss.elastic.co/u/zaeemmasood)
#### Post date: [July 29, 2021, 4:02pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906/5 "2021-07-29T16:02:46Z")
Thanks.
> [@grumo35](#):
>
> ```auto
> mutate{
> add_field => {"[@metadata][target_index] => "index-%{[filename]}-%{+YYYY.MM.dd}"}}
> }
>
> ```
Do I need to replace "filename" above with the name of the file e.g. MQA\_PRD\_STATS-\*.txt and the other one in my case?
Also how would the output part look like?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 26, 2021, 4:03pm UTC](https://discuss.elastic.co/t/logstash-file-input-output-to-different-indices/279906/6 "2021-08-26T16:03:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.