# All files collected under on index only. Is it possible to have multiple indexes for multiple files?
**URL:** https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806
**Category:** Logstash
**Created:** [December 23, 2019, 5:14am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806 "2019-12-23T05:14:57Z")
**Posts on this page:** 20
**Page:** 2
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 4:25pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/21 "2019-12-23T16:25:32Z")
four of them as mentioned in above comment. SHould I remove the logstash.conf since I am using pipeline.conf only?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 4:26pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/22 "2019-12-23T16:26:32Z")
Hi @Badger, I had logstash as index one time and then logstash-date-00001 as other times. are these both in ILM? Should I disable them to get the indexes I am manually typing?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 23, 2019, 4:33pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/23 "2019-12-23T16:33:38Z")
That is what I would expect.
The default rollover alias is called `logstash` , with a default pattern for the rollover index of `{now/d}-00001` , which will name indices on the date that the index is rolled over, followed by an incrementing number.
Try
```
ilm_enabled => false
```
and see if you get what you want.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 23, 2019, 4:34pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/24 "2019-12-23T16:34:15Z")
Those for files will get concatenated into a single pipeline and data from all inputs will go to all outputs unless you [set them up as separate piplines](https://www.elastic.co/guide/en/logstash/7.5/multiple-pipelines.html).
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 5:21pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/25 "2019-12-23T17:21:44Z")
@Mehak_Bhargava
Remove all the config files other that the one you need to use.
Only one input, filter, output will be supported per pipeline.
Keep only one config file with one input, filter and output and then test your results, you will get the idea!
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 5:25pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/26 "2019-12-23T17:25:54Z")
@Badger, Made the following change and still one index is show called: logstash-2019.12.23-000001
```auto
input {
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/log2.log
type => {"access"}
}
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/logz.log
type => {"errors"}
}
file{
path: /home/mehak/Documents/filebeat-7.4.0-linux-x86_64/logs/dispatcher-log.log
type => {"dispatch"}
}
}
filter {
if[type] =="access"{
grok {
match => {"message" => "%{COMBINEDAPACHELOG}"}
} else if [type] == "errors" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}else [type] == "dispatcher" {
grok {
match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}\[%{DATA:threadId}]%{SPACE}%{LOGLEVEL:logLevel}%{SPACE}%{JAVACLASS:javaClass}%{SPACE}-%{SPACE}?(\[%{NONNEGINT:incidentId}])%{GREEDYDATA:message}" }
}
}
}
output {
if [type] == "access" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "access-index"
}
} else if [type] == "errors" {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "errors-index"
}
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 5:31pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/27 "2019-12-23T17:31:55Z")
@Christian_Dahlqvist,
Currently the file I am executing is called pipeline.conf which is outside the config directory in logstash. Should I replace my pipeline.conf contents into the logstash.conf directory inside config folder and then have the pipeline.conf inside config folder with the description of multiple pipelines?
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 5:33pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/28 "2019-12-23T17:33:21Z")
Comment all the conditions in the filter for the time being and remove if conditions in the output as well and keep only one Elasticsearch instance. For now,
Like this,
output {
elasticsearch {
hosts =\> ["localhost:9200"]
sniffing =\> true
manage\_template =\> false
ilm\_enabled =\> false
index =\> "access-index"
}
}
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 5:41pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/29 "2019-12-23T17:41:21Z")
> [@Mehak\_Bhargava](#):
>
> Should I replace my pipeline.conf contents into the logstash.conf directory inside config folder and then have the pipeline.conf inside config folder with the description of multiple pipelines?
Pipeline configurations are in pipeline.yml file
You can define multiple pipelines with configuration directory, but only one input, output and filter will be supported per pipeline.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 5:42pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/30 "2019-12-23T17:42:31Z")
@mancharagopan Updated teh output and filter in pipeline.conf and I still have logstash-2019.12.23-000001 as index
```auto
input {
beats {
port => 5044
}
}
filter {
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
ilm_enabled => false
index => "access-index"
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 5:44pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/31 "2019-12-23T17:44:12Z")
Where is the pipeline.conf file located?
Please share your pipeline.yml config.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 5:49pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/32 "2019-12-23T17:49:29Z")
my pipeline.config file is in /home/mehak/Documents/logstash-7.4.0 folder.
This is my pipelines.yml in config folder-
```auto
# List of pipelines to be loaded by Logstash
#
# This document must be a list of dictionaries/hashes, where the keys/values are pipeline settings.
# Default values for omitted settings are read from the `logstash.yml` file.
# When declaring multiple pipelines, each MUST have its own `pipeline.id`.
#
# Example of two pipelines:
#
# - pipeline.id: test
# pipeline.workers: 1
# pipeline.batch.size: 1
# config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
# - pipeline.id: another_test
# queue.type: persisted
# path.config: "/tmp/logstash/*.config"
#
# Available options:
#
# # name of the pipeline
# pipeline.id: mylogs
#
# # The configuration string to be used by this pipeline
# config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
#
# # The path from where to read the configuration text
# path.config: "/etc/conf.d/logstash/myconfig.cfg"
#
# # How many worker threads execute the Filters+Outputs stage of the pipeline
# pipeline.workers: 1 (actually defaults to number of CPUs)
#
# # How many events to retrieve from inputs before sending to filters+workers
# pipeline.batch.size: 125
#
# # How long to wait in milliseconds while polling for the next event
# # before dispatching an undersized batch to filters+outputs
# pipeline.batch.delay: 50
#
# # Internal queuing model, "memory" for legacy in-memory based queuing and
# # "persisted" for disk-based acked queueing. Defaults is memory
# queue.type: memory
#
# # If using queue.type: persisted, the page data files size. The queue data consists of
# # append-only data files separated into pages. Default is 64mb
# queue.page_capacity: 64mb
#
# # If using queue.type: persisted, the maximum number of unread events in the queue.
# # Default is 0 (unlimited)
# queue.max_events: 0
#
# # If using queue.type: persisted, the total capacity of the queue in number of bytes.
# # Default is 1024mb or 1gb
# queue.max_bytes: 1024mb
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 5:53pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/33 "2019-12-23T17:53:49Z")
Is this the full configuration of pipeline.yml?
I don’t see any pipeline configuration, all are commended lines.
Is your pipeline.conf file exactly inside
`/home/mehak/Documents/logstash-7.4.0` folder?
Can you share the output of
`ls /home/mehak/Documents/logstash-7.4.0`
command?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 6:21pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/34 "2019-12-23T18:21:25Z")
Yes, the above is the entire pipeline.yml file. Should I uncomment the pipeline.id:test and pipeline.id: another\_test and if so, what should be the path.config for it? My pipeline.conf path?
Yes that is where the pipline.conf is stored.
Here is the output of ls-
```auto
mehak@mehak-VirtualBox:~$ ls /home/mehak/Documents/logstash-7.4.0
bin Gemfile logs NOTICE.TXT x-pack
config Gemfile.lock logstash-core pipeline.conf
CONTRIBUTORS lib logstash-core-plugin-api tools
data LICENSE.txt modules vendor
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 7:38pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/35 "2019-12-23T19:38:32Z")
@Mehak_Bhargava
Create a directory `conf.d` inside
`/home/mehak/Documents/logstash-7.4.0/`
Move your `pipeline.conf` into `conf.d` directory.
Uncomment `pipeline.id` and `path.config`
Change `path.config` with your `pipeline.conf` directory.
Like this,
`- pipeline.id: test `
` path.config: "/home/mehak/Documents/logstash-7.4.0/conf.d/pipeline.conf"`
And then restart logstash to take effect.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 7:44pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/36 "2019-12-23T19:44:33Z")
@mancharagopan
Made the changes, and still have same logstash-date-0001 index and this is my pipelines.yml file now-
```auto
- pipeline.id: test
path.config: "/home/mehak/Documents/logstash-7.4.0/config/pipeline.config"
pipeline.workers: 1
pipeline.batch.size: 1
```
These are the config folder files-
```auto
mehak@mehak-VirtualBox:~$ ls /home/mehak/Documents/logstash-7.4.0/config
jvm.options logstash.yml pipelines.yml
log4j2.properties pipeline.conf startup.options
mehak@mehak-VirtualBox:~$
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 23, 2019, 7:49pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/37 "2019-12-23T19:49:21Z")
@Mehak_Bhargava
I edited my last comment please check.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 23, 2019, 8:07pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/38 "2019-12-23T20:07:03Z")
@mancharagopan
updated pipelines.yml-
```auto
- pipeline.id: test
path.config: "/home/mehak/Documents/logstash-7.4.0/conf.d/pipeline.conf"
pipeline.workers: 1
pipeline.batch.size: 1
```
And these are file locations-
```auto
mehak@mehak-VirtualBox:~$ ls /home/mehak/Documents/logstash-7.4.0/config
jvm.options log4j2.properties logstash.yml pipelines.yml startup.options
mehak@mehak-VirtualBox:~$ ls /home/mehak/Documents/logstash-7.4.0/conf.d
pipeline.conf
```
Still same index in kibana.
There is this one error that shows in filebeat in between reading the log files. This has always been there-
```auto
2019-12-23T12:05:46.099-0800 ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 127.0.0.1:5044: connect: connection refused
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [December 24, 2019, 1:02am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/39 "2019-12-24T01:02:04Z")
Where is the filebeat installed? Same machine as logstash or different machine?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [December 30, 2019, 6:53pm UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/40 "2019-12-30T18:53:31Z")
Hi @mancharagopan, filebeat is on same machine.
[Previous page](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806.md?page=1)
[Next page](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806.md?page=3)