# Executing multiple .conf files in one instance
**URL:** https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749
**Category:** Logstash
**Created:** [August 14, 2023, 11:51am UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749 "2023-08-14T11:51:06Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@Tony\_Stark](https://discuss.elastic.co/u/Tony_Stark)
#### Post date: [August 14, 2023, 11:51am UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/1 "2023-08-14T11:51:06Z")
I am trying to execute multiple .conf files in logstash with logstash -f "path\*.conf" but logstash processes the .conf file the same number of times as the number of .conf files I have , if I have 5 .conf files , I get output for one document as loglevel:[INFO,INFO,INFO,INFO,INFO]. How can I prevent the duplicates.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 14, 2023, 12:26pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/2 "2023-08-14T12:26:45Z")
> [@Tony\_Stark](#):
>
> I am trying to execute multiple .conf files in logstash with logstash -f "path\*.conf" but logstash processes the .conf file the same number of times as the number of .conf files I have
This is the expected behavior, when you run Logstash this way it will merge all the `*.conf` files in one single pipeline, all inputs will be snt to all outputs unless you use conditionals in your outputs.
When you want to run multiple pipelines (_\*.conf files_) in a way that they are independent from each other you need to use the `pipelines.yml` file and configure the multiple pipelines in this file, after that you need to run Logstash as a service or without the `-f` option.
The documentation for multiple pipelines can be found [here](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html).
---
### Author:  [@Tony\_Stark](https://discuss.elastic.co/u/Tony_Stark)
#### Post date: [August 14, 2023, 12:59pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/3 "2023-08-14T12:59:08Z")
I tried that , I don’t know why but only the first pipeline was executed , logstash started and ended both pipelines but only the logs of the first pipeline were indexed.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 14, 2023, 1:05pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/4 "2023-08-14T13:05:02Z")
> [@Tony\_Stark](#):
>
> I don’t know why but only the first pipeline was executed
You would need to share your `pipelines.yml` and logstash logs when you start it.
---
### Author:  [@Tony\_Stark](https://discuss.elastic.co/u/Tony_Stark)
#### Post date: [August 16, 2023, 6:58am UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/5 "2023-08-16T06:58:39Z")
```auto
- pipeline.id: my_first_pipeline
path.config: "D:\\logstash-8.8.1\\config\\logstash_1.conf"
- pipeline.id: my_second_pipeline
path.config: "D:\\logstash-8.8.1\\config\\logstash_2.conf"
```
---
### Author:  [@Tony\_Stark](https://discuss.elastic.co/u/Tony_Stark)
#### Post date: [August 16, 2023, 7:01am UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/6 "2023-08-16T07:01:26Z")
`[2023-08-16T12:15:22,511][INFO][logstash.runner] Log4j configuration path used is: D:\logstash-8.8.1\config\log4j2.properties [2023-08-16T12:15:22,584][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.8.1", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-mswin32]"} [2023-08-16T12:15:22,615][INFO][logstash.runner] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2023-08-16T12:15:34,138][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [2023-08-16T12:15:34,986][INFO][org.reflections.Reflections] Reflections took 906 ms to scan 1 urls, producing 132 keys and 464 values [2023-08-16T12:15:38,098][INFO][logstash.javapipeline] Pipeline `my\_first\_pipeline`is configured with`pipeline.ecs\_compatibility: v8`setting. All plugins in this pipeline will default to`ecs\_compatibility =\> v8`unless explicitly configured otherwise. [2023-08-16T12:15:38,101][INFO][logstash.javapipeline] Pipeline`my\_second\_pipeline`is configured with`pipeline.ecs\_compatibility: v8`setting. All plugins in this pipeline will default to`ecs\_compatibility =\> v8`unless explicitly configured otherwise. [2023-08-16T12:15:38,272][INFO][logstash.outputs.elasticsearch][my_first_pipeline] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]} [2023-08-16T12:15:38,289][INFO][logstash.outputs.elasticsearch][my_second_pipeline] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]} [2023-08-16T12:15:39,137][INFO][logstash.outputs.elasticsearch][my_first_pipeline] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@localhost:9200/]}} [2023-08-16T12:15:39,186][INFO][logstash.outputs.elasticsearch][my_second_pipeline] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@localhost:9200/]}} [2023-08-16T12:15:39,414][WARN][logstash.outputs.elasticsearch][my_second_pipeline] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@localhost:9200/"} [2023-08-16T12:15:39,424][INFO][logstash.outputs.elasticsearch][my_second_pipeline] Elasticsearch version determined (8.8.1) {:es_version=>8} [2023-08-16T12:15:39,416][WARN][logstash.outputs.elasticsearch][my_first_pipeline] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@localhost:9200/"} [2023-08-16T12:15:39,464][WARN][logstash.outputs.elasticsearch][my_second_pipeline] Detected a 6.x and above cluster: the`type`event field won't be used to determine the document _type {:es_version=>8} [2023-08-16T12:15:39,487][INFO][logstash.outputs.elasticsearch][my_first_pipeline] Elasticsearch version determined (8.8.1) {:es_version=>8} [2023-08-16T12:15:39,680][INFO][logstash.outputs.elasticsearch][my_second_pipeline] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"forex"} [2023-08-16T12:15:39,681][WARN][logstash.outputs.elasticsearch][my_first_pipeline] Detected a 6.x and above cluster: the`type` event field won't be used to determine the document _type {:es_version=>8} [2023-08-16T12:15:39,745][INFO][logstash.outputs.elasticsearch][my_second_pipeline] Data streams auto configuration (`data\_stream =\> auto`or unset) resolved to`false` [2023-08-16T12:15:39,825][INFO][logstash.outputs.elasticsearch][my_first_pipeline] Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"index"=>"holiday"} [2023-08-16T12:15:40,006][WARN][logstash.filters.grok][my_second_pipeline] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [2023-08-16T12:15:40,008][INFO][logstash.outputs.elasticsearch][my_first_pipeline] Data streams auto configuration (`data\_stream =\> auto`or unset) resolved to`false` [2023-08-16T12:15:40,106][INFO][logstash.outputs.elasticsearch][my_second_pipeline] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8} [2023-08-16T12:15:43,376][INFO][logstash.outputs.elasticsearch][my_first_pipeline] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8} [2023-08-16T12:15:43,368][WARN][logstash.filters.grok][my_first_pipeline] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated [2023-08-16T12:15:43,675][INFO][logstash.javapipeline][my_first_pipeline] Starting pipeline {:pipeline_id=>"my_first_pipeline", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["D:/logstash-8.8.1/config/logstash_2.conf"], :thread=>"#"} [2023-08-16T12:15:43,630][INFO][logstash.javapipeline][my_second_pipeline] Starting pipeline {:pipeline_id=>"my_second_pipeline", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["D:/logstash-8.8.1/config/logstash_1.conf"], :thread=>"#"} [2023-08-16T12:15:45,812][INFO][logstash.javapipeline][my_second_pipeline] Pipeline Java execution initialization time {"seconds"=>2.11} [2023-08-16T12:15:45,818][INFO][logstash.javapipeline][my_first_pipeline] Pipeline Java execution initialization time {"seconds"=>2.12} [2023-08-16T12:15:46,175][INFO][logstash.inputs.file][my_second_pipeline] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"D:/logstash-8.8.1/data/plugins/inputs/file/.sincedb_9457514873289fb1eaf23c9dcf288064", :path=>["C:/Users/SoniAns/Documents/tcilforex.log"]} [2023-08-16T12:15:46,224][INFO][logstash.inputs.file][my_first_pipeline] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"D:/logstash-8.8.1/data/plugins/inputs/file/.sincedb_618e495e1a65e7878af0fe49b655ec67", :path=>["D:/tcilholiday.log"]} [2023-08-16T12:15:46,276][INFO][logstash.javapipeline][my_second_pipeline] Pipeline started {"pipeline.id"=>"my_second_pipeline"} [2023-08-16T12:15:46,327][INFO][logstash.javapipeline][my_first_pipeline] Pipeline started {"pipeline.id"=>"my_first_pipeline"} [2023-08-16T12:15:46,324][INFO][filewatch.observingtail][my_second_pipeline][1103a5f15e480902d0349b8b5ca23721d069f4b5dd8081f859f3f44ae8b482c9] START, creating Discoverer, Watch with file and sincedb collections [2023-08-16T12:15:46,343][INFO][filewatch.observingtail][my_first_pipeline][69e4728f18e1f0f92cb26647c04f6fda8a809fee27b84e828b607d9d44703dcb] START, creating Discoverer, Watch with file and sincedb collections [2023-08-16T12:15:46,450][INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:my_first_pipeline, :my_second_pipeline], :non_running_pipelines=>[]}`
`
> Blockquote
`
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 16, 2023, 12:26pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/7 "2023-08-16T12:26:19Z")
> [2023-08-16T12:15:46,450][INFO][logstash.agent] Pipelines running {:count=\>2, :running\_pipelines=\>[:my\_first\_pipeline, :my\_second\_pipeline], :non\_running\_pipelines=\>}
Both of your pipelines are running.
But there is something a little confusing
> [2023-08-16T12:15:43,675][INFO][logstash.javapipeline][**my\_first\_pipeline**] Starting pipeline {:pipeline\_id=\>"my\_first\_pipeline", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>250, "pipeline.sources"=\>[" **D:/logstash-8.8.1/config/logstash\_2.conf**"]
And
> [2023-08-16T12:15:43,630][INFO][logstash.javapipeline][**my\_second\_pipeline**] Starting pipeline {:pipeline\_id=\>"my\_second\_pipeline", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>250, "pipeline.sources"=\>[" **D:/logstash-8.8.1/config/logstash\_1.conf**"]
Did you edit anything in your log before sharing? It seems that the log was edited, this can lead to confusion.
To understand why you are receiving data from just one pipeline you would need to share the conf file for each one of your pipelines.
Share them using the preformatted text option, the ``, just paste the configuration, select it and click on the buttom to properly format as code.
---
### Author:  [@Tony\_Stark](https://discuss.elastic.co/u/Tony_Stark)
#### Post date: [August 16, 2023, 12:39pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/8 "2023-08-16T12:39:40Z")
I didn't edit the log files , which is why I was confused too, I couldn't troubleshoot.
Here is the conf file
```auto
input {
file {
path => ["D:/forexsample.log"]
start_position => "beginning"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => "previous"
}
}
}
filter {
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:loglevel} \(%{GREEDYDATA:class}\:%{GREEDYDATA:method}\:%{NUMBER:line}\) - %{GREEDYDATA:error_message}"
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
user => "elastic"
index => "forex"
password => "VcBQRCx+Db9c2ATbLMFe"
}
stdout {
codec => rubydebug
}
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 16, 2023, 12:44pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/9 "2023-08-16T12:44:13Z")
> [@Tony\_Stark](#):
>
> I didn't edit the log files
Then something may be wrong, because your `pipelines.yml` looks like this:
```auto
- pipeline.id: my_first_pipeline
path.config: "D:\\logstash-8.8.1\\config\\logstash_1.conf"
- pipeline.id: my_second_pipeline
path.config: "D:\\logstash-8.8.1\\config\\logstash_2.conf"
```
So `my_first_pipeline` should run `logstash_1.conf` and `my_second_pipeline` should run `logstash_2.conf`
But your logs says that they are inverted. Double check the `pipelines.yml` to confirm it, to troubleshoot you may remove/comment one of the pipelines and see if it is indeed running this `pipelines.yml`.
> [@Tony\_Stark](#):
>
> Here is the conf file
You have 2 conf files, please share both and specify which pipeline is by the name of the conf file.
---
### Author:  [@Tony\_Stark](https://discuss.elastic.co/u/Tony_Stark)
#### Post date: [August 16, 2023, 12:48pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/10 "2023-08-16T12:48:52Z")
Both the conf files are the same, only the path and index have been changed.
Ya I had inter-changed the logstash\_1 and logstash\_2 just to see if they would work other way round.
Can you please help me troubleshoot if there are any issues in the conf file
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 13, 2023, 12:49pm UTC](https://discuss.elastic.co/t/executing-multiple-conf-files-in-one-instance/340749/11 "2023-09-13T12:49:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.