# Convert Logstash Main pipeline to multiple pipelines **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [November 14, 2022, 7:18pm UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899 "2022-11-14T19:18:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [November 14, 2022, 7:18pm UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/1 "2022-11-14T19:18:20Z")
Hello team, I have a main pipeline that has multiple inputs like beats, kafka and filters for many application and single and big output file with many if, else if and else. The pipeline is so messed now and am not certain if I add any more else if to my output its being processed. I want to convert to multiple pipelines where I want to separate based on apps. This is what I understood after reading documents 1. Create pipelines.yml ```yaml - pipeline.id: app1 path.config: "app1.config" - pipeline.id: app2 path.config: "app2.config" - pipeline.id: app3 path.config: "app3.config" ``` 1. create app1.config ```auto input{beats} filter{if XXX then ...} output{elasticsearch} ``` 1. create app2.config ```auto input{beats} filter{if XXX then ...} output{elasticsearch} ``` 1. create app3.config ```auto input{kafka} filter{if XXX then ...} output{elasticsearch} ``` Please help me if this is correct and also if you have any documentation on how to start converting... Can I just go ahead and start creating these files by taking out one configuration at a time or any process to follow. Please guide me. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 14, 2022, 8:49pm UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/2 "2022-11-14T20:49:09Z")
> [@adityak248](#): > > Please help me if this is correct and also if you have any documentation on how to start converting... Can I just go ahead and start creating these files by taking out one configuration at a time Yes, that looks correct. I would suggest you do change only one configuration at a time. ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [November 15, 2022, 6:31pm UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/3 "2022-11-15T18:31:33Z")
So here is what I have ```auto [2022-11-15T18:30:28,967][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]} ``` ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [November 15, 2022, 6:34pm UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/4 "2022-11-15T18:34:49Z")
I run Logstash on docker and mount a volume with pipelines. ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [November 15, 2022, 10:13pm UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/5 "2022-11-15T22:13:44Z")
okay... one thing I messed up was the pipelines.yml location is in config but not configs. Simple typo caused this error. To summarize for others ```auto /usr/share/logstash/config/pipelines.yml and pipeline configurations in /usr/share/logstash/pipeline ``` Restart of docker is needed. FYI ---
**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\ **Post date:** [November 16, 2022, 1:49am UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/6 "2022-11-16T01:49:00Z")
However @Badger, need your help. I am unable to make changes to the pipelines. Restarted the docker many times and also added following configuration still it wont change. ```auto - name: starting elasticsearch logstash container 1 delegate_to: 1.2.3.4 become: true docker_container: name: logstash-1 image: "docker.elastic.co/logstash/logstash:7.10.0" user: root env: node.name: es-logstash-1 http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: xpack.monitoring.enabled: "true" xpack.monitoring.elasticsearch.username: elc xpack.monitoring.elasticsearch.password: XXX xpack.management.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certificates/ca/ca.crt xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certificates/ca/ca.crt ES_JAVA_OPTS: -Xms8g -Xmx8g config.test_and_exit: "false" config.reload.automatic: "false" config.reload.interval: 60s volumes: - "/etc/ssl/certs/.logstash_certs:/usr/share/logstash/config/certificates" - "/logstashdata/conf.d:/usr/share/logstash/pipeline" - "/logstashdata/pipelines.yml:/usr/share/logstash/config/pipelines.yml" ulimits: - nofile:65535:65535 - memlock:-1:-1 network_mode: host state: started ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 16, 2022, 1:55am UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/7 "2022-11-16T01:55:01Z")
I have never used docker so I cannot provide help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 14, 2022, 1:55am UTC](https://discuss.elastic.co/t/convert-logstash-main-pipeline-to-multiple-pipelines/318899/8 "2022-12-14T01:55:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.