Convert Logstash Main pipeline to multiple pipelines

Elastic Stack Logstash
1

Hello team,
I have a main pipeline that has multiple inputs like beats, kafka and filters for many application and single and big output file with many if, else if and else.

The pipeline is so messed now and am not certain if I add any more else if to my output its being processed.

I want to convert to multiple pipelines where I want to separate based on apps. This is what I understood after reading documents

  1. Create pipelines.yml
- pipeline.id: app1
  path.config: "app1.config"
- pipeline.id: app2
  path.config: "app2.config"
- pipeline.id: app3
  path.config: "app3.config"
  1. create app1.config
input{beats}
filter{if XXX then ...}
output{elasticsearch}
  1. create app2.config
input{beats}
filter{if XXX then ...}
output{elasticsearch}
  1. create app3.config
input{kafka}
filter{if XXX then ...}
output{elasticsearch}

Please help me if this is correct and also if you have any documentation on how to start converting... Can I just go ahead and start creating these files by taking out one configuration at a time or any process to follow. Please guide me.

2

Yes, that looks correct. I would suggest you do change only one configuration at a time.

3

So here is what I have

[2022-11-15T18:30:28,967][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
4

I run Logstash on docker and mount a volume with pipelines.

5

okay... one thing I messed up was the pipelines.yml location is in config but not configs. Simple typo caused this error. To summarize for others

/usr/share/logstash/config/pipelines.yml
and pipeline configurations in 
/usr/share/logstash/pipeline

Restart of docker is needed. FYI

6

However @Badger, need your help. I am unable to make changes to the pipelines. Restarted the docker many times and also added following configuration still it wont change.

- name: starting elasticsearch logstash container 1
      delegate_to: 1.2.3.4
      become: true
      docker_container:
        name: logstash-1
        image: "docker.elastic.co/logstash/logstash:7.10.0"
        user: root
        env:
          node.name: es-logstash-1
          http.host: "0.0.0.0"
          xpack.monitoring.elasticsearch.hosts: 
          xpack.monitoring.enabled: "true"
          xpack.monitoring.elasticsearch.username: elc
          xpack.monitoring.elasticsearch.password: XXX
          xpack.management.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certificates/ca/ca.crt
          xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certificates/ca/ca.crt
          ES_JAVA_OPTS: -Xms8g -Xmx8g
          config.test_and_exit: "false"
          config.reload.automatic: "false"
          config.reload.interval: 60s

        volumes:
          - "/etc/ssl/certs/.logstash_certs:/usr/share/logstash/config/certificates"
          - "/logstashdata/conf.d:/usr/share/logstash/pipeline"
          - "/logstashdata/pipelines.yml:/usr/share/logstash/config/pipelines.yml"
        ulimits:
          - nofile:65535:65535
          - memlock:-1:-1
        network_mode: host
        state: started
7

I have never used docker so I cannot provide help.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.