How to set sequence to run multiple pipeline

Elastic Stack Logstash
1

Hi all,
I'm working on multiple pipeline, I've created 3 different pipeline on single instance.
but on running logstash i'm not getting the output in the sequece.

is there any way to get output according mentioned sequence in pipelines.yml file?

Following is my pipelilnes.yml file:

- pipeline.id: main1
  pipeline.workers: 1
  config.string: |
          input{
                  exec {
                         command => "ls"
                         interval => 60
                  }
          }

          output{
                  stdout{}
          }


- pipeline.id: main2
  pipeline.workers: 2
  config.string: |
          input{
                  exec {
                         command => "pwd"
                         interval => 60
                  }      
          }       
          
          output{
                  stdout{}
          }

          

- pipeline.id: main3
  pipeline.workers: 3
  config.string: |
          input{
                  exec {
                         command => "ifconfig"
                         interval => 60
                  }      
          }       
          
          output{
                  stdout{}
          }

this is my output:

Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2018-10-11T17:33:55,070][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.0"}
[2018-10-11T17:33:57,878][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main2", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-11T17:33:57,879][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main1", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-11T17:33:57,880][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main3", "pipeline.workers"=>3, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-10-11T17:33:57,921][INFO ][logstash.inputs.exec     ] Registering Exec Input {:type=>nil, :command=>"pwd", :interval=>60, :schedule=>nil}
[2018-10-11T17:33:57,921][INFO ][logstash.inputs.exec     ] Registering Exec Input {:type=>nil, :command=>"ls", :interval=>60, :schedule=>nil}
[2018-10-11T17:33:57,921][INFO ][logstash.inputs.exec     ] Registering Exec Input {:type=>nil, :command=>"ifconfig", :interval=>60, :schedule=>nil}
[2018-10-11T17:33:57,940][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main1", :thread=>"#<Thread:0x5e8bca93@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:156 run>"}
[2018-10-11T17:33:57,940][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main3", :thread=>"#<Thread:0x10abee3f@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:46 run>"}
[2018-10-11T17:33:57,941][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main2", :thread=>"#<Thread:0x98b76e5@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:156 sleep>"}
[2018-10-11T17:33:57,993][INFO ][logstash.agent           ] Pipelines running {:count=>3, :running_pipelines=>[:main1, :main2, :main3], :non_running_pipelines=>[]}
[2018-10-11T17:33:58,452][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
       "message" => "output of the command",
          "host" => "chitresh-HP-Compaq-Elite-8300-CMT",
    "@timestamp" => 2018-10-11T12:03:58.034Z,
       "command" => "ifconfig",
      "@version" => "1"
}
{
       "message" => "output of the command",
          "host" => "chitresh-HP-Compaq-Elite-8300-CMT",
    "@timestamp" => 2018-10-11T12:03:58.034Z,
       "command" => "ls",
      "@version" => "1"
}
{
       "message" => "output of the command",
          "host" => "chitresh-HP-Compaq-Elite-8300-CMT",
    "@timestamp" => 2018-10-11T12:03:58.034Z,
       "command" => "pwd",
      "@version" => "1"
}
^C[2018-10-11T17:34:07,961][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2018-10-11T17:34:08,105][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main3", :thread=>"#<Thread:0x10abee3f@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:46 run>"}
[2018-10-11T17:34:08,153][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main2", :thread=>"#<Thread:0x98b76e5@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:156 run>"}
[2018-10-11T17:34:08,155][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"main1", :thread=>"#<Thread:0x5e8bca93@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:156 run>"}

Any help will be appreciate.

2

Logstash has no guarantees of event processing in any specific order. The timestamps show that the events were generated at the same millisecond, as the pipelines are separate threaded executions it is uncertain which thread gest to write to stdout first.

3

Thanks for your comment,
but I already got my solution, we can't run pipeline in sequence but we can give some delay time to run other pipelines by using batch delay so it will help to run all pipeline one by one.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.