Logstash: input file plugin and tcp plugins

Elastic Stack Logstash
1

I have 4 pipelines. 3 using tcp plugin and 1 using file plugin.
pipeline with input file plugin works fine independently, but when i add 3 other tcp plugin pipeline, the input file plugin pipeline stops running (after successfully starting) as shown below.

Any ideas?

Pipelines running {:count=>3, :running_pipelines=>[:stp26x, :security1100, :tcap1100], :non_running_pipelines=>[:rawdiameter1200]} ```
2

What else does it log?

3 
[2021-05-14T17:43:01,579][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"rawdiameter1200", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/rawdiameter1200/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/101-date.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/200-descriptions.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/201-mncMcc.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/202-inbound-outbound-messages.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/203-requests-answers-errors.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/300-removals.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/900-output.conf"], :thread=>"#<Thread:0x43876859@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:121 run>"}
[2021-05-14T17:43:12,442][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"rawdiameter1200"}
[2021-05-14T17:43:12,923][INFO ][filewatch.observingread  ] START, creating Discoverer, Watch with file and sincedb collections
[2021-05-14T17:45:07,795][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"stp26x", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/stp26x/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/stp26x/101-date.conf", "/opt/elastic-release/logstash/pipeline/stp26x/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/stp26x/300-traffic.conf", "/opt/elastic-release/logstash/pipeline/stp26x/301-lookups.conf", "/opt/elastic-release/logstash/pipeline/stp26x/400-msuFlagging.conf", "/opt/elastic-release/logstash/pipeline/stp26x/900-output.conf"], :thread=>"#<Thread:0x58d28e2e@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
[2021-05-14T17:45:09,094][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"security1100", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/security1100/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/security1100/102-date.conf", "/opt/elastic-release/logstash/pipeline/security1100/200-tcapLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/201-sccpLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/300-secMessagesLookup.conf", "/opt/elastic-release/logstash/pipeline/security1100/301-secMessages.conf", "/opt/elastic-release/logstash/pipeline/security1100/302-JanKOrdering.yml", "/opt/elastic-release/logstash/pipeline/security1100/400-fieldRemoval.conf", "/opt/elastic-release/logstash/pipeline/security1100/900-output.conf"], :thread=>"#<Thread:0x4d71f52b@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:53 run>"}
[2021-05-14T17:45:10,343][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"tcap1100", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/tcap1100/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/102-date.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/201-sccpLookups.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/300-secMessagesLookup.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/301-secMessages.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/400-fieldRemoval.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/900-output.conf"], :thread=>"#<Thread:0x7eec1966 run>"}
[2021-05-14T17:46:16,374][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"tcap1100"}
[2021-05-14T17:46:16,404][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5003", :ssl_enable=>"false"}
[2021-05-14T17:46:24,681][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"stp26x"}
[2021-05-14T17:46:24,689][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5006", :ssl_enable=>"false"}
[2021-05-14T17:46:25,954][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"security1100"}
[2021-05-14T17:46:25,963][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5002", :ssl_enable=>"false"}
[2021-05-14T17:46:26,070][INFO ][logstash.agent           ] Pipelines running {:count=>3, :running_pipelines=>[:stp26x, :security1100, :tcap1100], :non_running_pipelines=>[:rawdiameter1200]}
[2021-05-14T17:46:26,793][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
4

Can you increase log.level to debug?

5

Sorry i am not as experienced, where can is set this?

6

You can set it logstash.yml, or on the command line using --log.level debug

7

Thanks. should i be looking for some pattern? its a giant file...

8

i see this and other messages....

a
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x69bc39c3@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x1432e029@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x6744353c@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x1494a046@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x4e44cfba@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x49b88325@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x5643f03d@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x5f2328c0@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x3e987eb8@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x535f29e3@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x44e5758d@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline    ] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x1860c57a@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170 dead>"}
9

I would expect there to be an exception before those start. Can you try disabling java execution? Add --java-execution false to the command line and see if you get a different message.

10

can i add this to pipeline.yml?

11

actually not sure where do you mean by command line? i am doing systemctl restart logstash

12

ok this is the error

[2021-05-14T20:59:26,332][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:security1100, :exception=>"LogStash::ConfigurationError", :message=>"Could not determine ID for filter/date, source don't matched: [file]/opt/elastic-release/logstash/pipeline/security1100/102-date.conf:3:1:```\nfilter {\ndate {\n  match => [ \"observation_time\", \"UNIX_MS\" ]\n}\n\n#date {\n#  match => [ \"observation_hour\", \"UNIX_MS\" ]\n#  target => \"observation_hour\"\n#}\n}\n\n```", :backtrace=>["org/logstash/plugins/PluginFactoryExt.java:237:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:93:in `plugin'", "(eval):1263:in `initialize'", "org/jruby/RubyKernel.java:1048:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}

but this is on all pipelines, even the ones that kept running.
any help is appreciated...!

13

Add pipeline.java_execution: false to logstash.yml

14

i did and it generated errors in my previous post.

15

This is a known issue. I am out of ideas as to why the pipeline would stop without logging any indication of why it did so.

16

ok at least now i know that i am not going crazy here!

interestingly input file filter works find by itself, and only fails when i add other pipelines with TCP input filter. so do you think this behavior is because of ' Could not determine ID for filter/date`?

17

why does this happen only with pipeline.java_execution: false?

I think there is something else going on.

18

Which problem are you referring to?

19

this only happens when pipeline.java_execution: false

[2021-05-14T20:59:26,332][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:security1100, :exception=>"LogStash::ConfigurationError", :message=>"Could not determine ID for filter/date, source don't matched: [file]/opt/elastic-release/logstash/pipeline/security1100/102-date.conf:3:1:```\nfilter {\ndate {\n  match => [ \"observation_time\", \"UNIX_MS\" ]\n}\n\n#date {\n#  match => [ \"observation_hour\", \"UNIX_MS\" ]\n#  target => \"observation_hour\"\n#}\n}\n\n```", :backtrace=>["org/logstash/plugins/PluginFactoryExt.java:237:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:93:in `plugin'", "(eval):1263:in `initialize'", "org/jruby/RubyKernel.java:1048:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}```
20

Yes, and the open issue suggests it only happens for the second file of the configuration. It will probably never get fixed because the ruby execution engine will be decommissioned.