# Logstash: input file plugin and tcp plugins

**URL:** <https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058>\
**Category:** Logstash\
**Created:** [May 14, 2021, 6:53pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058 "2021-05-14T18:53:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 6:53pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/1 "2021-05-14T18:53:35Z")

</div>

I have 4 pipelines. 3 using tcp plugin and 1 using file plugin.  
pipeline with input file plugin works fine independently, but when i add 3 other tcp plugin pipeline, the input file plugin pipeline stops running (after successfully starting) as shown below.

Any ideas?

````auto
Pipelines running {:count=>3, :running_pipelines=>[:stp26x, :security1100, :tcap1100], :non_running_pipelines=>[:rawdiameter1200]} ```
````

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2021, 6:57pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/2 "2021-05-14T18:57:12Z")

</div>

What else does it log?

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 7:00pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/3 "2021-05-14T19:00:39Z")

</div>

```auto
[2021-05-14T17:43:01,579][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"rawdiameter1200", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/rawdiameter1200/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/101-date.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/200-descriptions.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/201-mncMcc.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/202-inbound-outbound-messages.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/203-requests-answers-errors.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/300-removals.conf", "/opt/elastic-release/logstash/pipeline/rawdiameter1200/900-output.conf"], :thread=>"#<Thread:0x43876859@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:121 run>"}
[2021-05-14T17:43:12,442][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"rawdiameter1200"}
[2021-05-14T17:43:12,923][INFO][filewatch.observingread] START, creating Discoverer, Watch with file and sincedb collections
[2021-05-14T17:45:07,795][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"stp26x", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/stp26x/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/stp26x/101-date.conf", "/opt/elastic-release/logstash/pipeline/stp26x/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/stp26x/300-traffic.conf", "/opt/elastic-release/logstash/pipeline/stp26x/301-lookups.conf", "/opt/elastic-release/logstash/pipeline/stp26x/400-msuFlagging.conf", "/opt/elastic-release/logstash/pipeline/stp26x/900-output.conf"], :thread=>"#<Thread:0x58d28e2e@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
[2021-05-14T17:45:09,094][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"security1100", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/security1100/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/security1100/102-date.conf", "/opt/elastic-release/logstash/pipeline/security1100/200-tcapLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/201-sccpLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/300-secMessagesLookup.conf", "/opt/elastic-release/logstash/pipeline/security1100/301-secMessages.conf", "/opt/elastic-release/logstash/pipeline/security1100/302-JanKOrdering.yml", "/opt/elastic-release/logstash/pipeline/security1100/400-fieldRemoval.conf", "/opt/elastic-release/logstash/pipeline/security1100/900-output.conf"], :thread=>"#<Thread:0x4d71f52b@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:53 run>"}
[2021-05-14T17:45:10,343][INFO][logstash.javapipeline] Starting pipeline {:pipeline_id=>"tcap1100", "pipeline.workers"=>16, "pipeline.batch.size"=>150, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2400, "pipeline.sources"=>["/opt/elastic-release/logstash/pipeline/tcap1100/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/102-date.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/201-sccpLookups.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/300-secMessagesLookup.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/301-secMessages.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/400-fieldRemoval.conf", "/opt/elastic-release/logstash/pipeline/tcap1100/900-output.conf"], :thread=>"#<Thread:0x7eec1966 run>"}
[2021-05-14T17:46:16,374][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"tcap1100"}
[2021-05-14T17:46:16,404][INFO][logstash.inputs.tcp] Starting tcp input listener {:address=>"0.0.0.0:5003", :ssl_enable=>"false"}
[2021-05-14T17:46:24,681][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"stp26x"}
[2021-05-14T17:46:24,689][INFO][logstash.inputs.tcp] Starting tcp input listener {:address=>"0.0.0.0:5006", :ssl_enable=>"false"}
[2021-05-14T17:46:25,954][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"security1100"}
[2021-05-14T17:46:25,963][INFO][logstash.inputs.tcp] Starting tcp input listener {:address=>"0.0.0.0:5002", :ssl_enable=>"false"}
[2021-05-14T17:46:26,070][INFO][logstash.agent] Pipelines running {:count=>3, :running_pipelines=>[:stp26x, :security1100, :tcap1100], :non_running_pipelines=>[:rawdiameter1200]}
[2021-05-14T17:46:26,793][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2021, 7:12pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/4 "2021-05-14T19:12:23Z")

</div>

Can you increase log.level to debug?

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 7:16pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/5 "2021-05-14T19:16:19Z")

</div>

Sorry i am not as experienced, where can is set this?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2021, 7:31pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/6 "2021-05-14T19:31:48Z")

</div>

You can set it logstash.yml, or on the command line using `--log.level debug`

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 7:47pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/7 "2021-05-14T19:47:39Z")

</div>

Thanks. should i be looking for some pattern? its a giant file...

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 7:50pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/8 "2021-05-14T19:50:36Z")

</div>

i see this and other messages....

```auto
a
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x69bc39c3@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x1432e029@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x6744353c@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x1494a046@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x4e44cfba@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x49b88325@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,733][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x5643f03d@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x5f2328c0@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x3e987eb8@/usr/share/logstash/logstash-core/lib/logsta
sh/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x535f29e3@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x44e5758d@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170 dead>"}
[2021-05-14T19:47:02,734][DEBUG][logstash.javapipeline] Shutdown waiting for worker thread {:pipeline_id=>"rawdiameter1200", :thread=>"#<Thread:0x1860c57a@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:170 dead>"}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2021, 8:49pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/9 "2021-05-14T20:49:48Z")

</div>

I would expect there to be an exception before those start. Can you try disabling java execution? Add `--java-execution false` to the command line and see if you get a different message.

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 8:52pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/10 "2021-05-14T20:52:06Z")

</div>

> [@Badger](#):
>
> --java-execution false

can i add this to pipeline.yml?

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 8:54pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/11 "2021-05-14T20:54:36Z")

</div>

> [@hsalim](#):
>
> > [@Badger](#):
> >
> > --java-execution false

actually not sure where do you mean by command line? i am doing systemctl restart logstash

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 9:00pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/12 "2021-05-14T21:00:45Z")

</div>

ok this is the error

````auto
[2021-05-14T20:59:26,332][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:security1100, :exception=>"LogStash::ConfigurationError", :message=>"Could not determine ID for filter/date, source don't matched: [file]/opt/elastic-release/logstash/pipeline/security1100/102-date.conf:3:1:```\nfilter {\ndate {\n match => [\"observation_time\", \"UNIX_MS\"]\n}\n\n#date {\n# match => [\"observation_hour\", \"UNIX_MS\"]\n# target => \"observation_hour\"\n#}\n}\n\n```", :backtrace=>["org/logstash/plugins/PluginFactoryExt.java:237:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:93:in `plugin'", "(eval):1263:in `initialize'", "org/jruby/RubyKernel.java:1048:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}

````

but this is on all pipelines, even the ones that kept running.  
any help is appreciated...!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2021, 9:23pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/13 "2021-05-14T21:23:21Z")

</div>

Add `pipeline.java_execution: false` to logstash.yml

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 9:26pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/14 "2021-05-14T21:26:02Z")

</div>

i did and it generated errors in my previous post.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 14, 2021, 9:59pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/15 "2021-05-14T21:59:16Z")

</div>

> [@hsalim](#):
>
> `Could not determine ID for filter/date`

This is a [known issue](https://github.com/elastic/logstash/issues/11698). I am out of ideas as to why the pipeline would stop without logging any indication of why it did so.

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 14, 2021, 10:13pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/16 "2021-05-14T22:13:23Z")

</div>

ok at least now i know that i am not going crazy here!

interestingly input file filter works find by itself, and only fails when i add other pipelines with TCP input filter. so do you think this behavior is because of ' Could not determine ID for filter/date`?

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 15, 2021, 3:25am UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/17 "2021-05-15T03:25:02Z")

</div>

why does this happen only with pipeline.java\_execution: false?

I think there is something else going on.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2021, 3:11pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/18 "2021-05-15T15:11:50Z")

</div>

> [@hsalim](#):
>
> why does this happen only with pipeline.java\_execution: false?

Which problem are you referring to?

---

<div class="post-metadata">

**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\
**Post date:** [May 15, 2021, 3:34pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/19 "2021-05-15T15:34:21Z")

</div>

this only happens when pipeline.java\_execution: false

````auto
[2021-05-14T20:59:26,332][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:security1100, :exception=>"LogStash::ConfigurationError", :message=>"Could not determine ID for filter/date, source don't matched: [file]/opt/elastic-release/logstash/pipeline/security1100/102-date.conf:3:1:```\nfilter {\ndate {\n match => [\"observation_time\", \"UNIX_MS\"]\n}\n\n#date {\n# match => [\"observation_hour\", \"UNIX_MS\"]\n# target => \"observation_hour\"\n#}\n}\n\n```", :backtrace=>["org/logstash/plugins/PluginFactoryExt.java:237:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:210:in `plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:93:in `plugin'", "(eval):1263:in `initialize'", "org/jruby/RubyKernel.java:1048:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}```
````

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2021, 3:41pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/20 "2021-05-15T15:41:03Z")

</div>

Yes, and the open issue suggests it only happens for the second file of the configuration. It will probably never get fixed because the ruby execution engine will be decommissioned.

[Next page](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058.md?page=2)