# Logstash: input file plugin and tcp plugins **URL:** \ **Category:** Logstash\ **Created:** [May 14, 2021, 6:53pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058 "2021-05-14T18:53:35Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Post date:** [May 15, 2021, 3:51pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/21 "2021-05-15T15:51:31Z")
sorry what do you mean by 'second file of the configuration'? and what is your suggestion on path forward, if we want to run 4 parallel pipelines (3 with TCP input and 1 with file input), we cant do that? appreciate your insights. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 15, 2021, 4:03pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/22 "2021-05-15T16:03:57Z")
> [@hsalim](#): > > what do you mean by 'second file of the configuration'? I mean that if you have ``` "pipeline.sources"=>[ "/opt/elastic-release/logstash/pipeline/security1100/100-ingestion.conf", "/opt/elastic-release/logstash/pipeline/security1100/102-date.conf", "/opt/elastic-release/logstash/pipeline/security1100/200-tcapLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/201-sccpLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/202-gtaLookups.conf", "/opt/elastic-release/logstash/pipeline/security1100/300-secMessagesLookup.conf", "/opt/elastic-release/logstash/pipeline/security1100/301-secMessages.conf", "/opt/elastic-release/logstash/pipeline/security1100/302-JanKOrdering.yml", "/opt/elastic-release/logstash/pipeline/security1100/400-fieldRemoval.conf", "/opt/elastic-release/logstash/pipeline/security1100/900-output.conf"], ``` then the error will occur for security1100/102-date.conf ---
**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Post date:** [May 15, 2021, 4:29pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/23 "2021-05-15T16:29:51Z")
ok. So i understand that this known issue is preventing me from running 4 parallel pipelines (3 with TCP input and 1 with file input)? is what you are suggesting is that the two are linked? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 15, 2021, 5:33pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/24 "2021-05-15T17:33:01Z")
> [@hsalim](#): > > is what you are suggesting is that the two are linked? No, I do not think they are linked. You said you only get that "Could not determine ID for filter/date" when the java execution engine is disabled, but the pipeline with the file input stops even when it is enabled. ---
**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Post date:** [May 15, 2021, 10:29pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/25 "2021-05-15T22:29:14Z")
do you have any insights on the pipeline issue i am encountering? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 15, 2021, 10:38pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/26 "2021-05-15T22:38:01Z")
No, sorry. ---
**Author:** ![Andrea\_Selva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrea_selva/32/46143_2.png) [@Andrea\_Selva](https://discuss.elastic.co/u/Andrea_Selva)\ **Post date:** [May 18, 2021, 8:38am UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/27 "2021-05-18T08:38:29Z")
Hi @hsalim we could try to reproduce it, but some information is needed to work on this. Are you able to provide a minimal pipeline config that reproduces it? Maybe a pipeline with the minimal input filter and output that manifest the problem. Which version of Logstash are you using? ---
**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Post date:** [May 20, 2021, 6:27pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/28 "2021-05-20T18:27:39Z")
Hi @Andrea_Selva issue we are having is files that are waiting to be ingested are not being marked as new\_discovery, see below, any ideas? ```auto [2021-05-20T18:20:02,794][TRACE][filewatch.discoverer] discover_files {"count"=>2254} [2021-05-20T18:20:02,794][TRACE][filewatch.discoverer] discover_files handling: {"new discovery"=>false, "watched_file details"=>""} ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 20, 2021, 6:38pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/29 "2021-05-20T18:38:49Z")
So it has read 354 bytes from a 351 byte file. That suggests [inode reuse](https://github.com/logstash-plugins/logstash-input-file/issues/213). ---
**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Post date:** [May 20, 2021, 6:48pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/30 "2021-05-20T18:48:50Z")
@Badger i am not using sincedb config is below, so not sure how inode reuse will impact me? and if for some weird reason it does what is the solution? and by the way this only happens on 3 node cluster not on a single node. ```auto input { file { path => "/home/ftpuser/tdr/input/diameter/SYSTEM/local/*.gz" max_open_files => 8000 mode => "read" start_position => "beginning" sincedb_path => "/dev/null" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 20, 2021, 7:03pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/31 "2021-05-20T19:03:16Z")
> [@hsalim](#): > > i am not using sincedb The sincedb\_path option determines whether the in-memory sincedb is persisted to disk across restarts. The in-memory db is always used. Reliably determining whether a file has been read really requires maintaining a hash of the part of the file that has been read. That means re-hashing the file every time additional data is read, which would be extremely expensive. The file input uses a much cheaper technique which sometimes gets things wrong. When in read mode and using file\_completion\_action delete, the problem could be solved by deleting sincedb entries as part of the file deletion process. If someone contributed a PR that implemented that I expect elastic would be happy to take a look at it. ---
**Author:** ![hsalim](https://avatars.discourse-cdn.com/v4/letter/h/c67d28/32.png) [@hsalim](https://discuss.elastic.co/u/hsalim)\ **Post date:** [May 20, 2021, 7:13pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/32 "2021-05-20T19:13:40Z")
Yes, i read up and now understand better role of sincedb. so my files do not change once written to directory for ingestion. file\_completion\_action delete is active (as default). i am now testing with sincedb\_clean\_after 45 seconds. lets see... any other ideas and suggestions are welcome. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 17, 2021, 7:13pm UTC](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058/33 "2021-06-17T19:13:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/logstash-input-file-plugin-and-tcp-plugins/273058.md?page=1)