# Multiple file inputs cause re-reading of file
**URL:** https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276
**Category:** Logstash
**Tags:** docker
**Created:** [August 21, 2024, 12:14pm UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276 "2024-08-21T12:14:27Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@henning\_l](https://discuss.elastic.co/u/henning_l)
#### Post date: [August 21, 2024, 12:14pm UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/1 "2024-08-21T12:14:27Z")
I am running an ELK stack and tailoring the Logstash configurations to elicit data from several different logs. Because the logs are formatted differently I need to create multiple file inputs. The following is the current configuration:
```auto
input {
file {
mode => "read"
path => "/usr/share/logstash/ingest_data/**/*"
file_completed_action => "log"
file_completed_log_path => "/usr/share/logstash/finalised_data/logstash_completed.log"
exclude => ["/usr/share/logstash/ingest_data/app/zsh/*"]
}
file {
mode => "read"
path => "/usr/share/logstash/ingest_data/app/zsh/*"
file_completed_action => "log"
file_completed_log_path => "/usr/share/logstash/finalised_data/logstash_completed.log"
codec => multiline {
pattern => "\\$"
what => "next"
}
}
}
```
However, my current configurations result in logstash reading files in the directory `/usr/share/logstash/ingest_data/app/zsh/` twice, even though I explicitly configured the first file input to exclude files in that directory (I checked the file `/usr/share/logstash/finalised_data/logstash_completed.log` and two entries are created when I add a single file)
I cannot find an explanation for this other than unintentional behaviour when having multiple file inputs.
To debug the issue tried to use only one of the file inputs at a time. When using the first input a zsh log is not read, and when using the second input a zsh log is read just once as intended.
I am running a docker compose stack as described [here](https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose) with the version 8.7.1.
---
### Author:  [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)
#### Post date: [August 22, 2024, 5:16am UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/2 "2024-08-22T05:16:00Z")
Hello Henning,
When having multiple file inputs, you need to configure `sincedb_path` explicitly.
> Path of the sincedb database file (keeps track of the current position of monitored log files) that will be written to disk. The default will write sincedb files to `/plugins/inputs/file` NOTE: it must be a file path and not a directory path
If not set, the values of both file inputs will overwrite each other. Maybe this is the reason for your issues.
Best regards
Wolfram
---
### Author:  [@henning\_l](https://discuss.elastic.co/u/henning_l)
#### Post date: [August 22, 2024, 5:58am UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/3 "2024-08-22T05:58:29Z")
Hi Wolfram
Thanks for your answer. I have tried to follow your suggestion and by setting the sincedb\_path explicitly (the field `file_input` is just to make it easier for me to debug):
```auto
input {
file {
mode => "read"
path => "/usr/share/logstash/ingest_data/**/*"
file_completed_action => "log"
file_completed_log_path => "/usr/share/logstash/finalised_data/logstash_completed.log"
exclude => ["/usr/share/logstash/ingest_data/app/zsh/*"]
sincedb_path => "/usr/share/logstash/file_all"
add_field => { "file_input" => "all" }
}
file {
mode => "read"
path => "/usr/share/logstash/ingest_data/app/zsh/*"
file_completed_action => "log"
file_completed_log_path => "/usr/share/logstash/finalised_data/logstash_completed.log"
codec => multiline {
pattern => "\\$"
what => "next"
}
sincedb_path => "/usr/share/logstash/file_zsh"
add_field => { "file_input" => "zsh" }
}
}
```
However, this renders the same output. Any suggestions?
---
### Author:  [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)
#### Post date: [August 28, 2024, 4:49am UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/4 "2024-08-28T04:49:11Z")
Hello Henning,
I am not sure, but I think your `exclude` may be wrong. According to the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-exclude):
> Exclusions (matched against the filename, not full path). Filename patterns are valid here, too. For example, if you have
>
> path =\> "/var/log/\*"
>
> In Tail mode, you might want to exclude gzipped files:
>
> exclude =\> "\*.gz"
This could explain why it doesn't exclude the `zsh` logs from the file input.
How many directories do you have under `/usr/share/logstash/ingest_data`? Would it be possible to list them separately under `path`, e.g.:
```auto
file {
mode => "read"
path => [
"/usr/share/logstash/ingest_data/dir1/**/*",
"/usr/share/logstash/ingest_data/dir2/**/*",
"/usr/share/logstash/ingest_data/dir3/**/*",
"/usr/share/logstash/ingest_data/app/not_zsh/**/*"
]
file_completed_action => "log"
file_completed_log_path => "/usr/share/logstash/finalised_data/logstash_completed.log"
exclude => ["/usr/share/logstash/ingest_data/app/zsh/*"]
}
```
Best regards
Wolfram
---
### Author:  [@henning\_l](https://discuss.elastic.co/u/henning_l)
#### Post date: [August 28, 2024, 1:44pm UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/5 "2024-08-28T13:44:14Z")
Hi Wolfram
After doing some additional testing, I have realised, that the issue is not with multiple filters - my bad. When only using the initial file input, logs are still processed in the pipeline even though they are not supposed to.
With that out of the way (and maybe I should create a new topic, since the title does not describe the true issue), you suggestion of splitting the path up in several paths is doable, but I am planning my project to be widely extendable, i.e. I want to have many directories/subdirectories. Thus, it is not a suitable solution for me.
I my understanding of the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-exclude), it is possible to exclude all files in a subdirectory, though its parents directory is part of the `path` option.
I have tried to strip as much of my Logstash configuration, which reads files located in `/usr/share/logstash/ingest_data/app/zsh/`:
```auto
input {
file {
mode => "read"
path => ["/usr/share/logstash/ingest_data/**/*"]
exclude => ["/usr/share/logstash/ingest_data/app/zsh/*"]
sincedb_path => "/usr/share/logstash/file_all"
}
}
filter { }
output {
elasticsearch {
index => "logstash-%{+YYYY.MM.dd}"
hosts=> "${ELASTIC_HOSTS}"
user=> "${ELASTIC_USER}"
password=> "${ELASTIC_PASSWORD}"
cacert=> "certs/ca/ca.crt"
}
stdout { codec => rubydebug }
}
```
Adding a log file to the aforementioned directory triggers it to run through the pipeline and print the following in the standard output
```auto
...
{
"message" => ": 1724069235:0;tail -n650 ~/.zsh_history >> material/.zsh_history",
"@version" => "1",
"host" => {
"name" => "a480e196997b"
},
"log" => {
"file" => {
"path" => "/usr/share/logstash/ingest_data/app/zsh/zsh_history"
}
},
"@timestamp" => 2024-08-28T13:27:26.167928882Z,
"event" => {
"original" => ": 1724069235:0;tail -n650 ~/.zsh_history >> material/.zsh_history"
}
}
...
```
I do not know whether this helps in finding a solution?
Cheers, Henning
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 28, 2024, 3:21pm UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/6 "2024-08-28T15:21:53Z")
> [@henning\_l](#):
>
> ```auto
> path => ["/usr/share/logstash/ingest_data/**/*"]
> exclude => ["/usr/share/logstash/ingest_data/app/zsh/*"]
>
> ```
I do not expect that to work. As @Wolfram_Haussig said, the exclude option takes a filename pattern. If you look at the [source code](https://github.com/logstash-plugins/logstash-input-file/blob/55a4a7099f05f29351672417036c1342850c7adc/lib/filewatch/discoverer.rb#L40) you will see that it is calling [basename](https://ruby-doc.org/core-2.6.3/File.html#method-c-basename) so all of the directory names are discarded before the comparison is made.
My understanding is that [fnmatch?](https://ruby-doc.org/core-2.6.3/File.html#method-c-fnmatch-3F) requires the whole pattern to match. Thus basename will reduce /usr/share/logstash/ingest\_data/app/zsh/foo.txt to foo.txt and foo,txt does not match /usr/share/logstash/ingest\_data/app/zsh/\*. In terms of the source, watched\_file.pathname would match, but watched\_file.pathname.basename does not.
---
### Author:  [@henning\_l](https://discuss.elastic.co/u/henning_l)
#### Post date: [August 29, 2024, 11:18am UTC](https://discuss.elastic.co/t/multiple-file-inputs-cause-re-reading-of-file/365276/7 "2024-08-29T11:18:21Z")
Finally, I understand how works. Thank you both @Wolfram_Haussig and @Badger. The capital 'I' in "In Tail mode..." in the [docs](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-exclude) mislead me to believe it described two different examples, but now I understand it is the same example with a path (`path => /var/log/*`) and a pattern (`exclude => "*.gz"`) for excluding files GunZip files.
I marked @Wolfram_Haussig's answer as a solution as i best described a solution to my question.