Input threads with wild cards

Elastic Stack Logstash
1

Hi,

As I understand, each input section would be run with separate thread.
But if I use wild cards in an input section, does this create multiple threads or still a single thread would be created?

I ask this because, if a wildcard results in huge number of files, then how Logstash would handle this?

Thanks.

2

I believe it's a single thread per file input. I haven't looked at the code, but it's my experience that each file will be read until EOF before Logstash continues with the next file.

3

@magnusbaeck
Okay. So, the same thread would read all the files matching the wildcard one after the another.
But my concern is we have many log files in multiple folders. But all log files are similar. So, filtering is same.
So, I am trying to load balance between multiple threads rather one thread take all the load.

What do you recommend here?

4

Magnus is spot on, it's a single thread for the input.

Look at setting threads in the input and using workers for Logstash.

5

Keep in mind that your performance is (also) bound by the filter and output throughput so increasing parallelism on the input side won't necessarily help. Bulk-reading messages from a big file is probably quite fast.

6

It's pretty bloody fast, yes :stuck_out_tongue: