Logstash file input read order

Elastic Stack Logstash
1

Hi to All,

I have one configuration file with multiple file inputs. Since deploy we have parsed hundreds of files, I now want to re parse everything but I want to re parse them ordered by creation time.

Could you tell me if and how this can be accomplished?

Kind Regards,
Ioannis Ntantis

2

There's nothing built into Logstash for this. My best suggestion would be to rename the files according to the creation time and add them, in order, to a single file input's path parameter. Note that you don't have to touch the original files since you can use links (either soft or hard).

3

So if I get it right, when file plugin point to a dir (containing subdir), logstash reads them (assume -w 1) by alphabetical order?

So:

+dir
|_+c
| |_ab
| |_aa
|_+a
  |_z

is read: dir/a/z then dir/c/aa then dir/c/ab by logstash?

4

Ok, in fact, read order is the same than ls -U. Any idea how I can somehow make the directory entry the same as the alphabetical order?

5

The directory traversal order is not configurable in Logstash. I can't think of a reasonable workaround except what I said earlier or having an external program that concatenates the files in the right order into a single file that Logstash is configured to read.

6

Ok, thanks. I'm currently trying a workaround using rsync to copy files from repo to entrypoint

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.