Logstash file input read order

Hi to All,

I have one configuration file with multiple file inputs. Since deploy we have parsed hundreds of files, I now want to re parse everything but I want to re parse them ordered by creation time.

Could you tell me if and how this can be accomplished?

Kind Regards,
Ioannis Ntantis

There's nothing built into Logstash for this. My best suggestion would be to rename the files according to the creation time and add them, in order, to a single file input's path parameter. Note that you don't have to touch the original files since you can use links (either soft or hard).

So if I get it right, when file plugin point to a dir (containing subdir), logstash reads them (assume -w 1) by alphabetical order?


| |_ab
| |_aa

is read: dir/a/z then dir/c/aa then dir/c/ab by logstash?

Ok, in fact, read order is the same than ls -U. Any idea how I can somehow make the directory entry the same as the alphabetical order?

The directory traversal order is not configurable in Logstash. I can't think of a reasonable workaround except what I said earlier or having an external program that concatenates the files in the right order into a single file that Logstash is configured to read.

Ok, thanks. I'm currently trying a workaround using rsync to copy files from repo to entrypoint

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.