Multiple Logfiles from diffrent ClusterNodes to one merged Logfile

Elastic Stack Logstash
#1

Hi,

i hope someone can help me.

I want merge/aggregate logfiles from diffrent Clusternodes to one Logfile on logfile Storage.

Currently I use log / slog, but this is no longer being developed. I evaluate the index and the logfiles already Elasticsearch and Logstash use, I am now seeking to aggregate the files with logstash a possibility.
It is important that the log files are put together in the correct sequence.

So far I have found no such Logstash plugin or an appropriate documentation.

Thanks to all the help or even try.

Martin

#2

It is important that the log files are put together in the correct sequence.

What does this mean, specifically? That the events in a particular input file end up in the same order in the output file, or that all events from an input file are laid out consecutively in the output file? In other words, given these input files:

fileA:

A1
A2
A3

fileB:

B1
B2
B3

Would

B1
A1
A2
B2
A3
B3

be okay in the output file or do you need this:

A1
A2
A3
B1
B2
B3

Logstash has many input plugins that you can use to collect data from your machines, and the file output plugin can write some or all of those events to files. It's not clear why that wouldn't be sufficient in your case.

#3

HI Magnus,

i need

A1
B1
C1
D1
F1
A2
B2
D2

Decisive for the arrangement of the timestamp is in file. If I have in 5 seconds 10 entries in the log file A and only 1 in the log file B then should all look like this later.

00:01:01A1
00:01:02A1
00:01:03A1
00:01:04B1
00:01:05A1
00:01:06A1

I hope I have made myself clear

#4

Oh, okay. There are no stock plugins in Logstash to do this.

#5