Hi guys. I am facing a situation where every of my log files are single exceptions, and I do not wish to store them line by line. Instead, I would prefer to store them file by file, which means, I wish to combine the entire log file as one event.
I've check the multiline codec, but it doesn't seem to work, I wonder if there are any ways to do it.
Thanks in advance.
1 Like
Multiline is going to be the best best.
What is your problem with it, what does your config look like?
I think I have a similar case. I process previously created log bundles and correlate data points. Many times, one file functions as a single data point and thus one event. I may run splits on the data once I have it in the pipeline but I need the entire file in the pipeline to provide correlation points.
Because there is no standard character to represent the end of a file using pattern match in multiline and file acts like tail -0f causing multiline to continue to wait for the end, I cannot treat one file as a single event without massaging the data before entering the logstash pipeline.
What I will do is add ENDOFFILE and then pattern match against it to indicate end of stream for multiline.
I too, would appreciate any information about whole file intake.
Thanks
2 Likes
+1
We need to be able to treat things like upgrade logs. Each upgrade event produces a log file. We need to be able to go over each of these upgrade events and read the raw log output in the message to troubleshoot some deployments.
If anyone has a solution for ingesting files as events and maintaining original formatting in the message, please let me know.
Thanks!
1 Like
Hi,
Is there any progress about this issue?