Log insertion with Filebeat

Hi ,

I use Filebeat to recover my scheduler's log .

It's functional but the elasticsearch insertion is not optimal ... it inserts one hit by line on my log.

Ideally , i would like to insert the entire log i one hit..

Do you have any idea to permit this?

Thanks

Regards

Hey @JulienN,

in principle you can configure multiline rules to match the whole file as a single event.

You may face some problems depending on how your scheduler writes its logs. For example filebeat needs that all lines are terminated by a new line, this is important to get the last line of a log file. Also, the file needs to be properly rotated, filebeat is not going to start reading from the beginning a file that it has already read.

Could you give more details on how these log files are written?

Hi Jaime ,

Thanks for your reply !!

Yes the multiline could be the solution , but my logs are a bit different by machine.

For example , a log can begin by timestamp YYYY-MM-DD , or directly by functional details of the execution (a character) .. so it's difficult to determine a single pattern.. have you any idea ??

Thanks

Regards

Julien

I found the good pattern with multiple tests on https://play.golang.org/

Now with this pattern , it would be good :

multiline.pattern: '^=[A-Z]+|^$'
multiline.negate: false

But i've a problem , it's exactly the same thing , one hit on Elastic by line on my logs...

Any ideas ??

Regards

Do you mean that with this multiline configuration, the result is the same? Could you share the whole configuration you are using now?