Aggregate data using filebeat

[szabgab (Gabor Szabo)]

I have a log file that looks like this:

TIMESTAMP FIELD VALUE

e.g.

    100 load 1
    100 mem 23
    100 free 7
    103 load 2
    103 mem 17
    103 free 9
     ...

I would like to aggregate the lines with the same timestamp to be a single message. Something along the lines of this:

    {
       "my.timestamp"; 100
       "my.load" : 1,
       "my.mem": 23,
       "my.free": 7
    }

In Logstash it seems I can do this, though I am not yet sure how to separate data arriving from different filebeat agents. I wonder if this would be possible in filebeat already?

I tried to use the multiline feature, but I don't now who to tell it "a message is done when the timestamp changes.

I tried to use the "script" option in filebeat. I can convert the individual log strings to key-value pairs, but I don't know how to aggregate them.

Any suggestions?

[ChrsMark (Chris Mark)]

Hi!

I'm not sure there is something like this in Filebeat. Filebeat aims to be a lightweight agent and will avoid to perform processing on the edge. So if it's doable in Logstash you should stick with that.

[deepybee (Dan Palmer-Bancel)]

You could likely achieve this with the embedded JavaScript script processor but as above it’s somewhat contrary to the design of Beats to do this sort of processing at edge - I personally am not a fan of using Beats in this fashion. It’s overly cumbersome to setup and maintain and will probably reduce your ingest rate significantly.

You are far better using multiline to collapse the lines into a single event and handing the string parsing in an ingest pipeline in Elasticsearch.

[szabgab (Gabor Szabo)]

I thought about JavaScript, but as I understand it runs my JS code on every line and I could not figure out how to pass the collected data between lines.

I also tried multiline, but could not figure out how to recognize the beginning and the end of the section.