# Aggregate data using filebeat

**URL:** <https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 21, 2021, 11:41am UTC](https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907 "2021-03-21T11:41:11Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![szabgab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szabgab/32/83232_2.png) [@szabgab](https://discuss.elastic.co/u/szabgab)\
**Post date:** [March 21, 2021, 11:41am UTC](https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907/1 "2021-03-21T11:41:11Z")

</div>

I have a log file that looks like this:

`TIMESTAMP FIELD VALUE`

e.g.

```auto
    100 load 1
    100 mem 23
    100 free 7
    103 load 2
    103 mem 17
    103 free 9
     ...

```

I would like to aggregate the lines with the same timestamp to be a single message. Something along the lines of this:

```auto
    {
       "my.timestamp"; 100
       "my.load" : 1,
       "my.mem": 23,
       "my.free": 7
    }

```

In Logstash it seems I can do this, though I am not yet sure how to separate data arriving from different filebeat agents. I wonder if this would be possible in filebeat already?

I tried to use the multiline feature, but I don't now who to tell it "a message is done when the timestamp changes.

I tried to use the "script" option in filebeat. I can convert the individual log strings to key-value pairs, but I don't know how to aggregate them.

Any suggestions?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [March 22, 2021, 9:49am UTC](https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907/2 "2021-03-22T09:49:24Z")

</div>

Hi!

I'm not sure there is something like this in Filebeat. Filebeat aims to be a lightweight agent and will avoid to perform processing on the edge. So if it's doable in Logstash you should stick with that.

---

<div class="post-metadata">

**Author:** ![deepybee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepybee/32/20381_2.png) [@deepybee](https://discuss.elastic.co/u/deepybee)\
**Post date:** [March 31, 2021, 6:36am UTC](https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907/3 "2021-03-31T06:36:55Z")

</div>

You could likely achieve this with the embedded JavaScript `script` processor but as above it’s somewhat contrary to the design of Beats to do this sort of processing at edge - I personally am not a fan of using Beats in this fashion. It’s overly cumbersome to setup and maintain and will probably reduce your ingest rate significantly.

You are far better using `multiline` to collapse the lines into a single event and handing the string parsing in an ingest pipeline in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![szabgab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/szabgab/32/83232_2.png) [@szabgab](https://discuss.elastic.co/u/szabgab)\
**Post date:** [March 31, 2021, 6:53am UTC](https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907/4 "2021-03-31T06:53:09Z")

</div>

I thought about JavaScript, but as I understand it runs my JS code on every line and I could not figure out how to pass the collected data between lines.

I also tried `multiline`, but could not figure out how to recognize the beginning and the end of the section.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 28, 2021, 8:53am UTC](https://discuss.elastic.co/t/aggregate-data-using-filebeat/267907/5 "2021-04-28T08:53:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.