How to reduce primary storage size

Elastic Stack Beats
1

hi all,
generally, when I use Logstash I like to remove the message field if the filter section doesn't contain any error. I do it to reduce the size of every document.
since this time I will not use Logstash, is there a way to do it? Should I use an ingest pipeline?
Is there any extra parameter or compression that I can configure?

2

What tool are you using to ingest the data?

3

sorry, you are right :slight_smile:
I'm using Filebeat.

4

I switched your post to #elastic-stack:beats

I'd use this

So the event is not even sent to elasticsearch. Which will also save your network bandwidth. :slight_smile:

5

thank you.
just one doubt: I cannot drop the message field using the Filebeat feature, right? I mean, if I would do it I will simply lose the document's content?

6

Yes, if you drop message you lose the actual contents of the message. If you want to reduce the size of the event you could drop metadata fields which are not important to you.

7

ok, thanks.
To modify already existing docs into indices, can I use an ingest pipeline?

8

Yes, you can reindex your existing docs from indices into another index like this:

POST _reindex
{
  "source": {
    "index": "source"
  },
  "dest": {
    "index": "dest",
    "pipeline": "your_ingest_pipeline"
  }
}

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-routing

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.