Filtering- Delete a field to save storage space

Elastic Stack Logstash
1

Hello ,
I am a beginner on the ELK solution,
I would like to know if by deleting fields and their contents from my logs I can gain storage space.
If so, how to do this task?
This is what I tried to do:
'
if [event][module] == "panw"
{
geoip {
source => "[client][nat][ip]"
target => "source_geoip"
}
geoip {
source => "[destination][nat][ip]"
target => "destination_geoip"
}

drop {

remove_field => [ "[destination_geoip][longitude]", "[destination_geoip][location]","[log_event][labels]" ,

"[destination_geoip][country_code3]", "[latitude]", "[agent][ephemeral_id]","[source_geoip][longitude]",

"[source_geoip][location]","[source_geoip][country_code3]", "[observer][serial_number]", "[network][community_id]",

"[message]", "[sort]" ]

remove_tag => [ "[destination_geoip][longitude]", "[destination_geoip][location]", "[log_event][labels]" ,

"[destination_geoip][country_code3]" "[latitude]", "[agent][ephemeral_id]", "[source_geoip][longitude]",

"[source_geoip][location]","[source_geoip][country_code3]", "[observer][serial_number]", "[network][community_id]",

"[message]", "[sort]" ]

}

}
'
The goal is to reduce storage space on Elasticsearch

Thank you for your attention

2

A drop filter will delete the event. If you just want to remove fields then most folk would use a mutate filter. remove_tag is used to remove strings from the tags array, not to remove fields.

In general, if you remove fields from your events the events will be smaller and will use less space in elasticsearch. The space used by elasticsearch will depend on many factors though. Number of replicas and shards, for example. And index_options can have a large impact.

3

I will try
thanks you

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.