Delete/pseudonomize entries selectively?

(Niels K.) #1

I use elasticsearch with fluentd and kibana as a logserver. Now I have the problem that I need to delete or pseudonomize logs after a certain time because of data
I.e.: all logs sent from dovecot should be kept only for 7 days and all apache-logs should get their IPs pseudonomized after 24 hours because of data protection laws. General system logs can be kept indefinitely. How can I realize something like that with elasticsearch?

(Christian Dahlqvist) #2

Use time-based indices and place data with different retention requirements in different indices. In order to change data after a certain time you will need to reindex, e.g. by using the update by query API, potentially together with an ingest node pipeline to handle the transformation.

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.