Namespaces and index clean-up

[ethrbunny]

I recently did some experimentation with getting data from 'auditd' into logstash. Along the way it created lots and lots of unwanted entries in the elastic index.

EG: These are items like '#033.31;1mbuilds#033.0;m.keyword'

-> how long will these entries persist?

Similarly there are entries in the index that really really need more explanation. I'd like to put these all in a namespace (of sorts) so that my future self will have some clue where they came from.

EG. 'a0', 'ppid'.

In an ideal world I'd have these appear as 'auditd.a0', etc. I suppose I can 'mutate' them all but Im hoping there is a more direct route.

[magnusbaeck (Magnus Bäck)]

-> how long will these entries persist?

Until someone or somebody deletes them. There's no default purging of old/unused data.

In an ideal world I'd have these appear as 'auditd.a0', etc. I suppose I can 'mutate' them all but Im hoping there is a more direct route.

That depends on how the fields are created in the first place.

[ethrbunny]

Until someone or somebody deletes them.

Do I need to delete all the saved indices to 'clean up' ?

That depends on how the fields are created in the first place.

I just map kv pairs from auditd

[magnusbaeck (Magnus Bäck)]

Do I need to delete all the saved indices to 'clean up' ?

What do you mean? If you want to clean up everything then you obviously need to delete all indexes, but you can choose to be more selective if you want to keep stuff around.

I just map kv pairs from auditd

Setting the kv filter's target option doesn't help?

[ethrbunny]

Ah - there it is.

" 'prefix': A string to prepend to all of the extracted keys.

For example, to prepend arg_ to all keys:

filter { kv { prefix => "arg_" } }"