Thousands of filebeat ingest pipelines / cleanup

[idl0r (Christian)]

Hey,

I’m not very familiar yet with ELK/SIEM and I‘m also not sure about the proper category here.

So we have like thousands of filebeat ingest pipelines. I assume they’re auto created during updates. Is it safe to delete old or even all ones and let it recreate needed ones? Is there something to clean those up, any hints or tips?

[idl0r (Christian)]

Or would that usually be done by the ILM? Because those policies doesn’t seem to work. They’re all stuck in the rollover / hot phase for years already and I don’t see what’s blocking / happening there

[leandrojmp (Leandro Pereira)]

@idl0r

So we have like thousands of filebeat ingest pipelines. I assume they’re auto created during updates.

Those ingest pipelines are created when you run the filebeat setup with their modules enabled, at some point in time this was done your system.

[idl0r (Christian)]

Is it safe to just delete anything that seems old? Is it possible to automatically clean those, like the ILM does or something more elegant?

[leandrojmp (Leandro Pereira)]

@idl0r

Is it safe to just delete anything that seems old?

If you are not using any of those modules with Filebeat you can remove them.

@idl0r

Is it possible to automatically clean those, like the ILM does or something more elegant?

No, those ingest pipelines are normally only installed when required, if the user does not need them anymore, it is up to them to remove.