Drop Events while using elastic kubernetes integration to collect logs

[Jobin_James]

Hello,

Elastic search Version: 8.12.2
ECK Operator Version: 2.11.0

I am using elastic Kubernetes integration for pushing logs from the k8s cluster using an elastic agent. I would like to know how can i drop the unwanted fields. I tried the processor, but it didn't work.

Thanks

[stephenb (Stephen Brown)]

Hi @Jobin_James

@Jobin_James

I tried the processor, but it didn't work.

You'll have to show us what you tried and what did not work and what you're trying to accomplish.

Standalone agent of Fleet Managed?

Yes, you can drop fields but they need to be available and some of the fields like the agent field are appended after the processor run.

So show us tell us what you're trying to accomplish and what you tried. Perhaps we can help

[Jobin_James]

Hello @stephenb,
Thanks for the answer.

I am using the fleet managed elastic agent to ingest logs from the k8s cluster to the Elasticsearch.

I have attached a screenshot of the indexed document and I would like to remove most of the fields related to cloud and ecs as this is irrelevant to my developers. Can I do this using the processor or should I use the ingest pipeline?

Thanks

[stephenb (Stephen Brown)]

BTW to me it is not clear if you want to Drop Events... or Remove Fields...

Ok... can you share what you tried?

Can you share your config?

For most the cloud fields you can turn off the add metadata processor.
Not sure Which ECS fields you are referring to?

Some of the fields (agent and host I believe) are added AFTER the integration processing but you can certainly drop them in an ingest pipeline.... in fact you can do that with any / all the fields and often users find that to be an easy way to centralized removing fields.

There is documentation on how to do that here and here but instead of adding fields you can remove them and

In short Remove Processor

In Kibana Dev Tools

PUT /_ingest/pipeline/logs-kubernetes.container_logs@custom
{
  "processors": [
    {
      "remove": {
      "field": ["user_agent", "URL"] <<< Add Fields Here 
      }
    }
  ]
}

If you want actually to Drop Whole Events...

You can do that with the processors at the Agent Level (again you would need to show us what you tried otherwise we are just guessing)

Or use Drop Event in the ingest Pipeline