Implement the prune filter using painless script in ingest pipeline

Elastic Stack Elasticsearch
1

Hi,

I see a remove processor for ingest pipelines in Elasticsearch. But I don't see any counterparts for logstash's prune filter. How can I implement one using painless ?
Also I am unable to create deep copies of ctx.keySet(),
so I can iterate over this new set and then ctx.remove() unnecessary fields from my document.

Any pointers would be welcome.

Regards,
Ahmad

2

I got it. This works. But it seems dirty and performance intensive :expressionless:

   {
 "script": {
   "lang": "painless",
   "source": """
    ctx.to_keep = new ArrayList();
    ctx.to_drop = new ArrayList();
    ctx.to_keep = ['field1', 'field2'];
    for (key in ctx.keySet()) {
      if (! ctx.to_keep.contains(key)) {
        ctx.to_drop.add(key);
      }
    }
    for (key in ctx.to_drop) {
        ctx.remove(key);
    }
   """
 }

I am not sure how efficient this is. Can someone from elastic please take a look here ?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.