Remove pipeline using field wildcard

manikandanid · February 11, 2019, 12:31pm

I want to remove a field starting with ElapsedTimeAt* in my remove processor pipeline. Is it possible?

dadoonet · February 11, 2019, 12:43pm

Reading the code, I don't think it is possible to use wildcards.

elastic/elasticsearch/blob/master/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/RemoveProcessor.java#L56-L65


if (ignoreMissing) {
    fields.forEach(field -> {
        String path = document.renderTemplate(field);
        if (document.hasField(path)) {
            document.removeField(path);
        }
    });
} else {
    fields.forEach(document::removeField);
}

manikandanid · February 11, 2019, 12:51pm

Thanks for the quick reply. This is what i am trying to achieve.

I have a incoming document and in Logstash i add a field for which the name is dynamic (filed name will be ElapsedTimeAt_timestamp where timestamp will be substituted with @timestamp value). This field i need to make it available in one index and should be removed before indexing into other index. Struggling to implement this. Is there any way to do this ??

dadoonet · February 11, 2019, 1:11pm

May using an ingest script processor?

manikandanid · February 11, 2019, 1:26pm

Can i have something like
ctx.ElapsedTimeAt_%{timestamp} ? Will it compute the column name dynamically ?

If you have any example that would help me great ..

dadoonet · February 11, 2019, 1:39pm

Yes you can have something similar to this if there is a field which contains the value you are looking for. See https://www.elastic.co/guide/en/elasticsearch/reference/6.6/accessing-data-in-pipelines.html#accessing-template-fields

For Ingest Script Processor, read https://www.elastic.co/guide/en/elasticsearch/reference/6.6/script-processor.html. I'm not a painless expert so I can't really tell how to read all the fields from your documents and compare to a regex/wildcard and remove it or not.

manikandanid · February 11, 2019, 3:16pm

Same here David.... very beginner in painless ... But will figure it out and update you if i am successful... BTW thanks much for the quick reply.

system · March 11, 2019, 3:16pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.