Remove pipeline using field wildcard

(Manikandan) #1

I want to remove a field starting with ElapsedTimeAt* in my remove processor pipeline. Is it possible?

(David Pilato) #2

Reading the code, I don't think it is possible to use wildcards.

1 Like
(Manikandan) #3

Thanks for the quick reply. This is what i am trying to achieve.

I have a incoming document and in Logstash i add a field for which the name is dynamic (filed name will be ElapsedTimeAt_timestamp where timestamp will be substituted with @timestamp value). This field i need to make it available in one index and should be removed before indexing into other index. Struggling to implement this. Is there any way to do this ??

(David Pilato) #4

May using an ingest script processor?

1 Like
(Manikandan) #5

Can i have something like
ctx.ElapsedTimeAt_%{timestamp} ? Will it compute the column name dynamically ?

If you have any example that would help me great ..

(David Pilato) #6

Yes you can have something similar to this if there is a field which contains the value you are looking for. See https://www.elastic.co/guide/en/elasticsearch/reference/6.6/accessing-data-in-pipelines.html#accessing-template-fields

For Ingest Script Processor, read https://www.elastic.co/guide/en/elasticsearch/reference/6.6/script-processor.html. I'm not a painless expert so I can't really tell how to read all the fields from your documents and compare to a regex/wildcard and remove it or not.

1 Like
(Manikandan) #7

Same here David.... very beginner in painless ... But will figure it out and update you if i am successful... BTW thanks much for the quick reply.

(system) closed #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.