Wondering that can we lookup on a list to remove the field not on the list

Elastic Stack Logstash
1

Hi folks,

We have an index with so many data form different application.
we are currently looking for a way like schema restriction and not allow people to generate new field.
Is there any way to use logstash to lookup on a list or sql data lookup, then remove all the other field not on the list.
We had look at prune plugin, but we have thousands field out there.
We cannot use logstash config to deal with thousands of field name

Let me know if you need more explain
Appreciate it.

2

If you are writing data to elasticsearch you could turn off dynamic mapping. If you need to do it in logstash you would have to write it in a ruby filter.

3

Dynamic mapping is good. but not in this case.
If we enable dynamic mapping, then it will create tons of field.
But it you disable it, it will drop the record even there only have one field you don't want to
Currently we already have more than 7000 field.
We are try to find a good way to control it.
But we dont want to manage it manually.
To manage the template with thousands of fields is terrible

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.