Unwanted fields in indice

Elastic Stack Elasticsearch
1

One of my indice has some field which are nowhere in my logstash config.
yes that fields are from other config file. but both config files are separate pipeline. both has different document_id

how come they appear in this indice?
what did I do wrong? anyone has seen such problem before?

2

Have you specified the pipelines in pipelines.yml? If not, having config in different files does not make them individual pipelines as Logstash concatenates flies into a single pipeline by default.

3

yes I have different piplelines (total 21 running at this time) happend in this one index only. don't know how. trying to figure out how to prevent this happening again.
and second how to remove them?

I don't want to recreate index pattern as I have few dashboard already design using that.
I can remove whole indice and recreate it but as situation has given me chance to try to remove individual field if possible.

4

Can you share your config and sample events that have been incorrectly processed?

5

pipelines

- pipeline.id: tms_change_control (data from this pipeline got in to second)
  path.config: "/etc/logstash/conf.d/tms_change_control.conf"
  pipeline.workers: 1

- pipeline.id: orion_network_inventory  (extra field from top one got here)
  path.config: "/etc/logstash/conf.d/orion_network_inventory.conf"
  pipeline.workers: 1

they both reading from completed different database
complete different columns

but I have "assignee, assignelocation, chage_time, change_date,' pretty much all field from that got in here.

don't understand it.
both config has different indexname, and different document_id in output sections.

possible that someone ran this config manually and pointted to wrong index?

6

I was just want to test it and hence I deleted index and repulled the data manually and all unwanted fields are gone.

not what I wanted to test (removing individual field) but for now I can put this to rest. Thanks Christian

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.