As a part of ELK migration we are trying to setup our new cluster in a shared AKS cluster. Issue is, we have the flow as Filebeat -> Ingress -> Logstash entry -> Logstash indexing -> Kafka -> Elastic. Currently as a part of migration we will have another source and both are finally ending up in the same elastic. What is the right way to differentiate from what source we got the data from in Elastic. Can someone help me on this query please?
More details below:
We have beats as a source sending logs to logstash for data structure and we have Kafka in between Logstash Elasticsearch cloud for message persistence. We don't have load balancing mechanism here . We need to migrate the exiting set up of logstash / Kafka to a new Kubenetes cluster . if we do the parallel ingestion of logs to current logstash / new logstash , then there will be a duplicate of messages at Elasticsearch .. current plan we have is to enable parallel data ingestion in file beat . Create duplicate index through new logstash and perform the validation ..if the messages are ingested properly then Remove parallel data ingestion by reconfiguring the output to point to the new Logstash DNS address.(Cease the data flow to the current logstash, rename the new indices to match the original indices and resume the flow from the current logstash).
Also need to know whether its a good approach because we will end up having lot of duplicated data if we go with this approach .