Good day, all
I have the following environments in place:
A functional Production environment:
A. Host1 with filebeat polling a file sending to:
B. A 3-node Kafka cluster on Topic1. (version: kafka_2.13-3.2.0)
C. 2 x Logstash hosts read off Topic1 (version: logstash-8.4.3-1.x86_64)
D. Then they write to a 3-node Elasticsearch search cluster (version: elasticsearch-8.6.2-1.x86_64)
I also have a Stage environment pretty much the same:
- 3-node Kafka cluster (kafka_2.13-3.2.0)
- Logstash (logstash-8.7.1-1.x86_64) host reading off the Stage Kafka Cluster
- And writing to a Stage Elastic search cluster (version: elasticsearch-8.7.1-1.x86_64)
Stage has a newer Logstash version vs Production, as when I upgraded stage, one of the Logstash pipelines configs did not want to start up and I had to roll back to the original version; this is at the heart of the issue.
I would like the data entering the Prod stack to also end up in the Stage stack. Specifically, I want the raw data from Filebeat to be written to BOTH stage and prod Kafka. The Stage Logstash instance needs to be used to play around with new logstash pipeline configs, while not interupting the flow of the same data up the Prod stack but still using copies of the same data for testing the Logstash config with a newer version of Logstash.
I had hoped that the filebeat output could include the FQDN's/IPs for both Prod and Stage Kafka clusters, but that's not possible. My ELK knowledge is still very much at the novice level, so any suggestions would be most welcome!
Regards,
Kevin Pillay