Data Pipeline Design Considerations

So I have a confluent kafka with multiple topics per network divided as follows

NET1: Data



NET2 : Data



I have 2 logstash servers reading from those topics on Kafka

1 reading the topics from NET1 going to Elastic
the second is reading only the topic data from NET2 also going to the same Elastic
If i have created 2 spaces to isolate the data once I called it NET1 and the second one is NET2

NET1:SYSLG, WIN ---> Logstash 1 -------------------- NET1 SPACE
KFK- -------------------------------- ------- | --------> ELASTIC |
NET2: SYSLG, WIN --- > Logstash2 --------------------- NET2 SPACE

what is best approach or procedures to write data for logstash 1 to SPACE 1 index called net1-syslog
How do I go about configuring that on Elasticsearch to map those incoming events from each logstash to their corresponding space and indices ? Any insight or help would be appreciated

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.