Hello,
I have a requirement that "Logstash need to read the data from two different Kafka topics, lookup the read data using filter plugin and finally send it to Elasticsearch".
Two Kafka topics are presented, one consists of Master data which will not change frequently(Batch Data) and other one is Transactional data which will change frequently(Realtime Data).
I can able to read from Kafka topics, but I have no idea to lookup the data using Logstash.
For example:
The kafka topics will have data like below as:
Master Data:
| EmpNo | EmpName | PostedUnit | DC |
|---|---|---|---|
| 00001 | Ram | xxx | yyy |
| 00002 | Sara | aaa | bbb |
| 00003 | Tina | iii | jjj |
Transactional Data:
| EmpNo | SwipeIN/OUT | SwipeDateTime |
|---|---|---|
| 00001 | IN | 11/10/2020 09:02 |
| 00002 | IN | 11/10/2020 08:00 |
| 00002 | OUT | 11/10/2020 18:15 |
| 00001 | OUT | 11/10/2020 19:20 |
| 00003 | IN | 11/10/2020 22:20 |
The data need to send to elasticsearch as below as:
Output Data which stored in Elasticsearch:
| EmpNo | EmpName | PostedUnit | DC | EmpNo | SwipeIN/OUT | SwipeDateTime |
|---|---|---|---|---|---|---|
| 00001 | Ram | xxx | yyy | 00001 | IN | 11/10/2020 09:02 |
| 00002 | Sara | aaa | bbb | 00002 | IN | 11/10/2020 08:00 |
| 00002 | Sara | aaa | bbb | 00002 | OUT | 11/10/2020 18:15 |
| 00001 | Ram | xxx | yyy | 00001 | OUT | 11/10/2020 19:20 |
| 00003 | Tina | iii | jjj | 00003 | IN | 11/10/2020 22:20 |
If it is possible by Logstash, please guide me how to lookup the Master Data with Transactional data.
Anyone kindly help me out to solve this issue.
Regards,
Sangeetha