ES version 7.3.2,
Java1.8,
spark 2.3.0
Problem Description:
hey all, I have a problem, the project needs to use spark streaming to consume Kafka data in real time and query the consumed data for ES. For data that is not queried in ES, it needs to be pushed to another topic of Kafka. Because the amount of data consumed by spark streaming from kafka is relatively large, ES is used. MultiSearchRequest to encapsulate a series of query requests, the problem is that after executing msearch, there is no way to know which query request parameter missed the data in the MultiSearchResponse returned from the ES.
The simple description is that 1000 request queries are now packaged as batch queries. Among them, 100 query miss data, you need to know the request parameters of these 100 queries based on the return value, you need to know the original Kafka data according to the request parameters, and then Push raw data to another topic.
Anyone who knows the solution or someone who has ideas is in trouble. Thank you very much!
Please let me know if you need more information from my side. Any help will be appreciated.
Regards,
chenk