I am very new to Elastic and thinking about using it in a project.
Think about text files that contain millions of JSON lines each have different structure, nearly one thousand:
Sample Text file:
Line1 JSON_structure1
Lİne2 JSON_structure2
Line3 JSON_structure3
..
Line99999..9 JSON_structure999
I am thinking about pushing all of these lines to Kafka from source system and then use Logstash-Elastic Hadoop and store parsed data in HDFS. What is the best way of doing this ? Does logstash support these kind of parsing operation ?