I'm using Elasticsearch to process user logs and clickstream data.
Each such log contains several primitive attributes (version, environment, userID, etc.), but the heart of it is a JSON "data" object, which holds all the relevant information regarding the desired log.
There are various types of logs, and their "data" object differs greatly from one another.
I'm looking for a solution that will allow me to index these documents with a different structure.
I know about the ignore_malformed parameter, that will allow me to suppress the error, but as I want the "data" object to be indexed, that's not a solution.
I can Strinfigy the entire "date" object, which will also suppress the error, but then searching through the JSON will be inferior.
Would appreciate additional insights.