Same field different types (primitive vs. complex)

please forgive me for this very basic question but I can't find an answer so far. I'm sending my logs to AWS CloudWatch and from there I'm using an AWS Lambda to push said logs to AWS hosted ElasticSearch.

The problem I have is that my logs even though are json they are not well structured. Sometimes a field can be a simple value like a string but the very next log line might have the same field as an object.
log line A:
{"msg": "foo bar", "status": "Success"}
log line B:
{"msg": "quux", "status": {"finished": "2020-02-05T18:37:15Z"}}

This way I can't really index all my log lines. What's the preferred way of dealing with this issue?

You should handle this before indexing these kind of logs to elasticsearch and convert them to any one type. Otherwise it will result in mapping conflict.

I don't think you can index these kind of logs to an index.

These are third party applications, I can't know what they're going to look like. Perhaps best option would be to store the log line as a string instead of parsing it?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.