Same field different types (primitive vs. complex)

Elastic Stack Elasticsearch
1

Hi,
please forgive me for this very basic question but I can't find an answer so far. I'm sending my logs to AWS CloudWatch and from there I'm using an AWS Lambda to push said logs to AWS hosted ElasticSearch.

The problem I have is that my logs even though are json they are not well structured. Sometimes a field can be a simple value like a string but the very next log line might have the same field as an object.
e.g.
log line A:
{"msg": "foo bar", "status": "Success"}
log line B:
{"msg": "quux", "status": {"finished": "2020-02-05T18:37:15Z"}}

This way I can't really index all my log lines. What's the preferred way of dealing with this issue?

2

You should handle this before indexing these kind of logs to elasticsearch and convert them to any one type. Otherwise it will result in mapping conflict.

I don't think you can index these kind of logs to an index.

3

These are third party applications, I can't know what they're going to look like. Perhaps best option would be to store the log line as a string instead of parsing it?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.