I have CloudWatch logs being sent to elasticsearch from functionbeat. There is a filed called "message" that contains all the information I want to perform visualizations on. I've spent an entire day trying to figure out how to use your KQL method to practice parsing this "message" field and I've failed miserably on every level.
As you'll see below I think part of the problem is everything inside this "message" field isn't being treated as JSON but instead a single string. This is really complicating matters. I'm going to need guidance and you're going to have to explain it to me like i'm 5. Perhaps the answer is to create a ingest pipeline but I've no idea what that would look like. The answer could easily be a complex KQL query too.
I'm removing junk data out of this document that isn't important indicated by the characters ...
{
"_index": "functionbeat-7.7.0-2020.05.23-000001",
"_type": "_doc",
"_id": "tuv6P3IBbkn-TM6wEe-z",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2020-05-23T05:20:47.689Z",
...
"message": "{\"id\":\"4\",\"type\":\"PassStateEntered\",\"details\":{\"input\":\"\\\"starting state1\\\"\",\"name\":\"state2\"},\"previous_event_id\":\"3\",\"event_timestamp\":\"1590211247689\",\"execution_arn\":\"arn:aws-us-gov:states:us-gov-west-1:097135049942:execution:syost-step-workflow-a:e3644418-d5a4-e182-6f96-1b5db9de714f\"}",
"message_type": "DATA_MESSAGE",
...
}