I have a fresh Elasticsearch/Kibana instance that I'm trying to feed data to from Cloudwatch. I was able to get Functionbeat set up easily and data flows in as expected, but all of the logs are simply being put into the "message" field.
I was previously using Elasticsearch/Kibana directly in AWS using the ElasticSearch service and it was able to automatically index the JSON that I was logging. I suspect my issue here is that my lambda log messages don't just contain the JSON that I logged, but also a timestamp, GUID, log_type, and finally the actual data the I told lambda to log. This lead me down the route of trying to use processors to "dissect" the metadata from the message, and I was able to get it to extract just the JSON in one log message, but when I then tried to use that field with "decode_json_fields" it doesn't have any effect. I would also like to be able to only send "DEBUG" messages to elasticsearch, which I figured would be done with a filter_pattern, but no matter what I put in that field it doesn't seem to do anything.
I've spent multiple days trying to figure this out and I have a feeling it shouldn't be this difficult, especially since I'm sure reading lambda logs out of cloudwatch is an extremely common use-case.
What am I doing wrong? What series of processors do I need to set up for functionbeat to be able to read my JSON? Should I log JSON or stringified JSON? (we use node.js with lambda)
Please help!
Here is a sample of the data that is logged in cloudwatch and I am trying to parse/extract:
2021-06-21T11:31:05.244-04:00 START RequestId: 21098901-a734-4ff6-83ea-6a9383e4795b Version: $LATEST
2021-06-21T11:31:05.263-04:00 2021-06-21T15:31:05.262Z 21098901-a734-4ff6-83ea-6a9383e4795b INFO {"zip":"90210"}
2021-06-21T11:31:05.748-04:00 2021-06-21T15:31:05.747Z 21098901-a734-4ff6-83ea-6a9383e4795b INFO []
2021-06-21T11:31:05.779-04:00 2021-06-21T15:31:05.778Z 21098901-a734-4ff6-83ea-6a9383e4795b DEBUG {"log_type":"Test Log Event","attr1":0,"attr2":0,"zip":"90210"}
2021-06-21T11:31:05.788-04:00 2021-06-21T15:31:05.788Z 21098901-a734-4ff6-83ea-6a9383e4795b INFO Response: { title: 'Log Message Title', request_id: '21098901-a734-4ff6-83ea-6a9383e4795b', request_timestamp: '2021-06-21 03:31 pm', attr: 0, status: true }
2021-06-21T11:31:05.814-04:00 2021-06-21T15:31:05.814Z 21098901-a734-4ff6-83ea-6a9383e4795b INFO SERVERLESS_ENTERPRISE {"c":true,"b":"${base_64_encoded_string}","origin":"sls-agent"}
2021-06-21T11:31:05.815-04:00 END RequestId: 21098901-a734-4ff6-83ea-6a9383e4795b