I am quite new to Elastic altogether, namely elastic agent, so looking for some guidance! To further preface, I am calling this parsing, but based on what I am requesting, that may not be the correct term for what I am attempting to accomplish.
I have an EA fleet that was just deployed as a daemonset on our k8s cluster using the journald and kubernetes integration. It is pulling data from the containers wonderfully.
So what I ended up doing is creating/editing the logs-kubernetes.container_logs@custom component template by pulling just the message field as such:
{
"index": {
"lifecycle": {
"name": "logs"
},
"codec": "best_compression",
"default_pipeline": "logs-kubernetes.container_logs-1.19.1",
"mapping": {
"total_fields": {
"limit": "1"
},
"ignore_malformed": "true"
},
"query": {
"default_field": [
"message"
]
}
}
}
Now, I am unsure how to parse data in this specific field; the message field in has data that is in json format in the value field. An example would be as such:
{"version":"1.0","timestamp":1711548806710,"tenantId":"8675309","environmentId":"123456","environmentName":"PreProd","type":"transactionSummary","transactionSummary":{"status":"Failure","statusDetail":404,"duration":1234,"proxy":{"id":"unknown","name":"unknown","revision":"unknown"},"runtime":{"id":"777777","name":"Test Runtime"},"entryPoint":{"type":"http","method":"GET","path":"/","host":"test-123-localhost.net"}}}
Now I basically want to pull the data from the message field and have it create custom fields based on what the schema of the log is in the message. I initially thought this was done under mapped fields in mappings within the component template, but that is for already indexed documents.
How do I go about doing this?
Uncertain if pertinent:
Stack Management version: 8.11.0
Elastic Agent Version: 8.10.4