Parsing Data in message field

I am quite new to Elastic altogether, namely elastic agent, so looking for some guidance! To further preface, I am calling this parsing, but based on what I am requesting, that may not be the correct term for what I am attempting to accomplish.

I have an EA fleet that was just deployed as a daemonset on our k8s cluster using the journald and kubernetes integration. It is pulling data from the containers wonderfully.

So what I ended up doing is creating/editing the logs-kubernetes.container_logs@custom component template by pulling just the message field as such:

{
  "index": {
    "lifecycle": {
      "name": "logs"
    },
    "codec": "best_compression",
    "default_pipeline": "logs-kubernetes.container_logs-1.19.1",
    "mapping": {
      "total_fields": {
        "limit": "1"
      },
      "ignore_malformed": "true"
    },
    "query": {
      "default_field": [
        "message"
      ]
    }
  }
}

Now, I am unsure how to parse data in this specific field; the message field in has data that is in json format in the value field. An example would be as such:

{"version":"1.0","timestamp":1711548806710,"tenantId":"8675309","environmentId":"123456","environmentName":"PreProd","type":"transactionSummary","transactionSummary":{"status":"Failure","statusDetail":404,"duration":1234,"proxy":{"id":"unknown","name":"unknown","revision":"unknown"},"runtime":{"id":"777777","name":"Test Runtime"},"entryPoint":{"type":"http","method":"GET","path":"/","host":"test-123-localhost.net"}}}

Now I basically want to pull the data from the message field and have it create custom fields based on what the schema of the log is in the message. I initially thought this was done under mapped fields in mappings within the component template, but that is for already indexed documents.

How do I go about doing this?

Uncertain if pertinent:
Stack Management version: 8.11.0
Elastic Agent Version: 8.10.4

Hi @Sheperd403 welcome to the community.

You are in headed in the right direction...

See this

Create an ingest pipeline

logs-kubernetes.container_logs@custom

Use Json processor

After that use the date processor if you want to set the @timestamp

1 Like