Parsing Data in message field

I am quite new to Elastic altogether, namely elastic agent, so looking for some guidance! To further preface, I am calling this parsing, but based on what I am requesting, that may not be the correct term for what I am attempting to accomplish.

I have an EA fleet that was just deployed as a daemonset on our k8s cluster using the journald and kubernetes integration. It is pulling data from the containers wonderfully.

So what I ended up doing is creating/editing the logs-kubernetes.container_logs@custom component template by pulling just the message field as such:

  "index": {
    "lifecycle": {
      "name": "logs"
    "codec": "best_compression",
    "default_pipeline": "logs-kubernetes.container_logs-1.19.1",
    "mapping": {
      "total_fields": {
        "limit": "1"
      "ignore_malformed": "true"
    "query": {
      "default_field": [

Now, I am unsure how to parse data in this specific field; the message field in has data that is in json format in the value field. An example would be as such:

{"version":"1.0","timestamp":1711548806710,"tenantId":"8675309","environmentId":"123456","environmentName":"PreProd","type":"transactionSummary","transactionSummary":{"status":"Failure","statusDetail":404,"duration":1234,"proxy":{"id":"unknown","name":"unknown","revision":"unknown"},"runtime":{"id":"777777","name":"Test Runtime"},"entryPoint":{"type":"http","method":"GET","path":"/","host":""}}}

Now I basically want to pull the data from the message field and have it create custom fields based on what the schema of the log is in the message. I initially thought this was done under mapped fields in mappings within the component template, but that is for already indexed documents.

How do I go about doing this?

Uncertain if pertinent:
Stack Management version: 8.11.0
Elastic Agent Version: 8.10.4

Hi @Sheperd403 welcome to the community.

You are in headed in the right direction...

See this

Create an ingest pipeline


Use Json processor

After that use the date processor if you want to set the @timestamp

1 Like