# Parsing Data in message field

**URL:** <https://discuss.elastic.co/t/parsing-data-in-message-field/356300>\
**Category:** Elastic Agent\
**Created:** [March 27, 2024, 3:14pm UTC](https://discuss.elastic.co/t/parsing-data-in-message-field/356300 "2024-03-27T15:14:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sheperd403](https://avatars.discourse-cdn.com/v4/letter/s/6bbea6/32.png) [@Sheperd403](https://discuss.elastic.co/u/Sheperd403)\
**Post date:** [March 27, 2024, 3:14pm UTC](https://discuss.elastic.co/t/parsing-data-in-message-field/356300/1 "2024-03-27T15:14:40Z")

</div>

I am quite new to Elastic altogether, namely elastic agent, so looking for some guidance! To further preface, I am calling this parsing, but based on what I am requesting, that may not be the correct term for what I am attempting to accomplish.

I have an EA fleet that was just deployed as a daemonset on our k8s cluster using the journald and kubernetes integration. It is pulling data from the containers wonderfully.

So what I ended up doing is creating/editing the logs-kubernetes.container\_logs@custom component template by pulling just the message field as such:

```auto
{
  "index": {
    "lifecycle": {
      "name": "logs"
    },
    "codec": "best_compression",
    "default_pipeline": "logs-kubernetes.container_logs-1.19.1",
    "mapping": {
      "total_fields": {
        "limit": "1"
      },
      "ignore_malformed": "true"
    },
    "query": {
      "default_field": [
        "message"
      ]
    }
  }
}

```

Now, I am unsure how to parse data in this specific field; the message field in has data that is in json format in the value field. An example would be as such:

```auto
{"version":"1.0","timestamp":1711548806710,"tenantId":"8675309","environmentId":"123456","environmentName":"PreProd","type":"transactionSummary","transactionSummary":{"status":"Failure","statusDetail":404,"duration":1234,"proxy":{"id":"unknown","name":"unknown","revision":"unknown"},"runtime":{"id":"777777","name":"Test Runtime"},"entryPoint":{"type":"http","method":"GET","path":"/","host":"test-123-localhost.net"}}}

```

 ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/e/9/e9b2fd0be97fe1350e63686caccc0a7a714e2416.png)

Now I basically want to pull the data from the message field and have it create custom fields based on what the schema of the log is in the message. I initially thought this was done under mapped fields in mappings within the component template, but that is for already indexed documents.

**How do I go about doing this?**

Uncertain if pertinent:  
Stack Management version: 8.11.0  
Elastic Agent Version: 8.10.4

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 28, 2024, 6:29am UTC](https://discuss.elastic.co/t/parsing-data-in-message-field/356300/2 "2024-03-28T06:29:10Z")

</div>

Hi @Sheperd403 welcome to the community.

You are in headed in the right direction...

See this

> **[Tutorial: Transform data with custom ingest pipelines | Fleet and Elastic...](https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html)**

Create an ingest pipeline

`logs-kubernetes.container_logs@custom`

Use Json processor

> **[JSON processor | Elasticsearch Guide \[8.13\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html)**

After that use the date processor if you want to set the `@timestamp`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 25, 2024, 6:29am UTC](https://discuss.elastic.co/t/parsing-data-in-message-field/356300/3 "2024-04-25T06:29:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.