# ECS Logger Mapping Logs to ECS Fields

**URL:** <https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932>\
**Category:** Logs\
**Tags:** ecs-elastic-common-schema\
**Created:** [June 4, 2021, 12:21am UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932 "2021-06-04T00:21:31Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mertayd](https://avatars.discourse-cdn.com/v4/letter/m/ecccb3/32.png) [@mertayd](https://discuss.elastic.co/u/mertayd)\
**Post date:** [June 4, 2021, 12:21am UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/1 "2021-06-04T00:21:31Z")

</div>

I am quite new using Elastic and I have log data from AWS CloudWatch that I have shipped into Elastic Cluster using filebeats however I couldn't understand how I can map my logs into ECS fields. I read a lot of documentation but neither of them shows actually do this. I am using ECS logger however all my logs are sent in messages and I can't aggregate on it. Main question is how can log my messages in a certain format so I can. aggregate and visualize on Kibana?

> **[Integrating custom logs with ECS for Elastic SIEM](https://www.elastic.co/webinars/integrating-custom-logs-with-ecs-for-elastic-siem)**
>
> Deep dive into the clean, organized belly of the Elastic Common Schema. We’ll show you how to better understand the SIEM app and we’ll walk through converting new formats to the Elastic Common Schema — enabling all of your logs to work with the SIEM...

I watched the webinar above and I didn't understand how he decides to map his data to particular ecs fields. And after that how can we make sure that our log will be mapped to those fields.

I am trying to create a mapping for example,  
event.name = postprocessor  
event.source = cirus  
event.type = success  
and etc...

I couldn't understand how to work with ECS logger so that I can visualize this data in Kibana. I would really appreciate the help.

---

<div class="post-metadata">

**Author:** ![Kerry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kerry/32/40330_2.png) [@Kerry](https://discuss.elastic.co/u/Kerry)\
**Post date:** [June 4, 2021, 9:24am UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/2 "2021-06-04T09:24:31Z")

</div>

Hi,

You can find overall documentation for ECS [here](https://www.elastic.co/guide/en/ecs/current/index.html).

The fields of interest in this scenario will be the [event](https://www.elastic.co/guide/en/ecs/current/ecs-event.html) fields and [log](https://www.elastic.co/guide/en/ecs/current/ecs-log.html) fields.

Adherence to these formats will allow you to view your logs in the Logs UI, but also things like Discover.

---

<div class="post-metadata">

**Author:** ![mertayd](https://avatars.discourse-cdn.com/v4/letter/m/ecccb3/32.png) [@mertayd](https://discuss.elastic.co/u/mertayd)\
**Post date:** [June 7, 2021, 5:50pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/3 "2021-06-07T17:50:11Z")

</div>

Hello Kerry, thank you for your reply!

I read all of those documents, however I couldn't find the part where they explain how to adhere those formats.

Here is my log from ECS-logger that is displayed as only under message field on Kibana  
2021-06-07T13:18:47.110Z 2021-06-07 13:18.47 [info] {'labels': {'scan.category': 'APP\_SECURTY\_CATEGORY', 'scan.type': 'VULNCHECK', 'scan.outcome': 'SUCCESSFULL', 'scan.reason': 'NO\_ERROR'}, 'message': 'Mertay Message', 'event': {}}

```auto
 logger = logging.getLogger("ecs-logger")
 logger.setLevel(logging.INFO)

 # Add an ECS formatter to the Handler
 handler = logging.StreamHandler()
 formatter = ecs_logging.StdlibFormatter()
 handler.setFormatter(formatter)
 logger.addHandler(handler)

 my_dict = { "labels": {
     "scan.category": "APP_SECURTY_CATEGORY",
     "scan.type": "VULNCHECK",
     "scan.outcome": "SUCCESSFULL",
     "scan.reason": "NO_ERROR",
      },
 "message": "Mertay Message",
 "event": { }
 }

 logger.info(my_dict)

```

Could you help me identify what I am doing wrong? Because I am sure this problem that I am having is very fundamental, I just don't have enough experience and couldn't find any guidance on how do I create custom fields and log them properly. I have read the article about how to create custom fields, and using their scheme but it doesn't help either. I really appreciate the help! Thank you!

---

<div class="post-metadata">

**Author:** ![ebeahan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebeahan/32/78989_2.png) [@ebeahan](https://discuss.elastic.co/u/ebeahan)\
**Post date:** [June 8, 2021, 3:49pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/4 "2021-06-08T15:49:47Z")

</div>

Hi, @mertayd!

Using `ecs-logging` with the `logging` module, you need to put your additional fields into the `extra` attribute.

```auto
my_dict = { "labels": {
     "scan.category": "APP_SECURTY_CATEGORY",
     "scan.type": "VULNCHECK",
     "scan.outcome": "SUCCESSFULL",
     "scan.reason": "NO_ERROR",
      },
 "event": { }
 }

 logger.info("Mertay Message", extra=my_dict)

```

Some more detail on using `ecs-logging-python` with the `logging` module is covered here: [Installation | ECS Logging Python Reference | Elastic](https://www.elastic.co/guide/en/ecs-logging/python/current/installation.html#logging)

A behavior I noted while testing this out with your example. The dot (`.`) character in key names will be auto-expanded.

For example, the `"scan.category": "APP_SECURITY_CATEGORY"` field under `labels` became:

```auto
  "labels": {
    "scan": {
      "category": "APP_SECURTY_CATEGORY"
    }
  }

```

If you're following the ECS spec for the [`labels` field](https://www.elastic.co/guide/en/ecs/current/ecs-base.html#field-labels), you will need to avoid using nested objects underneath `labels`.

---

<div class="post-metadata">

**Author:** ![mertayd](https://avatars.discourse-cdn.com/v4/letter/m/ecccb3/32.png) [@mertayd](https://discuss.elastic.co/u/mertayd)\
**Post date:** [June 8, 2021, 6:50pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/5 "2021-06-08T18:50:54Z")

</div>

Thank you for your prompt response @ebeahan !

However, even though I made the changes you suggested, I still see those fields in Elastic under message field.

Here is the updated snippet,

```auto
my_dict = { 
    "scan": {
        "category": "APP_SECURTY_CATEGORY",
        "type": "VULNCHECK",
        "outcome": "SUCCESSFULL",
        "reason": "NO_ERROR"
    }
}
def main(event, context):
    logger.info("My FIRST CORRECT MESSAGE!!!", extra=my_dict)
    return "Done"

```

Elastic Log

 ![Screen Shot 2021-06-08 at 11.50.01 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/0/10f98b474defb88f463b57921489d47b63578c23.png)

I want to see scan field separately so I can visualize it on Kibana. That is my main problem. I hope you can help. Thank you in advance!

---

<div class="post-metadata">

**Author:** ![mertayd](https://avatars.discourse-cdn.com/v4/letter/m/ecccb3/32.png) [@mertayd](https://discuss.elastic.co/u/mertayd)\
**Post date:** [June 16, 2021, 5:14pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/6 "2021-06-16T17:14:18Z")

</div>

I still need help here

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [June 16, 2021, 5:43pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/7 "2021-06-16T17:43:41Z")

</div>

Hi @mertayd,

from your screenshot it looks like your logged data are serialized into `aws.cloudwatch.message` as JSON on their way through cloudwatch. To deserialize the data you can use the [`decode_json_fields` processor](https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html) in your `aws` module configuration such that it contains something like the following:

```auto
- module: aws
  # ... other aws filesets ...

  cloudwatch:
    enabled: true
    # ... other cloudwatch config ...

    input:
      processors:
        - decode_json_fields:
            fields: ["aws.cloudwatch.message"] # the field to parse
            target: "" # write to root of document
            overwrite_keys: true # might be needed to override @timestamp and similar fields

```

I couldn't test this exact setup, but maybe it unblocks you in figuring it out. 🤞

_(edited: fix typo)_  
_(edited: add `overwrite_keys`)_

---

<div class="post-metadata">

**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\
**Post date:** [June 16, 2021, 5:48pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/8 "2021-06-16T17:48:03Z")

</div>

And another thought: Since you're adding your custom fields (such as `scan`) I'd recommend to enhance the index template to include mappings for these. Otherwise the dynamic mapping might cause the `scan.*` fields to be interpreted in a way that hinders querying.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 14, 2021, 5:48pm UTC](https://discuss.elastic.co/t/ecs-logger-mapping-logs-to-ecs-fields/274932/9 "2021-07-14T17:48:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.