# Timestamp not being overwriten (ECS)

**URL:** <https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544>\
**Category:** Beats\
**Tags:** ecs-elastic-common-schema, filebeat\
**Created:** [November 6, 2021, 4:00pm UTC](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544 "2021-11-06T16:00:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vcaleffi](https://avatars.discourse-cdn.com/v4/letter/v/a5b964/32.png) [@vcaleffi](https://discuss.elastic.co/u/vcaleffi)\
**Post date:** [November 6, 2021, 4:00pm UTC](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544/1 "2021-11-06T16:00:04Z")

</div>

Hello,

I'm very new to _filebeat_ and I'm trying to import logs created by my application (PHP - symfony). The application uses monolog and ECS formatter to write to a log file.

Monolog configuration:

```auto
monolog:
    handlers:
        main:
            type: fingers_crossed
            action_level: info
            handler: file_log
            formatter: formatter.ecs
            ....

```

_formatter.ecs_ is just the default ECS formatter nothing changed there.

And the log outputs are as expected (example):

> {"@timestamp":"2021-11-06T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error":"This shouldnt happen"}

I'm using filebeat to read this log file and send to ELK (7.15) with the given configuration:

```auto
filebeat.inputs:
...
- type: filestream
  enabled: true
  paths:
    - /app/var/log/prod.log
  fields_under_root: true
  overwrite_keys: true
  parsers:
    - ndjson:
      keys_under_root: true
      message_key: message
      overwrite_keys: true
...

```

It "works" but I'm sure Im missing something. When I was not using _ndjson_ parser the logs where arriving at ELK not interpreted, the whole log would be "message", no log level, no context and etc.

By using _ndjson_ parser it started to properly separate the message from the context but is it necessary if I'm already using ECS formatter? Should I use a different processor?  
And when using _ndjson_ with the configuration above the timestamp is not being overwritten.

And when using _Log_ instead of _filestream_ timestamp is also not overwritten

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 6, 2021, 6:16pm UTC](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544/2 "2021-11-06T18:16:12Z")

</div>

> [@vcaleffi](#):
>
> {"@timestamp":"2021-11-06T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error":"This shouldnt happen"}

Hi @vcaleffi You are close ... a couple things

Here is my test file First note you have a field `error` with just a text field that will need to be `error.message` because the [ECS field](https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-ecs.html#_error_2) `error` is an object not just a single concrete field. so it should be `error.message` or change it to a non ECS field `error_message` but `error.message` is a bit better IMHO

```auto
{"@timestamp":"2021-11-01T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-02T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-03T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-04T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-05T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}

```

Here is my filebeat.yml applicable section, all that is needed...

```auto
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /Users/sbrown/workspace/sample-data/filebeat-json/simple_test2_json.json
    
  json.keys_under_root: true
  json.overwrite_keys: true

```

Results are correct...

```auto
GET filebeat-7.14.1-2021.11.06-000001/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.14.1-2021.11.06-000001",
        "_type" : "_doc",
        "_id" : "Puxy9nwBcEQ7gMngUFH2",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-11-01T15:52:16.542Z", <!---- Correct Timestamp
          "input" : {
            "type" : "log"
          },
          "agent" : {
            "version" : "7.14.1",
            "hostname" : "ceres",
            "ephemeral_id" : "ef34eb6a-a7f1-4cfd-8e60-1826edadbb72",
            "id" : "f9d4f40a-24d5-44c2-b6ac-5c6d94648844",
            "name" : "ceres",
            "type" : "filebeat"
          },
          "ecs" : {
            "version" : "1.10.0"
          },
          "host" : {
            "name" : "ceres",
            "ip" : [
              "fe80::cd9:2824:1cbe:c9c2",
              "192.168.2.107",
              "fe80::554f:937:fb03:1fc0",
              "fe80::24b5:758f:aea5:fdda",
              "fe80::aede:48ff:fe00:1122"
            ],
            "mac" : [
              "8c:85:90:ae:b0:b2",
              "82:de:c3:e6:d4:01",
              "82:de:c3:e6:d4:00",
              "82:de:c3:e6:d4:05",
              "82:de:c3:e6:d4:04",
              "a0:ce:c8:51:95:38",
              "82:de:c3:e6:d4:01",
              "0e:85:90:ae:b0:b2",
              "92:c0:83:bf:96:fb",
              "92:c0:83:bf:96:fb",
              "ac:de:48:00:11:22"
            ],
            "hostname" : "ceres",
            "architecture" : "x86_64",
            "os" : {
              "type" : "macos",
              "platform" : "darwin",
              "version" : "10.16",
              "family" : "darwin",
              "name" : "Mac OS X",
              "kernel" : "20.6.0",
              "build" : "20G165"
            },
            "id" : "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          },
          "container" : {
            "id" : "filebeat-json"
          },
          "message" : "Some error message",
          "log.level" : "ERROR",
          "ecs.version" : "1.2.0",
          "error.message" : "This shouldnt happen",
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/sample-data/filebeat-json/simple_test2_json.json"
            },
            "logger" : "app",
            "offset" : 0
          }
        }
      }
.....

```

---

<div class="post-metadata">

**Author:** ![vcaleffi](https://avatars.discourse-cdn.com/v4/letter/v/a5b964/32.png) [@vcaleffi](https://discuss.elastic.co/u/vcaleffi)\
**Post date:** [November 6, 2021, 6:41pm UTC](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544/3 "2021-11-06T18:41:03Z")

</div>

Hey @stephenb Thank you so much for your reply.  
This is perfect. Its now working as I wanted. And it was way easier than what I was going for

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 4, 2021, 8:41pm UTC](https://discuss.elastic.co/t/timestamp-not-being-overwriten-ecs/288544/4 "2021-12-04T20:41:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.