# Best practice for JSON logging (e.g. timestamp processing)

**URL:** <https://discuss.elastic.co/t/best-practice-for-json-logging-e-g-timestamp-processing/284472>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [September 17, 2021, 10:00am UTC](https://discuss.elastic.co/t/best-practice-for-json-logging-e-g-timestamp-processing/284472 "2021-09-17T10:00:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![crazylogging](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@crazylogging](https://discuss.elastic.co/u/crazylogging)\
**Post date:** [September 17, 2021, 10:00am UTC](https://discuss.elastic.co/t/best-practice-for-json-logging-e-g-timestamp-processing/284472/1 "2021-09-17T10:00:24Z")

</div>

Hi,  
I'm quite new to the elastic stack so please excuse the question if it doesn't make any sense.  
I've got an application (let's call it JBoss) which runs in a docker container (all components in my scenario do so) and it produces json logs. In my understanding there is no need for a logstash instance, so I configured the following things  
**filebeat.yml**

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      hints.default_config.enabled: false

```

**docker-compose.yml**  
JBoss (further details not relevant I guess)

```auto
   labels:
      co.elastic.logs/enabled: true
      co.elastic.logs/json.keys_under_root: true
      co.elastic.logs/json.overwrite_keys: true
      co.elastic.logs/json.message_key: message
      co.elastic.logs/json.add_error_key: true

```

**This is one log entry from JBoss:**

```auto
{"timestamp":"2021-09-17T09:15:24.33Z","sequence":639,"loggerClassName":"org.jboss.logging.Logger","loggerName":"org.keycloak.transaction.JtaTransactionWrapper","level":"DEBUG","message":"JtaTransactionWrapper commit","threadName":"Timer-2","threadId":163,"mdc":{},"ndc":"","hostName":"83add8eceadd","processName":"jboss-modules.jar","processId":172}

```

The output in Kibana looks like this (shortened):

```auto
{
  "_index": "filebeat-7.14.0-2021.09.17-000001",
  "_type": "_doc",
  "_id": "tzAq83sBuczAaH_z4Yww",
  "_score": 1,
  "_source": {
    "@timestamp": "2021-09-17T09:51:16.728Z",
    "ndc": "",
    "loggerClassName": "org.jboss.logging.Logger",
    "log": {
      "offset": 2919243,
      "file": {
        "path": "/var/lib/docker/containers/83add8eceadd31f392e55c4af15cee46e409304bd9a874953442393c4ca1b28f/83add8eceadd31f392e55c4af15cee46e409304bd9a874953442393c4ca1b28f-json.log"
      }
    },
    "threadId": 163,
    "timestamp": "2021-09-17T09:51:16.727Z",
    "message": "Executed scheduled task AbstractLastSessionRefreshStoreFactory$$Lambda$2041/0x0000000841731840",
    "sequence": 1842,
    "host": {
      "architecture": "x86_64",
      "os": {
        "codename": "Core",
        "type": "linux",
        "platform": "centos",
        "version": "7 (Core)",
        "family": "redhat",
        "name": "CentOS Linux",
        "kernel": "5.10.47-linuxkit"
      },
      "id": "3c2fb2b6af196dc020fd58ec005618e7",
      "containerized": true,
      "ip": [
        "172.20.0.5"
      ]
      ],
      "hostname": "de5d54c85632",
      "name": "de5d54c85632"
    },
    "processName": "jboss-modules.jar",
    "level": "DEBUG",
    "input": {
      "type": "container"
    },
    "loggerName": "org.keycloak.services.scheduled.ScheduledTaskRunner",
    "hostName": "83add8eceadd",
    "stream": "stdout",
    "processId": 171,
   }
}

```

**Question:**  
My question is now how to deal e.g. with the timestamp field from the JBoss log which is slightly different from the @timestamp field. Is there a need for copying the timestamp field to the @timestamp field, as one is the date of processing the event and one is the real timestamp from the log entry? If so how to achieve this?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 20, 2021, 4:47am UTC](https://discuss.elastic.co/t/best-practice-for-json-logging-e-g-timestamp-processing/284472/2 "2021-09-20T04:47:33Z")

</div>

You could try [Copy fields | Filebeat Reference [7.14] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/copy-fields.html) where you copy `timestamp` to `@ timestamp`.

---

<div class="post-metadata">

**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\
**Post date:** [September 21, 2021, 1:04am UTC](https://discuss.elastic.co/t/best-practice-for-json-logging-e-g-timestamp-processing/284472/3 "2021-09-21T01:04:59Z")

</div>

Yes u should overwrite `@timestamp` with the times from the log. The `@timestamp` field should be when the log actually happened. The ECS fields `event.ingested` is when the log was actually ingested into ES.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2021, 3:05am UTC](https://discuss.elastic.co/t/best-practice-for-json-logging-e-g-timestamp-processing/284472/4 "2021-10-19T03:05:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.