Timestamp not being overwriten (ECS)

Elastic Stack Beats
,
1

Hello,

I'm very new to filebeat and I'm trying to import logs created by my application (PHP - symfony). The application uses monolog and ECS formatter to write to a log file.

Monolog configuration:

monolog:
    handlers:
        main:
            type: fingers_crossed
            action_level: info
            handler: file_log
            formatter: formatter.ecs
            ....

formatter.ecs is just the default ECS formatter nothing changed there.

And the log outputs are as expected (example):

{"@timestamp":"2021-11-06T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error":"This shouldnt happen"}

I'm using filebeat to read this log file and send to ELK (7.15) with the given configuration:

filebeat.inputs:
...
- type: filestream
  enabled: true
  paths:
    - /app/var/log/prod.log
  fields_under_root: true
  overwrite_keys: true
  parsers:
    - ndjson:
      keys_under_root: true
      message_key: message
      overwrite_keys: true
...

It "works" but I'm sure Im missing something. When I was not using ndjson parser the logs where arriving at ELK not interpreted, the whole log would be "message", no log level, no context and etc.

By using ndjson parser it started to properly separate the message from the context but is it necessary if I'm already using ECS formatter? Should I use a different processor?
And when using ndjson with the configuration above the timestamp is not being overwritten.

And when using Log instead of filestream timestamp is also not overwritten

2

Hi @vcaleffi You are close ... a couple things

Here is my test file First note you have a field error with just a text field that will need to be error.message because the ECS field error is an object not just a single concrete field. so it should be error.message or change it to a non ECS field error_message but error.message is a bit better IMHO

{"@timestamp":"2021-11-01T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-02T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-03T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-04T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}
{"@timestamp":"2021-11-05T12:52:16.542954-03:00","log.level":"ERROR","message":"Some error message","ecs.version":"1.2.0","log":{"logger":"app"},"error.message":"This shouldnt happen"}

Here is my filebeat.yml applicable section, all that is needed...

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    -  /Users/sbrown/workspace/sample-data/filebeat-json/simple_test2_json.json
    
  json.keys_under_root: true
  json.overwrite_keys: true

Results are correct...

GET filebeat-7.14.1-2021.11.06-000001/_search

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.14.1-2021.11.06-000001",
        "_type" : "_doc",
        "_id" : "Puxy9nwBcEQ7gMngUFH2",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2021-11-01T15:52:16.542Z", <!---- Correct Timestamp
          "input" : {
            "type" : "log"
          },
          "agent" : {
            "version" : "7.14.1",
            "hostname" : "ceres",
            "ephemeral_id" : "ef34eb6a-a7f1-4cfd-8e60-1826edadbb72",
            "id" : "f9d4f40a-24d5-44c2-b6ac-5c6d94648844",
            "name" : "ceres",
            "type" : "filebeat"
          },
          "ecs" : {
            "version" : "1.10.0"
          },
          "host" : {
            "name" : "ceres",
            "ip" : [
              "fe80::cd9:2824:1cbe:c9c2",
              "192.168.2.107",
              "fe80::554f:937:fb03:1fc0",
              "fe80::24b5:758f:aea5:fdda",
              "fe80::aede:48ff:fe00:1122"
            ],
            "mac" : [
              "8c:85:90:ae:b0:b2",
              "82:de:c3:e6:d4:01",
              "82:de:c3:e6:d4:00",
              "82:de:c3:e6:d4:05",
              "82:de:c3:e6:d4:04",
              "a0:ce:c8:51:95:38",
              "82:de:c3:e6:d4:01",
              "0e:85:90:ae:b0:b2",
              "92:c0:83:bf:96:fb",
              "92:c0:83:bf:96:fb",
              "ac:de:48:00:11:22"
            ],
            "hostname" : "ceres",
            "architecture" : "x86_64",
            "os" : {
              "type" : "macos",
              "platform" : "darwin",
              "version" : "10.16",
              "family" : "darwin",
              "name" : "Mac OS X",
              "kernel" : "20.6.0",
              "build" : "20G165"
            },
            "id" : "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          },
          "container" : {
            "id" : "filebeat-json"
          },
          "message" : "Some error message",
          "log.level" : "ERROR",
          "ecs.version" : "1.2.0",
          "error.message" : "This shouldnt happen",
          "log" : {
            "file" : {
              "path" : "/Users/sbrown/workspace/sample-data/filebeat-json/simple_test2_json.json"
            },
            "logger" : "app",
            "offset" : 0
          }
        }
      }
.....
1 Like
3

Hey @stephenb Thank you so much for your reply.
This is perfect. Its now working as I wanted. And it was way easier than what I was going for

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.