Help with Indexing log with custom time instead of timestamp

Elastic Stack Beats
,
1

Hi everyone

I've established ELK with GitHub - deviantony/docker-elk: The Elastic stack (ELK) powered by Docker and Compose.
I have log with timestamp included -

//{"type":"CustoumLogType","Param1":1,"Param2":1,"Param3":1,"logging_time":"2021-06-25 20:10:10"}

My problem is that the logs are indexed with @timestamp of index time and not the indexed timestamp.

I've tried to replace @timestamp with logging_time from my log following this example -

The config of the whole ELK is default like the docker, the filebeat config is as follows -


filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /logs/*_log.json
  json.keys_under_root: true  
  json.add_error_key: true

- type: filestream
  enabled: true
  paths:
    - /logs/*_log.json
  json.keys_under_root: true
  json.add_error_key: true

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  username: "elastic"
  password: "changeme"

processors:
  #https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html
  - timestamp:
      field: logging_time
      target_field: @timestamp
      layouts:
        - UNIX_MS
      test: 
        - '1624523887321814'
  - add_host_metadata:
      when.not.contains.tags: forwarded

No matter what i tried, ES ignored logging_time time and indexed it as text field and not as time.

I want to build kibana histograms using that time.

Really would for help with this!
Thanks

2

Hi @Core_Sec Welcome to the community

From the docs here

The timestamp processor parses a timestamp from a field. By default the timestamp processor writes the parsed result to the @timestamp field. You can specify a different field by setting the target_field parameter. The timestamp value is parsed according to the layouts parameter. Multiple layouts can be specified and they will be used sequentially to attempt parsing the timestamp field.

The timestamp layouts used by this processor are different than the formats supported by date processors in Logstash and Elasticsearch Ingest Node.
SUPER IMPORTANT
The layouts are described using a reference time that is based on this specific time:

Mon Jan 2 15:04:05 MST 2006

Since MST is GMT-0700, the reference time is:

01/02 03:04:05PM '06 -0700. <<!--- (NOTE see the pattern 1234567)

To define your own layout, rewrite the reference time in a format that matches the timestamps you expect to parse. For more layout examples and details see the Go time package documentation.

Your example time 2021-06-25 20:10:10 is certainly not UNIX_MS so your format needs to expressed in terms of the reference time.

So your time

2021-06-25 20:10:10

Would be expressed as

2006-01-02 15:04:05

processors:
  #https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html
  - timestamp:
      field: logging_time
      target_field: @timestamp
      layouts:
        - '2006-01-02 15:04:05' <!-- Should be the correct layout assuming your time is UTC
      test: 
        - '2021-06-25 20:10:10'
  - add_host_metadata:
      when.not.contains.tags: forwarded. <!--- Make sure this is a good condition or this will never execute

Give it a try

3

Thanks for you answer and welcoming!

I accidentally uploaded the config after some tries, originally used the layout that you've said.
I've removed the add_host_metadata, my time is UTC indeed.

It still doesn't work, i', getting timestamp to be the indexing time and the logging_time is just a text field:

@timestamp
Jun 26, 2021 @ 22:02:18.895

logging_time
2021-06-25 19:15:60

processors:
  #https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html
  - timestamp:
      field: logging_time
      layouts:
        - '2006-01-02 15:04:05'
      test: 
        - '2021-06-25 22:22:30'

Any idead?

4

If If you run filebeat with -d "*" you'll get a bunch of debug.

Plus can you provide a couple actual sample lines of your logs.

And a couple result documents that are in filebeat.

Plus can you show the actual input section of your filebeat.yml

And what version are you using

5

Hi,

I'm getting lot's of debug data? what's relevant?

Couple of sample lines from the log:

{"type":"LogType1","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:23:30"}
{"type":"LogType2","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:24:30"}
{"type":"LogType3","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:25:30"}

Documents from ES:

{
  "_index": "filebeat-7.13.2-2021.06.26-000001",
  "_type": "_doc",
  "_id": "iFKzSXoBEvvJtRoZhrBK",
  "_version": 1,
  "_score": null,
  "fields": {
    "logging_time": [
      "2021-06-25 19:15:60"
    ],
    "channel": [
      3
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      510
    ],
    "type": [
      "LogType1"
    ],
    "agent.hostname": [
      "WorkPC.local"
    ],
    "agent.type": [
      "filebeat"
    ],
    "@timestamp": [
      "2021-06-26T19:02:18.895Z"
    ],
    "agent.id": [
      "b5197535-ffd0-48ce-b54c-5eb7d9bd7c19"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "log.file.path": [
      "/logs/9_log.json"
    ],
    "agent.ephemeral_id": [
      "5fa1591e-6195-4ae3-846d-7c5bc1834565"
    ],
    "block": [
      2
    ],
    "agent.version": [
      "7.13.2"
    ],
    "agent.name": [
      "WorkPC.local"
    ],
    "page": [
      6
    ],
    "host.name": [
      "WorkPC.local"
    ]
  },
  "sort": [
    1624734138895
  ]
}

Input section:

filebeat.inputs:

- type: log
  enabled: true

  paths:
    - /logs/*_log.json
  json.keys_under_root: true  
  json.add_error_key: true
- type: filestream

  enabled: true

  paths:
    - /logs/*_log.json
  json.keys_under_root: true
  json.add_error_key: true

Using Filebeat filebeat version 7.13.2 (amd64), libbeat 7.13.2 [686ba416a74193f2e69dcfa2eb142f4364a79307 built 2021-06-10 21:04:13 +0000 UTC]

Using ES 7.13.2

Thanks!

6

First you have both the log and filestream enabled and pointed to the same logs, probably don't want that.

my log file (yours)

{"type":"LogType1","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:23:30"}
{"type":"LogType2","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:24:30"}
{"type":"LogType3","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:25:30"}

My minimal filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log
  json.keys_under_root: true  
  json.add_error_key: true  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - timestamp:
      field: logging_time
      layouts:
        - '2006-01-02 15:04:05'
      test: 
        - '2021-06-25 20:10:10'

my command

./filebeat -e -c ./filebeat-minimum.yml

My results Kibana - Dev Tools

GET filebeat-7.13.2-2021.06.27-000001/_search
{
  "_source": false,
  "fields": [ "*" ]
}

results
Note : the @timestamp is set to logging_time

{
  "took" : 43,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.13.2-2021.06.27-000001",
        "_type" : "_doc",
        "_id" : "eF0KS3oBgW1UHy0ALsPE",
        "_score" : 1.0,
        "fields" : {
          "logging_time" : [
            "2021-06-25 19:23:30" <!---- logging_time
          ],
          "host.os.name.text" : [
            "Mac OS X"
          ],
          "channel" : [
            2
          ],
          "host.hostname" : [
            "ceres"
          ],
          "type" : [
            "LogType1"
          ],
          "host.mac" : [
            "8c:85:90:ae:b0:b2",
            "82:de:c3:e6:d4:05",
            "82:de:c3:e6:d4:04",
            "82:de:c3:e6:d4:01",
            "82:de:c3:e6:d4:00",
            "82:de:c3:e6:d4:01",
            "0e:85:90:ae:b0:b2",
            "d6:54:f2:7a:9c:4c",
            "d6:54:f2:7a:9c:4c",
            "ac:de:48:00:11:22"
          ],
          "host.os.build" : [
            "20F71"
          ],
          "host.ip" : [
            "fe80::84d:3bc2:8250:27e0",
            "192.168.2.205",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::377:1386:b3db:403e",
            "fe80::fa7e:d5f1:2ea7:76ee",
            "fe80::aede:48ff:fe00:1122"
          ],
          "agent.type" : [
            "filebeat"
          ],
          "host.os.version" : [
            "10.16"
          ],
          "host.os.kernel" : [
            "20.5.0"
          ],
          "host.os.name" : [
            "Mac OS X"
          ],
          "block" : [
            1
          ],
          "agent.name" : [
            "ceres"
          ],
          "host.name" : [
            "ceres"
          ],
          "host.id" : [
            "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          ],
          "host.os.type" : [
            "macos"
          ],
          "input.type" : [
            "log"
          ],
          "log.offset" : [
            0
          ],
          "agent.hostname" : [
            "ceres"
          ],
          "host.architecture" : [
            "x86_64"
          ],
          "@timestamp" : [
            "2021-06-25T19:23:30.000Z"  <!--- Set to logging_time
          ],
          "agent.id" : [
            "1d5a1f47-65b3-4d65-bcaf-ad2f378e846c"
          ],
          "host.os.platform" : [
            "darwin"
          ],
          "ecs.version" : [
            "1.8.0"
          ],
          "log.file.path" : [
            "/Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log"
          ],
          "agent.ephemeral_id" : [
            "02a798bf-8b33-4341-b449-19068cde55d9"
          ],
          "agent.version" : [
            "7.13.2"
          ],
          "page" : [
            3
          ],
          "host.os.family" : [
            "darwin"
          ]
        }
      },
      {
        "_index" : "filebeat-7.13.2-2021.06.27-000001",
        "_type" : "_doc",
        "_id" : "eV0KS3oBgW1UHy0ALsPE",
        "_score" : 1.0,
        "fields" : {
          "logging_time" : [
            "2021-06-25 19:24:30"
          ],
          "host.os.name.text" : [
            "Mac OS X"
          ],
          "channel" : [
            2
          ],
          "host.hostname" : [
            "ceres"
          ],
          "type" : [
            "LogType2"
          ],
          "host.mac" : [
            "8c:85:90:ae:b0:b2",
            "82:de:c3:e6:d4:05",
            "82:de:c3:e6:d4:04",
            "82:de:c3:e6:d4:01",
            "82:de:c3:e6:d4:00",
            "82:de:c3:e6:d4:01",
            "0e:85:90:ae:b0:b2",
            "d6:54:f2:7a:9c:4c",
            "d6:54:f2:7a:9c:4c",
            "ac:de:48:00:11:22"
          ],
          "host.os.build" : [
            "20F71"
          ],
          "host.ip" : [
            "fe80::84d:3bc2:8250:27e0",
            "192.168.2.205",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::377:1386:b3db:403e",
            "fe80::fa7e:d5f1:2ea7:76ee",
            "fe80::aede:48ff:fe00:1122"
          ],
          "agent.type" : [
            "filebeat"
          ],
          "host.os.version" : [
            "10.16"
          ],
          "host.os.kernel" : [
            "20.5.0"
          ],
          "host.os.name" : [
            "Mac OS X"
          ],
          "block" : [
            1
          ],
          "agent.name" : [
            "ceres"
          ],
          "host.name" : [
            "ceres"
          ],
          "host.id" : [
            "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          ],
          "host.os.type" : [
            "macos"
          ],
          "input.type" : [
            "log"
          ],
          "log.offset" : [
            88
          ],
          "agent.hostname" : [
            "ceres"
          ],
          "host.architecture" : [
            "x86_64"
          ],
          "@timestamp" : [
            "2021-06-25T19:24:30.000Z"
          ],
          "agent.id" : [
            "1d5a1f47-65b3-4d65-bcaf-ad2f378e846c"
          ],
          "host.os.platform" : [
            "darwin"
          ],
          "ecs.version" : [
            "1.8.0"
          ],
          "log.file.path" : [
            "/Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log"
          ],
          "agent.ephemeral_id" : [
            "02a798bf-8b33-4341-b449-19068cde55d9"
          ],
          "agent.version" : [
            "7.13.2"
          ],
          "page" : [
            3
          ],
          "host.os.family" : [
            "darwin"
          ]
        }
      },
      {
        "_index" : "filebeat-7.13.2-2021.06.27-000001",
        "_type" : "_doc",
        "_id" : "el0KS3oBgW1UHy0ALsPE",
        "_score" : 1.0,
        "fields" : {
          "logging_time" : [
            "2021-06-25 19:25:30"
          ],
          "host.os.name.text" : [
            "Mac OS X"
          ],
          "channel" : [
            2
          ],
          "host.hostname" : [
            "ceres"
          ],
          "type" : [
            "LogType3"
          ],
          "host.mac" : [
            "8c:85:90:ae:b0:b2",
            "82:de:c3:e6:d4:05",
            "82:de:c3:e6:d4:04",
            "82:de:c3:e6:d4:01",
            "82:de:c3:e6:d4:00",
            "82:de:c3:e6:d4:01",
            "0e:85:90:ae:b0:b2",
            "d6:54:f2:7a:9c:4c",
            "d6:54:f2:7a:9c:4c",
            "ac:de:48:00:11:22"
          ],
          "host.os.build" : [
            "20F71"
          ],
          "host.ip" : [
            "fe80::84d:3bc2:8250:27e0",
            "192.168.2.205",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::377:1386:b3db:403e",
            "fe80::fa7e:d5f1:2ea7:76ee",
            "fe80::aede:48ff:fe00:1122"
          ],
          "agent.type" : [
            "filebeat"
          ],
          "host.os.version" : [
            "10.16"
          ],
          "host.os.kernel" : [
            "20.5.0"
          ],
          "host.os.name" : [
            "Mac OS X"
          ],
          "block" : [
            1
          ],
          "agent.name" : [
            "ceres"
          ],
          "host.name" : [
            "ceres"
          ],
          "host.id" : [
            "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          ],
          "host.os.type" : [
            "macos"
          ],
          "input.type" : [
            "log"
          ],
          "log.offset" : [
            176
          ],
          "agent.hostname" : [
            "ceres"
          ],
          "host.architecture" : [
            "x86_64"
          ],
          "@timestamp" : [
            "2021-06-25T19:25:30.000Z"
          ],
          "agent.id" : [
            "1d5a1f47-65b3-4d65-bcaf-ad2f378e846c"
          ],
          "host.os.platform" : [
            "darwin"
          ],
          "ecs.version" : [
            "1.8.0"
          ],
          "log.file.path" : [
            "/Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log"
          ],
          "agent.ephemeral_id" : [
            "02a798bf-8b33-4341-b449-19068cde55d9"
          ],
          "agent.version" : [
            "7.13.2"
          ],
          "page" : [
            3
          ],
          "host.os.family" : [
            "darwin"
          ]
        }
      }
    ]
  }
}
7

Awesome! It worked!
Thanks for the help!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.