# Help with Indexing log with custom time instead of timestamp

**URL:** <https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [June 26, 2021, 3:30pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126 "2021-06-26T15:30:36Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Core\_Sec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/core_sec/32/78091_2.png) [@Core\_Sec](https://discuss.elastic.co/u/Core_Sec)\
**Post date:** [June 26, 2021, 3:30pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/1 "2021-06-26T15:30:36Z")

</div>

Hi everyone

I've established ELK with [GitHub - deviantony/docker-elk: The Elastic stack (ELK) powered by Docker and Compose.](https://github.com/deviantony/docker-elk)  
I have log with timestamp included -

```auto
//{"type":"CustoumLogType","Param1":1,"Param2":1,"Param3":1,"logging_time":"2021-06-25 20:10:10"}

```

My problem is that the logs are indexed with @timestamp of index time and not the indexed timestamp.

I've tried to replace @timestamp with logging\_time from my log following this example -

> **[Timestamp | Filebeat Reference \[7.13\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html)**

The config of the whole ELK is default like the docker, the filebeat config is as follows -

```auto

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /logs/*_log.json
  json.keys_under_root: true  
  json.add_error_key: true

- type: filestream
  enabled: true
  paths:
    - /logs/*_log.json
  json.keys_under_root: true
  json.add_error_key: true

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

setup.kibana:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  username: "elastic"
  password: "changeme"

processors:
  #https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html
  - timestamp:
      field: logging_time
      target_field: @timestamp
      layouts:
        - UNIX_MS
      test: 
        - '1624523887321814'
  - add_host_metadata:
      when.not.contains.tags: forwarded

```

No matter what i tried, ES ignored logging\_time time and indexed it as text field and not as time.

I want to build kibana histograms using that time.

Really would for help with this!  
Thanks

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 26, 2021, 6:19pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/2 "2021-06-26T18:19:15Z")

</div>

Hi @Core_Sec Welcome to the community

From the docs [here](https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html)

> The `timestamp` processor parses a timestamp from a field. By default the timestamp processor writes the parsed result to the `@timestamp` field. You can specify a different field by setting the `target_field` parameter. The timestamp value is parsed according to the `layouts` parameter. Multiple layouts can be specified and they will be used sequentially to attempt parsing the timestamp field.
> 
> The timestamp layouts used by this processor are different than the formats supported by date processors in Logstash and Elasticsearch Ingest Node.  
> **SUPER IMPORTANT**  
> **The `layouts` are described using a reference time that is based on this specific time:**
> 
> **Mon Jan 2 15:04:05 MST 2006**
> 
> Since MST is GMT-0700, the reference time is:
> 
> 01/02 03:04:05PM '06 -0700. _\<\<!--- (NOTE see the pattern 1234567)_
> 
> To define your own layout, rewrite the reference time in a format that matches the timestamps you expect to parse. For more layout examples and details see the [Go time package documentation](https://godoc.org/time#pkg-constants).

Your example time `2021-06-25 20:10:10` is certainly not `UNIX_MS` so your format needs to expressed in terms of the reference time.

So your time

`2021-06-25 20:10:10`

Would be expressed as

`2006-01-02 15:04:05`

```auto
processors:
  #https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html
  - timestamp:
      field: logging_time
      target_field: @timestamp
      layouts:
        - '2006-01-02 15:04:05' <!-- Should be the correct layout assuming your time is UTC
      test: 
        - '2021-06-25 20:10:10'
  - add_host_metadata:
      when.not.contains.tags: forwarded. <!--- Make sure this is a good condition or this will never execute

```

Give it a try

---

<div class="post-metadata">

**Author:** ![Core\_Sec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/core_sec/32/78091_2.png) [@Core\_Sec](https://discuss.elastic.co/u/Core_Sec)\
**Post date:** [June 26, 2021, 7:06pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/3 "2021-06-26T19:06:14Z")

</div>

> [@stephenb](#):
>
> `target_field`

Thanks for you answer and welcoming!

I accidentally uploaded the config after some tries, originally used the layout that you've said.  
I've removed the `add_host_metadata`, my time is UTC indeed.

It still doesn't work, i', getting timestamp to be the indexing time and the logging\_time is just a text field:

```auto
@timestamp
Jun 26, 2021 @ 22:02:18.895

logging_time
2021-06-25 19:15:60

processors:
  #https://www.elastic.co/guide/en/beats/filebeat/current/processor-timestamp.html
  - timestamp:
      field: logging_time
      layouts:
        - '2006-01-02 15:04:05'
      test: 
        - '2021-06-25 22:22:30'

```

Any idead?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 26, 2021, 7:23pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/4 "2021-06-26T19:23:58Z")

</div>

If If you run filebeat with` -d "*"` you'll get a bunch of debug.

Plus can you provide a couple actual sample lines of your logs.

And a couple result documents that are in filebeat.

Plus can you show the actual input section of your filebeat.yml

And what version are you using

---

<div class="post-metadata">

**Author:** ![Core\_Sec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/core_sec/32/78091_2.png) [@Core\_Sec](https://discuss.elastic.co/u/Core_Sec)\
**Post date:** [June 26, 2021, 8:05pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/5 "2021-06-26T20:05:03Z")

</div>

Hi,

I'm getting lot's of debug data? what's relevant?

Couple of sample lines from the log:

```auto
{"type":"LogType1","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:23:30"}
{"type":"LogType2","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:24:30"}
{"type":"LogType3","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:25:30"}

```

Documents from ES:

```auto
{
  "_index": "filebeat-7.13.2-2021.06.26-000001",
  "_type": "_doc",
  "_id": "iFKzSXoBEvvJtRoZhrBK",
  "_version": 1,
  "_score": null,
  "fields": {
    "logging_time": [
      "2021-06-25 19:15:60"
    ],
    "channel": [
      3
    ],
    "input.type": [
      "log"
    ],
    "log.offset": [
      510
    ],
    "type": [
      "LogType1"
    ],
    "agent.hostname": [
      "WorkPC.local"
    ],
    "agent.type": [
      "filebeat"
    ],
    "@timestamp": [
      "2021-06-26T19:02:18.895Z"
    ],
    "agent.id": [
      "b5197535-ffd0-48ce-b54c-5eb7d9bd7c19"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "log.file.path": [
      "/logs/9_log.json"
    ],
    "agent.ephemeral_id": [
      "5fa1591e-6195-4ae3-846d-7c5bc1834565"
    ],
    "block": [
      2
    ],
    "agent.version": [
      "7.13.2"
    ],
    "agent.name": [
      "WorkPC.local"
    ],
    "page": [
      6
    ],
    "host.name": [
      "WorkPC.local"
    ]
  },
  "sort": [
    1624734138895
  ]
}

```

Input section:

```auto
filebeat.inputs:

- type: log
  enabled: true

  paths:
    - /logs/*_log.json
  json.keys_under_root: true  
  json.add_error_key: true
- type: filestream

  enabled: true

  paths:
    - /logs/*_log.json
  json.keys_under_root: true
  json.add_error_key: true

```

Using Filebeat `filebeat version 7.13.2 (amd64), libbeat 7.13.2 [686ba416a74193f2e69dcfa2eb142f4364a79307 built 2021-06-10 21:04:13 +0000 UTC] `

Using ES 7.13.2

Thanks!

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [June 27, 2021, 1:28am UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/6 "2021-06-27T01:28:55Z")

</div>

First you have both the log and filestream enabled and pointed to the same logs, probably don't want that.

my log file (yours)

```auto
{"type":"LogType1","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:23:30"}
{"type":"LogType2","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:24:30"}
{"type":"LogType3","channel":2,"block":1,"page":3,"logging_time":"2021-06-25 19:25:30"}

```

My minimal filebeat.yml

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log
  json.keys_under_root: true  
  json.add_error_key: true  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
  hosts: ["localhost:9200"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - timestamp:
      field: logging_time
      layouts:
        - '2006-01-02 15:04:05'
      test: 
        - '2021-06-25 20:10:10'

```

my command

`./filebeat -e -c ./filebeat-minimum.yml`

My results Kibana - Dev Tools

```auto
GET filebeat-7.13.2-2021.06.27-000001/_search
{
  "_source": false,
  "fields": ["*"]
}

```

results  
Note : the `@timestamp` is set to `logging_time`

```auto
{
  "took" : 43,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.13.2-2021.06.27-000001",
        "_type" : "_doc",
        "_id" : "eF0KS3oBgW1UHy0ALsPE",
        "_score" : 1.0,
        "fields" : {
          "logging_time" : [
            "2021-06-25 19:23:30" <!---- logging_time
          ],
          "host.os.name.text" : [
            "Mac OS X"
          ],
          "channel" : [
            2
          ],
          "host.hostname" : [
            "ceres"
          ],
          "type" : [
            "LogType1"
          ],
          "host.mac" : [
            "8c:85:90:ae:b0:b2",
            "82:de:c3:e6:d4:05",
            "82:de:c3:e6:d4:04",
            "82:de:c3:e6:d4:01",
            "82:de:c3:e6:d4:00",
            "82:de:c3:e6:d4:01",
            "0e:85:90:ae:b0:b2",
            "d6:54:f2:7a:9c:4c",
            "d6:54:f2:7a:9c:4c",
            "ac:de:48:00:11:22"
          ],
          "host.os.build" : [
            "20F71"
          ],
          "host.ip" : [
            "fe80::84d:3bc2:8250:27e0",
            "192.168.2.205",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::377:1386:b3db:403e",
            "fe80::fa7e:d5f1:2ea7:76ee",
            "fe80::aede:48ff:fe00:1122"
          ],
          "agent.type" : [
            "filebeat"
          ],
          "host.os.version" : [
            "10.16"
          ],
          "host.os.kernel" : [
            "20.5.0"
          ],
          "host.os.name" : [
            "Mac OS X"
          ],
          "block" : [
            1
          ],
          "agent.name" : [
            "ceres"
          ],
          "host.name" : [
            "ceres"
          ],
          "host.id" : [
            "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          ],
          "host.os.type" : [
            "macos"
          ],
          "input.type" : [
            "log"
          ],
          "log.offset" : [
            0
          ],
          "agent.hostname" : [
            "ceres"
          ],
          "host.architecture" : [
            "x86_64"
          ],
          "@timestamp" : [
            "2021-06-25T19:23:30.000Z" <!--- Set to logging_time
          ],
          "agent.id" : [
            "1d5a1f47-65b3-4d65-bcaf-ad2f378e846c"
          ],
          "host.os.platform" : [
            "darwin"
          ],
          "ecs.version" : [
            "1.8.0"
          ],
          "log.file.path" : [
            "/Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log"
          ],
          "agent.ephemeral_id" : [
            "02a798bf-8b33-4341-b449-19068cde55d9"
          ],
          "agent.version" : [
            "7.13.2"
          ],
          "page" : [
            3
          ],
          "host.os.family" : [
            "darwin"
          ]
        }
      },
      {
        "_index" : "filebeat-7.13.2-2021.06.27-000001",
        "_type" : "_doc",
        "_id" : "eV0KS3oBgW1UHy0ALsPE",
        "_score" : 1.0,
        "fields" : {
          "logging_time" : [
            "2021-06-25 19:24:30"
          ],
          "host.os.name.text" : [
            "Mac OS X"
          ],
          "channel" : [
            2
          ],
          "host.hostname" : [
            "ceres"
          ],
          "type" : [
            "LogType2"
          ],
          "host.mac" : [
            "8c:85:90:ae:b0:b2",
            "82:de:c3:e6:d4:05",
            "82:de:c3:e6:d4:04",
            "82:de:c3:e6:d4:01",
            "82:de:c3:e6:d4:00",
            "82:de:c3:e6:d4:01",
            "0e:85:90:ae:b0:b2",
            "d6:54:f2:7a:9c:4c",
            "d6:54:f2:7a:9c:4c",
            "ac:de:48:00:11:22"
          ],
          "host.os.build" : [
            "20F71"
          ],
          "host.ip" : [
            "fe80::84d:3bc2:8250:27e0",
            "192.168.2.205",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::377:1386:b3db:403e",
            "fe80::fa7e:d5f1:2ea7:76ee",
            "fe80::aede:48ff:fe00:1122"
          ],
          "agent.type" : [
            "filebeat"
          ],
          "host.os.version" : [
            "10.16"
          ],
          "host.os.kernel" : [
            "20.5.0"
          ],
          "host.os.name" : [
            "Mac OS X"
          ],
          "block" : [
            1
          ],
          "agent.name" : [
            "ceres"
          ],
          "host.name" : [
            "ceres"
          ],
          "host.id" : [
            "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          ],
          "host.os.type" : [
            "macos"
          ],
          "input.type" : [
            "log"
          ],
          "log.offset" : [
            88
          ],
          "agent.hostname" : [
            "ceres"
          ],
          "host.architecture" : [
            "x86_64"
          ],
          "@timestamp" : [
            "2021-06-25T19:24:30.000Z"
          ],
          "agent.id" : [
            "1d5a1f47-65b3-4d65-bcaf-ad2f378e846c"
          ],
          "host.os.platform" : [
            "darwin"
          ],
          "ecs.version" : [
            "1.8.0"
          ],
          "log.file.path" : [
            "/Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log"
          ],
          "agent.ephemeral_id" : [
            "02a798bf-8b33-4341-b449-19068cde55d9"
          ],
          "agent.version" : [
            "7.13.2"
          ],
          "page" : [
            3
          ],
          "host.os.family" : [
            "darwin"
          ]
        }
      },
      {
        "_index" : "filebeat-7.13.2-2021.06.27-000001",
        "_type" : "_doc",
        "_id" : "el0KS3oBgW1UHy0ALsPE",
        "_score" : 1.0,
        "fields" : {
          "logging_time" : [
            "2021-06-25 19:25:30"
          ],
          "host.os.name.text" : [
            "Mac OS X"
          ],
          "channel" : [
            2
          ],
          "host.hostname" : [
            "ceres"
          ],
          "type" : [
            "LogType3"
          ],
          "host.mac" : [
            "8c:85:90:ae:b0:b2",
            "82:de:c3:e6:d4:05",
            "82:de:c3:e6:d4:04",
            "82:de:c3:e6:d4:01",
            "82:de:c3:e6:d4:00",
            "82:de:c3:e6:d4:01",
            "0e:85:90:ae:b0:b2",
            "d6:54:f2:7a:9c:4c",
            "d6:54:f2:7a:9c:4c",
            "ac:de:48:00:11:22"
          ],
          "host.os.build" : [
            "20F71"
          ],
          "host.ip" : [
            "fe80::84d:3bc2:8250:27e0",
            "192.168.2.205",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::d454:f2ff:fe7a:9c4c",
            "fe80::377:1386:b3db:403e",
            "fe80::fa7e:d5f1:2ea7:76ee",
            "fe80::aede:48ff:fe00:1122"
          ],
          "agent.type" : [
            "filebeat"
          ],
          "host.os.version" : [
            "10.16"
          ],
          "host.os.kernel" : [
            "20.5.0"
          ],
          "host.os.name" : [
            "Mac OS X"
          ],
          "block" : [
            1
          ],
          "agent.name" : [
            "ceres"
          ],
          "host.name" : [
            "ceres"
          ],
          "host.id" : [
            "CB562E90-69DE-5D41-AC64-4EEDC79D5CB0"
          ],
          "host.os.type" : [
            "macos"
          ],
          "input.type" : [
            "log"
          ],
          "log.offset" : [
            176
          ],
          "agent.hostname" : [
            "ceres"
          ],
          "host.architecture" : [
            "x86_64"
          ],
          "@timestamp" : [
            "2021-06-25T19:25:30.000Z"
          ],
          "agent.id" : [
            "1d5a1f47-65b3-4d65-bcaf-ad2f378e846c"
          ],
          "host.os.platform" : [
            "darwin"
          ],
          "ecs.version" : [
            "1.8.0"
          ],
          "log.file.path" : [
            "/Users/sbrown/workspace/elastic-install/7.13.2/filebeat-7.13.2-darwin-x86_64/test_json.log"
          ],
          "agent.ephemeral_id" : [
            "02a798bf-8b33-4341-b449-19068cde55d9"
          ],
          "agent.version" : [
            "7.13.2"
          ],
          "page" : [
            3
          ],
          "host.os.family" : [
            "darwin"
          ]
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![Core\_Sec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/core_sec/32/78091_2.png) [@Core\_Sec](https://discuss.elastic.co/u/Core_Sec)\
**Post date:** [June 27, 2021, 6:35pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/7 "2021-06-27T18:35:27Z")

</div>

Awesome! It worked!  
Thanks for the help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2021, 8:35pm UTC](https://discuss.elastic.co/t/help-with-indexing-log-with-custom-time-instead-of-timestamp/277126/8 "2021-07-25T20:35:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.