# Logstash does not created proper timestamp from field

**URL:** <https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955>\
**Category:** Logstash\
**Created:** [November 23, 2018, 1:55am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955 "2018-11-23T01:55:57Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 1:55am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/1 "2018-11-23T01:55:57Z")

</div>

Hello I have a rule where I am trying to convert the timestamp from the logfile into @timestamp. However since logstash 6.5.0 that is not working anymore which results that data not being imported in elasticsearch.

The rules look like this:

```
if [year] {
    mutate {
      add_field => {
        "timestamp_match" => "%{month} %{day} %{year} %{time} %{day_period}"
      }
      remove_field => ["month", "day", "year", "time", "day_period"]
    }

    mutate {
      convert => { "timestamp_match" => "string" }
    }

    date {
      match => [ "timestamp_match",
                 "MMM dd YYYY KK:mm:ss aa",
                 "MMM dd YYYY K:mm:ss aa" ]
      timezone => "UTC"
      target => "@timestamp"
    }

```

the result is looks like this: `"@timestamp" => 2018-11-22T19:16:23.000Z`

Any idea how to fix this issue?

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 8:13am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/2 "2018-11-23T08:13:47Z")

</div>

Does it look like this in Kibana or in raw elastic?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 10:35am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/3 "2018-11-23T10:35:32Z")

</div>

I cannot see any of the data with that timestamp format in kibana. I am assuming that it's not loaded into elasticsearch or kibana cannot show it since it has no valid timestamp.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 10:38am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/4 "2018-11-23T10:38:45Z")

</div>

So where are you seeing the result?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 10:40am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/5 "2018-11-23T10:40:42Z")

</div>

stdout and file output.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 10:43am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/6 "2018-11-23T10:43:39Z")

</div>

Can you output the timestamp\_match field also and see how that looks?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 10:45am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/7 "2018-11-23T10:45:56Z")

</div>

```auto
Nov 23 05:45:04 vps188864 logstash[28555]: "timestamp_match" => "Nov 23 2018 5:45:03 AM",
Nov 23 05:45:04 vps188864 logstash[28555]: "input" => {
Nov 23 05:45:04 vps188864 logstash[28555]: "type" => "log"
Nov 23 05:45:04 vps188864 logstash[28555]: },

```

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 10:48am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/8 "2018-11-23T10:48:25Z")

</div>

Sorry one more question, what do you want it to look like?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 10:49am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/9 "2018-11-23T10:49:06Z")

</div>

I don't care as long it is a valid timestamp which makes the data to be loaded into elasticsearch and will show up in kibana.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 10:58am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/10 "2018-11-23T10:58:10Z")

</div>

> [@zozo6015](#):
>
> date { match =\> ["timestamp\_match", "MMM dd YYYY KK:mm:ss aa", "MMM dd YYYY K:mm:ss aa"] timezone =\> "UTC" target =\> "@timestamp" }

Can you try the following:

```
date {
  match => [ "timestamp_match",
             "MMM dd YYYY hh:mm:ss aa",
             "MMM dd YYYY h:mm:ss aa" ]
  timezone => "UTC"
  target => "@timestamp"
}

```

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 11:01am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/11 "2018-11-23T11:01:31Z")

</div>

Also, your filter is currently working, so I don't understand why you don't like the converted timestamp?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 11:04am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/12 "2018-11-23T11:04:15Z")

</div>

It is not like I don't like it. It just prevents data loading up in elasticsearch.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 11:09am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/13 "2018-11-23T11:09:20Z")

</div>

I don't understand how, it is a correct and valid timestamp as far as I can see. Its the same as my timestamp. What errors do you get?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 11:09am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/14 "2018-11-23T11:09:52Z")

</div>

Not working. timestamp format is the same and I cannot see the data loaded up in kibana

`Nov 23 06:08:58 vps188864 logstash[6968]: "@timestamp" => 2018-11-23T06:08:50.000Z,`

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 11:11am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/15 "2018-11-23T11:11:15Z")

</div>

That is the trick no errors, The data is just does not show up in kibana. Only if I keep the timestamp generated by filebeat at the import time. It has few seconds of delay.

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 11:14am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/16 "2018-11-23T11:14:22Z")

</div>

Just as an information for the amount of missing data which is not loaded up in kibana [http://prntscr.com/llzmq3](http://prntscr.com/llzmq3)

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 11:19am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/17 "2018-11-23T11:19:21Z")

</div>

Can you try changing the target in the date filter to a new field to see?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 11:24am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/18 "2018-11-23T11:24:18Z")

</div>

That is what I was doing as a workaround.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 11:29am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/19 "2018-11-23T11:29:39Z")

</div>

Interesting, so I wonder if it doesn't like the timestamp field being overwritten.

Could you try a "remove field" on the timestamp field directly before your date plugin?

---

<div class="post-metadata">

**Author:** ![zozo6015](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zozo6015/32/12117_2.png) [@zozo6015](https://discuss.elastic.co/u/zozo6015)\
**Post date:** [November 23, 2018, 11:33am UTC](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955/20 "2018-11-23T11:33:40Z")

</div>

It does not like it.

```
#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:177:in `sprintf'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:68:in `event_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `block in multi_receive'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:38:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:373:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:372:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:324:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:286:in `block in start_workers'"]}
Nov 23 06:32:31 vps188864 logstash[7882]: [2018-11-23T06:32:31,472][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Nov 23 06:32:31 vps188864 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Nov 23 06:32:31 vps188864 systemd
```

[Next page](https://discuss.elastic.co/t/logstash-does-not-created-proper-timestamp-from-field/157955.md?page=2)