# Timestamp problem created using dissect

**URL:** <https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782>\
**Category:** Logstash\
**Created:** [November 21, 2018, 10:06pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782 "2018-11-21T22:06:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 21, 2018, 10:06pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/1 "2018-11-21T22:06:51Z")

</div>

I have this filter which works very well except for mucking up the date in dissection. (What's in the ellipsis below, ..., is too long and everything is working anyway.)

```
filter
{
  dissect
  {
    "message" => "%{acme.date} %{acme.time} CEF:%{acme.version}|...
  }
}

```

operating on data like this:

`message: 2018-11-08 21:57:37,208 CEF:0|...`

and it creates these fields:

```
acme.date: November 7th 2018, 17:00:00.000
acme.time: 21:57:37,208
acme.version: 0

```

I don't know where the `, 17:00:00,000` comes from, but I don't want it and I would prefer that `acme.date` contain `2018-11-08` in the end. (`acme.time` and `acme.version` are perfect.)

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 22, 2018, 10:17am UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/2 "2018-11-22T10:17:19Z")

</div>

This field, are you seeing this in Kibana or in raw elastic?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 22, 2018, 11:04am UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/3 "2018-11-22T11:04:31Z")

</div>

If you are using a date filter to parse the `acme.date` field, it will create a UTC timestamp. This will include time, and I suspect the fact that it is UTC is why you see it offset by a number of hours.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 22, 2018, 6:28pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/4 "2018-11-22T18:28:29Z")

</div>

Kibana

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 22, 2018, 6:30pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/5 "2018-11-22T18:30:56Z")

</div>

I am not intentionally using or not using a date filter--the code is above. I'm new to this. I find the Elastic documentation on this point well above entry level.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 8:10am UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/6 "2018-11-23T08:10:29Z")

</div>

Can you check what the raw value is in kibana rather than the index value. Go to the document and press "view single document" then press "JSON" - what is the value in the field now? Still the same or different?

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 2:33pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/7 "2018-11-23T14:33:05Z")

</div>

I click on the JSON tab and see that it is apparently what I want,

> ...  
> "acme.date": "2018-11-08",  
> ...

So, it's Kibana that displays it not the way I want. What do you suggest I do to my code to overcome that?

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 2:35pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/8 "2018-11-23T14:35:22Z")

</div>

You just need to go to Management \> Index Management \> Select the index \> find the field "acme.date" - what does it say the type is?

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 3:16pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/9 "2018-11-23T15:16:09Z")

</div>

Wait, I don't know what I was seeing that prompted my last response, maybe latent data from over the intervening holiday. I redid it. Here's what I'm really seeing (the `Table` tab results are as I said, but `JSON` tab results are):

```
  "fields": {
    "@timestamp": [
      "2018-11-23T14:50:13.846Z"
    ],
    "acme.date": [
      "2018-11-08T00:00:00.000Z"
    ]
  },

```

As for your next instructions,

> [@Eniqmatic](#):
>
> Management \> Index Management \> Select the index \> find the field "acme.date" - what does it say the type is?

I did exactly that (several times) and the result is (unencouragingly) this page:

```
Index management
Update your Elasticsearch indices indivudually... _ Include system indices
+----------------+
| Manage 1 index | acme.date
+----------------+
No indices to show

```

Maybe I'm doing something wrong. When I select the index, I'm looking at

```
x Name Health Status etc....
x filebeat-2018.11.23 yellow open etc....

```

I played around with this, but could not get anything but "No indices to show" no matter what field I chose.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 3:21pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/10 "2018-11-23T15:21:20Z")

</div>

Sorry my mistake, I should have said Index Patterns not Index Management, the rest is correct!

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 3:32pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/11 "2018-11-23T15:32:43Z")

</div>

If I try to follow that, clicking `Index Patterns` under `Kibana`, and entering _acme.date_ for `Filter` in the search field, I see:

```
Name Type Format Searchable Aggregatable Excluded
acme.date date x x

```

Is this what we're after?

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 3:36pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/12 "2018-11-23T15:36:23Z")

</div>

Yes, now change the format to date and use the format to change it to how you want:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/a/ba50c1dfc9a63720ef98c4e511fb7ce9712c6c0f.png)

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 3:40pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/13 "2018-11-23T15:40:37Z")

</div>

Okay, I see that this works.

However, this must be done as a manual operation in Kibana. I don't want my customer to have to perform this adjustment. What can I do, by configuration in Filebeat or Logstash, to avoid that? I need what comes out in Kibana to be as turn-key as it can.

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 3:43pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/14 "2018-11-23T15:43:10Z")

</div>

Then in logstash you set the field type to string/text rather than allowing it to be automatically picked up as a "date". Then Kibana will not change the results.

You can set the field type in the logstash config, except that if you try to change the field type now you will get an error because the field type is already set to date. So you would need to create a new index/delete old indexes (if the data is test and does not matter at the moment) and set the field type to string from the get go.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 3:50pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/15 "2018-11-23T15:50:57Z")

</div>

I'm in development; I can do anything I want (and can figure out how) to do.

Where do I set the type of this field seeing as I only create it in the `dissect` filter thus (see below) in the first place? (Filebeat sent it in as a subset of the `message` field originally. Without my filter, `acme.date` doesn't exist.) Is there additional syntax I can decorate this code with that will accomplish it?

```
filter
{
  dissect
  {
    mapping =>
    {
      "message" => "%{acme.date} %{acme.time} CEF:%{acme.version}|%{acme.device_vendor}|%{acme.device_product}|%{acme.device_version}|%{acme.device_event_class_id}|%{acme.name}|%{acme.severity}|%{acme.extensions}"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 23, 2018, 3:55pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/16 "2018-11-23T15:55:52Z")

</div>

After the dissect, try using the convert:

```
mutate {
    convert => { "%{acme.date}" => "string" }
  }

```

If you try this now without deleting you will probably get a "cannot index because field already has type date" or something like that. Then delete the index and it should start to go in properly.

If that does not work, then you specify the field type in the elasticsearch template before indexing:

[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html)

See the top example on that page, it has a field called "created\_at" of type date, you could do the same but remove the HH:MM:SS and change it to the correct format you want.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 4:18pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/17 "2018-11-23T16:18:27Z")

</div>

To confirm, the first solution does not do the trick.

For the second I'm not certain, in the framework of deployment of my ELK stack container to production, how to emit the `PUT` template. I guess I could just `curl` it at Elasticsearch from the _Dockerfile_ assuming Elasticsearch will be up to receive it which I doubt.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 23, 2018, 4:56pm UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/18 "2018-11-23T16:56:39Z")

</div>

(Thank you both very much for your help!)

---

<div class="post-metadata">

**Author:** ![Eniqmatic](https://avatars.discourse-cdn.com/v4/letter/e/ea5d25/32.png) [@Eniqmatic](https://discuss.elastic.co/u/Eniqmatic)\
**Post date:** [November 26, 2018, 8:24am UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/19 "2018-11-26T08:24:12Z")

</div>

What happens with the first solution, do you just get errors indexing?

Yes I guess that would be the best solution, you will have to do some work after deployment I think to get the solution you want.

---

<div class="post-metadata">

**Author:** ![RussellBateman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russellbateman/32/34928_2.png) [@RussellBateman](https://discuss.elastic.co/u/RussellBateman)\
**Post date:** [November 27, 2018, 12:09am UTC](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782/20 "2018-11-27T00:09:02Z")

</div>

Sorry, I'm back. With the first solution, I get no change at all. However, there are some other features I'm going to try.

[Next page](https://discuss.elastic.co/t/timestamp-problem-created-using-dissect/157782.md?page=2)