Timestamp problem created using dissect

Elastic Stack Logstash
1

I have this filter which works very well except for mucking up the date in dissection. (What's in the ellipsis below, ..., is too long and everything is working anyway.)

filter
{
  dissect
  {
    "message" => "%{acme.date} %{acme.time} CEF:%{acme.version}|...
  }
}

operating on data like this:

message: 2018-11-08 21:57:37,208 CEF:0|...

and it creates these fields:

acme.date: November 7th 2018, 17:00:00.000
acme.time: 21:57:37,208
acme.version: 0

I don't know where the , 17:00:00,000 comes from, but I don't want it and I would prefer that acme.date contain 2018-11-08 in the end. (acme.time and acme.version are perfect.)

2

This field, are you seeing this in Kibana or in raw elastic?

3

If you are using a date filter to parse the acme.date field, it will create a UTC timestamp. This will include time, and I suspect the fact that it is UTC is why you see it offset by a number of hours.

4

Kibana

5

I am not intentionally using or not using a date filter--the code is above. I'm new to this. I find the Elastic documentation on this point well above entry level.

6

Can you check what the raw value is in kibana rather than the index value. Go to the document and press "view single document" then press "JSON" - what is the value in the field now? Still the same or different?

7

I click on the JSON tab and see that it is apparently what I want,

...
"acme.date": "2018-11-08",
...

So, it's Kibana that displays it not the way I want. What do you suggest I do to my code to overcome that?

8

You just need to go to Management > Index Management > Select the index > find the field "acme.date" - what does it say the type is?

9

Wait, I don't know what I was seeing that prompted my last response, maybe latent data from over the intervening holiday. I redid it. Here's what I'm really seeing (the Table tab results are as I said, but JSON tab results are):

  "fields": {
    "@timestamp": [
      "2018-11-23T14:50:13.846Z"
    ],
    "acme.date": [
      "2018-11-08T00:00:00.000Z"
    ]
  },

As for your next instructions,

I did exactly that (several times) and the result is (unencouragingly) this page:

Index management
Update your Elasticsearch indices indivudually...             _ Include system indices
+----------------+
| Manage 1 index |   acme.date
+----------------+
No indices to show

Maybe I'm doing something wrong. When I select the index, I'm looking at

x   Name                      Health    Status    etc....
x   filebeat-2018.11.23       yellow    open      etc....

I played around with this, but could not get anything but "No indices to show" no matter what field I chose.

10

Sorry my mistake, I should have said Index Patterns not Index Management, the rest is correct!

11

If I try to follow that, clicking Index Patterns under Kibana, and entering acme.date for Filter in the search field, I see:

Name          Type      Format    Searchable    Aggregatable    Excluded
acme.date     date                x             x

Is this what we're after?

12

Yes, now change the format to date and use the format to change it to how you want:

image
image.png504×584 17.7 KB

13

Okay, I see that this works.

However, this must be done as a manual operation in Kibana. I don't want my customer to have to perform this adjustment. What can I do, by configuration in Filebeat or Logstash, to avoid that? I need what comes out in Kibana to be as turn-key as it can.

14

Then in logstash you set the field type to string/text rather than allowing it to be automatically picked up as a "date". Then Kibana will not change the results.

You can set the field type in the logstash config, except that if you try to change the field type now you will get an error because the field type is already set to date. So you would need to create a new index/delete old indexes (if the data is test and does not matter at the moment) and set the field type to string from the get go.

15

I'm in development; I can do anything I want (and can figure out how) to do.

Where do I set the type of this field seeing as I only create it in the dissect filter thus (see below) in the first place? (Filebeat sent it in as a subset of the message field originally. Without my filter, acme.date doesn't exist.) Is there additional syntax I can decorate this code with that will accomplish it?

filter
{
  dissect
  {
    mapping =>
    {
      "message" => "%{acme.date} %{acme.time} CEF:%{acme.version}|%{acme.device_vendor}|%{acme.device_product}|%{acme.device_version}|%{acme.device_event_class_id}|%{acme.name}|%{acme.severity}|%{acme.extensions}"
    }
  }
}
16

After the dissect, try using the convert:

mutate {
    convert => { "%{acme.date}" => "string" }
  }

If you try this now without deleting you will probably get a "cannot index because field already has type date" or something like that. Then delete the index and it should start to go in properly.

If that does not work, then you specify the field type in the elasticsearch template before indexing:

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

See the top example on that page, it has a field called "created_at" of type date, you could do the same but remove the HH:MM:SS and change it to the correct format you want.

17

To confirm, the first solution does not do the trick.

For the second I'm not certain, in the framework of deployment of my ELK stack container to production, how to emit the PUT template. I guess I could just curl it at Elasticsearch from the Dockerfile assuming Elasticsearch will be up to receive it which I doubt.

18

(Thank you both very much for your help!)

19

What happens with the first solution, do you just get errors indexing?

Yes I guess that would be the best solution, you will have to do some work after deployment I think to get the solution you want.

20

Sorry, I'm back. With the first solution, I get no change at all. However, there are some other features I'm going to try.

1 Like