Can't set my log file timestamp as Time Filter in Kibana

Elastic Stack Kibana
1

I am trying to process my log file in logstash using the following configuration:

Example of log file content:

2023-07-15T07:32:01,645 ERROR [00000003] :01234567891011 - ERROR: Some error message
2023-07-15T07:32:01,648 INFO  [00000006] :01234567891011 - 1 Some INFO message

grok filter used:

 grok {
      match => { "message" => [
                                "%{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:loglevel}  %{NOTSPACE} :%{USERNAME:user} - %{GREEDYDATA:msg}",
                                "%{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:loglevel} %{NOTSPACE} :%{USERNAME:user} - %{GREEDYDATA:msg}"
                            ]}}

Part of mapping related to the timestamp:

  "mappings": { 
       "properties": {
            "my_timestamp": {
                "type": "date",
                "ignore_malformed":true 
            }

Problem is when I want to create my index pattern in kibana, it suggets the @timestamp and my_timestamp as possible time fileters by when I set it to my_timestamp it does not work (No result at all), whereas when I use the @timestamp I can see all my data and see that the field my_timestamp was well read in ISO8601 but I can't use it)

Anyone had similar issue before ?

2

Hi @younes-gr Welcome to the community

Can you share the mapping for the index?

GET my-index-name

And can you show a couple sample docs

GET my-index/_search
{
  "fields": ["*"],
  "size": 2
}

Can you show a screen shot of setting up the index pattern?

3

Hello @stephenb, thanks for your feedback.
Here is the mapping file:

{"test_log":
  {"aliases":{},
  "mappings":{
    "properties":{
      "@timestamp":{"type":"date"},
      "@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "loglevel":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "message":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "msg":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "path":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "my_timestamp":{"type":"date","ignore_malformed":true},
      "user":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}
    }
  },
  "settings":{
    "index":{
      "creation_date":"1689673689010",
      "number_of_shards":"1",
      "number_of_replicas":"1",
      "uuid":"f5a8C8654asfcrFFEfvc5f",
      "version":{"created":"7010199"},
      "provided_name":"test_log"
      }
    }
  }
}

Sample docs:

{
  "took" : 23,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test_log",
        "_type" : "_doc",
        "_id" : "gbeb8zdfefb7eefv57efef7",
        "_score" : 1.0,
        "_ignored" : [
          "my_timestamp"
        ],
        "_source" : {
          "@version" : "1",
          "msg" : "ERROR: Some error message",
          "loglevel" : "ERROR",
          "user" : "0123456789101112",
          "host" : "my_ip_adress",
          "@timestamp" : "2023-07-18T09:50:47.453Z",
          "path" : "/home/sas_log_file.log",
          "message" : "2023-07-15T07:32:01,645 ERROR  [00000870] :0123456789101112 - ERROR: Some error message",
          "my_timestamp" : "2023-07-15T07:32:01,645"
        },
        "fields" : {
          "msg" : [
            "ERROR: Some error message"
          ],
          "loglevel.keyword" : [
            "ERROR"
          ],
          "@version.keyword" : [
            "1"
          ],
          "message" : [
            "2023-07-15T07:32:01,645 ERROR  [00000870] :0123456789101112 - ERROR: Some error message"
          ],
          "user.keyword" : [
            "0123456789101112"
          ],
          "msg.keyword" : [
            "ERROR: Some error message"
          ],
          "my_timestamp" : [
            "2023-07-15T07:32:01,645Z"
          ],
          "path" : [
            "/home/sas_log_file.log"
          ],
          "@timestamp" : [
            "2023-07-18T09:50:47.453Z"
          ],
          "loglevel" : [
            "ERROR"
          ],
          "message.keyword" : [
            "2023-07-15T07:32:01,645 ERROR  [00000870] :0123456789101112 - ERROR: Some error message"
          ],
          "@version" : [
            "1"
          ],
          "host" : [
            "my_ip_adress"
          ],
          "host.keyword" : [
            "my_ip_adress"
          ],
          "user" : [
            "0123456789101112"
          ],
          "path.keyword" : [
            "/home/sas_log_file.log"
          ]
        }
      },
      {
        "_index" : "test_log",
        "_type" : "_doc",
        "_id" : "gk4zdfzE7eR55fdfzef7",
        "_score" : 1.0,
        "_ignored" : [
          "my_timestamp"
        ],
        "_source" : {
          "@version" : "1",
          "msg" : "1 Some INFO message",
          "loglevel" : "INFO",
          "user" : "0123456789101112",
          "host" : "my_ip_adress",
          "@timestamp" : "2023-07-18T09:50:47.499Z",
          "path" : "/home/sas_log_file.log",
          "message" : "2023-07-15T07:32:01,648 INFO  [00000870] :0123456789101112 - 1 Some INFO message",
          "my_timestamp" : "2023-07-15T07:32:01,648"
        },
        "fields" : {
          "msg" : [
            "1 Some INFO message"
          ],
          "loglevel.keyword" : [
            "INFO"
          ],
          "@version.keyword" : [
            "1"
          ],
          "message" : [
            "2023-07-15T07:32:01,648 INFO  [00000870] :0123456789101112 - 1 Some INFO message"
          ],
          "user.keyword" : [
            "0123456789101112"
          ],
          "msg.keyword" : [
            "1 Some INFO message"
          ],
          "my_timestamp" : [
            "2023-07-15T07:32:01,648Z"
          ],
          "path" : [
            "/home/sas_log_file.log"
          ],
          "@timestamp" : [
            "2023-07-18T09:50:47.499Z"
          ],
          "loglevel" : [
            "INFO"
          ],
          "message.keyword" : [
            "2023-07-15T07:32:01,648 INFO  [00000870] :0123456789101112 - 1 Some INFO message"
          ],
          "@version" : [
            "1"
          ],
          "host" : [
            "my_ip_adress"
          ],
          "host.keyword" : [
            "my_ip_adress"
          ],
          "user" : [
            "0123456789101112"
          ],
          "path.keyword" : [
            "/home/sas_log_file.log"
          ]
        }
      }
    ]
  }
}

Screen of the index pattern (Only date fields):

Untitled
Untitled1595×609 39.6 KB

Hope those elements can be helpful !

4

You mean that when you use my_timestamp as the time field for your index pattern you do not see anything in discover?

What time range are you using? Increasing the time range also does not show anything?

5

Exactly, no data is shown in Discover tab. (Already tried altering the time range, but no result).

6

Oh, just saw this:

Did you created the mapping before indexing any data?

7

Note , for the milliseconds.

I think you need to put the correct date format in your mapping. I think it's incorrect therefore it's being ignored every time. If you had put ignored malformed false, it would not ingest.

1 Like
8

Problem solved based on @stephenb response.
Indeed I needed to specify the date format
I added the following code in my logstash conf file:

     date {
        match => [ "my_timestamp", "yyyy-MM-dd'T'HH:mm:ss,SSS"]
        target => "my_timestamp"
    }
1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.