# Can't set my log file timestamp as Time Filter in Kibana

**URL:** <https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628>\
**Category:** Kibana\
**Created:** [July 30, 2023, 6:25pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628 "2023-07-30T18:25:31Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![younes-gr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/younes-gr/32/124087_2.png) [@younes-gr](https://discuss.elastic.co/u/younes-gr)\
**Post date:** [July 30, 2023, 6:25pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/1 "2023-07-30T18:25:31Z")

</div>

I am trying to process my log file in logstash using the following configuration:

**Example of log file content:**

```auto
2023-07-15T07:32:01,645 ERROR [00000003] :01234567891011 - ERROR: Some error message
2023-07-15T07:32:01,648 INFO [00000006] :01234567891011 - 1 Some INFO message

```

**grok filter used:**

```auto
 grok {
      match => { "message" => [
                                "%{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:loglevel} %{NOTSPACE} :%{USERNAME:user} - %{GREEDYDATA:msg}",
                                "%{TIMESTAMP_ISO8601:my_timestamp} %{LOGLEVEL:loglevel} %{NOTSPACE} :%{USERNAME:user} - %{GREEDYDATA:msg}"
                            ]}}

```

**Part of mapping related to the timestamp:**

```auto
  "mappings": { 
       "properties": {
            "my_timestamp": {
                "type": "date",
                "ignore_malformed":true 
            }

```

Problem is when I want to create my index pattern in kibana, it suggets the @timestamp and my\_timestamp as possible time fileters by when I set it to my\_timestamp it does not work (No result at all), whereas when I use the @timestamp I can see all my data and see that the field my\_timestamp was well read in ISO8601 but I can't use it)

Anyone had similar issue before ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 31, 2023, 2:10am UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/2 "2023-07-31T02:10:17Z")

</div>

Hi @younes-gr Welcome to the community

Can you share the mapping for the index?

`GET my-index-name`

And can you show a couple sample docs

```auto
GET my-index/_search
{
  "fields": ["*"],
  "size": 2
}

```

Can you show a screen shot of setting up the index pattern?

---

<div class="post-metadata">

**Author:** ![younes-gr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/younes-gr/32/124087_2.png) [@younes-gr](https://discuss.elastic.co/u/younes-gr)\
**Post date:** [July 31, 2023, 1:10pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/3 "2023-07-31T13:10:30Z")

</div>

Hello @stephenb, thanks for your feedback.  
Here is the mapping file:

```auto
{"test_log":
  {"aliases":{},
  "mappings":{
    "properties":{
      "@timestamp":{"type":"date"},
      "@version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "host":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "loglevel":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "message":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "msg":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "path":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},
      "my_timestamp":{"type":"date","ignore_malformed":true},
      "user":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}
    }
  },
  "settings":{
    "index":{
      "creation_date":"1689673689010",
      "number_of_shards":"1",
      "number_of_replicas":"1",
      "uuid":"f5a8C8654asfcrFFEfvc5f",
      "version":{"created":"7010199"},
      "provided_name":"test_log"
      }
    }
  }
}

```

Sample docs:

```auto
{
  "took" : 23,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test_log",
        "_type" : "_doc",
        "_id" : "gbeb8zdfefb7eefv57efef7",
        "_score" : 1.0,
        "_ignored" : [
          "my_timestamp"
        ],
        "_source" : {
          "@version" : "1",
          "msg" : "ERROR: Some error message",
          "loglevel" : "ERROR",
          "user" : "0123456789101112",
          "host" : "my_ip_adress",
          "@timestamp" : "2023-07-18T09:50:47.453Z",
          "path" : "/home/sas_log_file.log",
          "message" : "2023-07-15T07:32:01,645 ERROR [00000870] :0123456789101112 - ERROR: Some error message",
          "my_timestamp" : "2023-07-15T07:32:01,645"
        },
        "fields" : {
          "msg" : [
            "ERROR: Some error message"
          ],
          "loglevel.keyword" : [
            "ERROR"
          ],
          "@version.keyword" : [
            "1"
          ],
          "message" : [
            "2023-07-15T07:32:01,645 ERROR [00000870] :0123456789101112 - ERROR: Some error message"
          ],
          "user.keyword" : [
            "0123456789101112"
          ],
          "msg.keyword" : [
            "ERROR: Some error message"
          ],
          "my_timestamp" : [
            "2023-07-15T07:32:01,645Z"
          ],
          "path" : [
            "/home/sas_log_file.log"
          ],
          "@timestamp" : [
            "2023-07-18T09:50:47.453Z"
          ],
          "loglevel" : [
            "ERROR"
          ],
          "message.keyword" : [
            "2023-07-15T07:32:01,645 ERROR [00000870] :0123456789101112 - ERROR: Some error message"
          ],
          "@version" : [
            "1"
          ],
          "host" : [
            "my_ip_adress"
          ],
          "host.keyword" : [
            "my_ip_adress"
          ],
          "user" : [
            "0123456789101112"
          ],
          "path.keyword" : [
            "/home/sas_log_file.log"
          ]
        }
      },
      {
        "_index" : "test_log",
        "_type" : "_doc",
        "_id" : "gk4zdfzE7eR55fdfzef7",
        "_score" : 1.0,
        "_ignored" : [
          "my_timestamp"
        ],
        "_source" : {
          "@version" : "1",
          "msg" : "1 Some INFO message",
          "loglevel" : "INFO",
          "user" : "0123456789101112",
          "host" : "my_ip_adress",
          "@timestamp" : "2023-07-18T09:50:47.499Z",
          "path" : "/home/sas_log_file.log",
          "message" : "2023-07-15T07:32:01,648 INFO [00000870] :0123456789101112 - 1 Some INFO message",
          "my_timestamp" : "2023-07-15T07:32:01,648"
        },
        "fields" : {
          "msg" : [
            "1 Some INFO message"
          ],
          "loglevel.keyword" : [
            "INFO"
          ],
          "@version.keyword" : [
            "1"
          ],
          "message" : [
            "2023-07-15T07:32:01,648 INFO [00000870] :0123456789101112 - 1 Some INFO message"
          ],
          "user.keyword" : [
            "0123456789101112"
          ],
          "msg.keyword" : [
            "1 Some INFO message"
          ],
          "my_timestamp" : [
            "2023-07-15T07:32:01,648Z"
          ],
          "path" : [
            "/home/sas_log_file.log"
          ],
          "@timestamp" : [
            "2023-07-18T09:50:47.499Z"
          ],
          "loglevel" : [
            "INFO"
          ],
          "message.keyword" : [
            "2023-07-15T07:32:01,648 INFO [00000870] :0123456789101112 - 1 Some INFO message"
          ],
          "@version" : [
            "1"
          ],
          "host" : [
            "my_ip_adress"
          ],
          "host.keyword" : [
            "my_ip_adress"
          ],
          "user" : [
            "0123456789101112"
          ],
          "path.keyword" : [
            "/home/sas_log_file.log"
          ]
        }
      }
    ]
  }
}

```

Screen of the index pattern (Only date fields):

 ![Untitled](https://us1.discourse-cdn.com/elastic/original/3X/2/8/28e21e9df0218004f76db302ee88758a7ba8459e.png)

Hope those elements can be helpful !

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 31, 2023, 1:18pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/4 "2023-07-31T13:18:45Z")

</div>

> [@younes-gr](#):
>
> No result at all

You mean that when you use `my_timestamp` as the time field for your index pattern you do not see anything in discover?

What time range are you using? Increasing the time range also does not show anything?

---

<div class="post-metadata">

**Author:** ![younes-gr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/younes-gr/32/124087_2.png) [@younes-gr](https://discuss.elastic.co/u/younes-gr)\
**Post date:** [July 31, 2023, 1:46pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/5 "2023-07-31T13:46:46Z")

</div>

Exactly, no data is shown in Discover tab. (Already tried altering the time range, but no result).

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 31, 2023, 1:51pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/6 "2023-07-31T13:51:21Z")

</div>

Oh, just saw this:

> [@younes-gr](#):
>
> ```auto
> "_ignored" : [
> "my_timestamp"
> ]
> 
> ```

Did you created the mapping before indexing any data?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 31, 2023, 2:09pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/7 "2023-07-31T14:09:28Z")

</div>

> [@younes-gr](#):
>
> `my_timestamp":{"type":"date","ignore_malformed":true`

> [@younes-gr](#):
>
> `"my_timestamp" : "2023-07-15T07:32:01,645"`

Note `,` for the milliseconds.

I think you need to put the correct date format in your mapping. I think it's incorrect therefore it's being ignored every time. If you had put ignored malformed false, it would not ingest.

---

<div class="post-metadata">

**Author:** ![younes-gr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/younes-gr/32/124087_2.png) [@younes-gr](https://discuss.elastic.co/u/younes-gr)\
**Post date:** [August 1, 2023, 8:23pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/8 "2023-08-01T20:23:52Z")

</div>

Problem solved based on @stephenb response.  
Indeed I needed to specify the date format  
I added the following code in my logstash conf file:

```auto
     date {
        match => ["my_timestamp", "yyyy-MM-dd'T'HH:mm:ss,SSS"]
        target => "my_timestamp"
    }

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 29, 2023, 8:24pm UTC](https://discuss.elastic.co/t/cant-set-my-log-file-timestamp-as-time-filter-in-kibana/339628/9 "2023-08-29T20:24:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.