How to configure index pattern to use custom timestamp in Kibana

Elastic Stack Kibana
1

I'm using,

  • filebeat version 5.0
  • elasticsearch version 5.0
  • kibana version 5.0

I have a log file with below format

{"timestamp":"2016-11-10 06:06:11","severity":"INFO","service": "login request","trace":"1e2rexxxxxxxx","span": "1e2rexxxxxxxx","exportable":"false","pid": "10144","thread": "http-nio-9200-exec-2","class":"org.apache.http.wire","logData": "some data"}
{"timestamp":"2016-11-10 06:06:11","severity":"DEBUG","service": "order request","trace":"1e3rexxxxxxxx","span": "1e3rexxxxxxxx","exportable":"false","pid": "10144","thread": "http-nio-9200-exec-2","class":"org.apache.http.wire","logData": "some more data"}

And I have created the index as below using the curl command.

curl -XPUT http://10.44.2.48:9200/toastconnector -d '
{
"mappings" : {
"default" : {
"properties" : {
"timestamp": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" },
"severity": { "type": "string", "index": "not_analyzed" },
"service": { "type": "string", "index": "not_analyzed" },
"trace": { "type": "string", "index": "not_analyzed" },
"span": { "type": "string", "index": "not_analyzed" },
"exportable": { "type": "string", "index": "not_analyzed" },
"pid": { "type": "string", "index": "not_analyzed" },
"thread": { "type": "string", "index": "not_analyzed" },
"class": { "type": "string", "index": "not_analyzed" },
"logData": { "type": "string", "index": "not_analyzed" }
}
}
}
}
';

I have a requirement to filter/search any 'service' (e.g. login request) within a certain date and time frame and I want it to be configured in my dashboard as well. When I try to configure index pattern in Kibana with the check box 'Index contains time-based events' option enabled, I get only the default @timestamp field (which I assume shows the index created date time) on 'Time-field name', but not the timestamp field I have in my log file. How can I do configure to pick the timestamp in my log file?

2

I went through the steps as you outlined and was able to successfully add the timestamp field created in the mapping.

Can you view the mapping API and ensure you have the timestamp field?

http://10.44.2.48:9200/toastconnector

3

Thank you @tsmalley. Yes, It is there in the mapping (http://10.44.2.48:9200/toastconnector) and it is being indexed too (refer: below screenshot).

indexed.png990×418 18.6 KB

But my question is, can I configure my custom timestamp field to use in 'Index contains time-based events'? (Refer the below screenshot, it shows only the default '@timestamp' field)

config_index.png980×597 37.6 KB

4

@tsmalley I'm not sure whether you have noticed my question in the previous post, since it is in the middle of 2 inline images. So i'm posting it again.

5

Your 'timestamp' field is mapped as a string, which is why it is not available for selection as a date field in Kibana. If you want to extract the timestamp from the logs you will need to do this using either Logstash or the new ingest node functionality. As this will require you to change the mapping for that field you will need to reindex your data.

6

In my mapping i have mapped it to date as "timestamp": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }, Why it is showing as string in Kibana? Any help would be appreciated.

7

Has that template been applied? What do you get if you retrieve the actual mappings for the index?

8

Yes, following is the template I added using template API.

{
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true,
        "norms": {
          "enabled": false
        }
      },
      "dynamic_templates" : [{
		"strings_as_keyword" : {
			"mapping" : {
				"ignore_above" : 1024,
				"index" : "not_analyzed",
				"type" : "string"
			},
			"match_mapping_type" : "string"
			}
		}],
      "properties": {
        "timestamp": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" },
        "severity": { "type": "string", "index": "not_analyzed" },
        "service": { "type": "string", "index": "not_analyzed" },
        "trace": { "type": "string", "index": "not_analyzed" },
		"span": { "type": "string", "index": "not_analyzed" },
        "exportable": { "type": "string", "index": "not_analyzed" },
		"pid": { "type": "string", "index": "not_analyzed" },
		"thread": { "type": "string", "index": "not_analyzed" },
		"class": { "type": "string", "index": "not_analyzed" },
		"logData": { "type": "string", "index": "not_analyzed" }
      }
    }
  },
  "settings": {
    "index.refresh_interval": "5s"
  },
  "template": "toastconnector-*"
}

And when I retrieve the index using http://10.44.2.48:9200/toastconnector, I get the below response.

{
	"toastconnector" : {
		"aliases" : {},
		"mappings" : {
			"_default_" : {
				"properties" : {
					"class" : {
						"type" : "keyword"
					},
					"exportable" : {
						"type" : "keyword"
					},
					"logData" : {
						"type" : "keyword"
					},
					"pid" : {
						"type" : "keyword"
					},
					"service" : {
						"type" : "keyword"
					},
					"severity" : {
						"type" : "keyword"
					},
					"span" : {
						"type" : "keyword"
					},
					"thread" : {
						"type" : "keyword"
					},
					"timestamp" : {
						"type" : "date",
						"format" : "yyyy-MM-dd HH:mm:ss"
					},
					"trace" : {
						"type" : "keyword"
					}
				}
			}
		},
		"settings" : {
			"index" : {
				"creation_date" : "1479385319679",
				"number_of_shards" : "5",
				"number_of_replicas" : "1",
				"uuid" : "2zv0hizeRjqxn89nV-lkhg",
				"version" : {
					"created" : "5000099"
				},
				"provided_name" : "toastconnector"
			}
		}
	}
}
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.