Not discovered logs by Kibana?

[yash_mangla (Yash M.)]

Hello Folks,

I'm facing an issue with kibana. It is not showing records on discover page.
this is my sample logs.

<44>date=2017-11-23 time=10:53:40 devname=abc devid=123a logid=0000000007 type=traffic subtype=forward level=warning vd=root srcip=61.178.64.9 srcport=61603 srcintf="wan1" dstip=182.74.220.156 dstport=445 dstintf="wan1" sessionid=66045 proto=6 action=deny policyid=0 policytype=policy dstcountry="India" srccountry="China" trandisp=noop service="SMB" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 msg="Denied by forward policy check (policy 0)"
<44>date=2017-11-23 time=10:53:39 devname=abc devid=123a logid=0000000007 type=traffic subtype=forward level=warning vd=root srcip=10.10.10.9 srcport=1783 srcintf="ODC" dstip=202.56.230.7 dstport=53 dstintf=unknown-0 sessionid=66006 proto=17 action=deny policyid=0 policytype=policy dstcountry="India" srccountry="Reserved" trandisp=noop service="DNS" duration=2 sentbyte=0 rcvdbyte=0 sentpkt=1 msg="reverse path check fail, drop"

this is logstash configuration file i'm using.
sample.conf

input {
	stdin {}
}
filter {
	grok {
		match => [ "message", "(<(?<ruleID>.*)>(?<msg>.*)|(?<msg>.*))" ]
	}
	mutate {
		gsub => [
		"msg", ":", ":",
		"msg", "^<[0-9][0-9][0-9]>", ""
		]
	}	
	kv {
		source => "msg"
		field_split =>  "\"\,\" "
		remove_field => ["msg"]
		add_field => {
			"hsdp_ts" => "%{date}T%{time}.000Z"
		}
	}
	date {
		match => ["hsdp_ts", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
		target => "@timestamp"
	}
	geoip {
        source => "srcip"
        target => "geoip"
        add_field => [ "[geoip][coordinates]","%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]","%{[geoip][latitude]}" ]
    }
	mutate {
		convert => {
			"policyid" => "integer"
			"sessionId" => "integer"
			"appid" => "integer"
			"proto" => "integer"
			"logid" => "integer"
			"sentpkt" => "integer"
			"sentbyte" => "integer"
			"rcvdpkt" => "integer"
			"rcvdbyte" => "integer"
			"tunnelid" => "integer"
		}
	}
}
output {
	stdout { codec => rubydebug }
	elasticsearch {
		hosts => ["localhost:9200"]
        #user => "elastic"
        #password =>"changeme"
        index => "test-%{+YYYY.MM.dd}"
		template => "C:\ELK\logstash-5.6.3\bin\test-template.json"
		template_name => "test-*"
	}
}

template used: test-template.json

{
  "template" : "test-*",
  "version" : 50001,
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "norms" : false},
      "dynamic_templates" : [ {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text",
            "norms" : false
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text", "norms" : false,
            "fields" : {
              "keyword" : { "type": "keyword", "ignore_above": 256 }
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date", "include_in_all": false },
        "@version": { "type": "keyword", "include_in_all": false },
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        },
        "location": { "type": "geo_point" }
      }
    }
  }
}

Logs are parsed properly. but i unable to see logs over discover page in kibana. i'm sharing my screen please point out to me in right direction. help will be appreciable.

Screen_1: index have proper fields with proper timestamp.

Screen_2: you can see we have some records in index.

Screen_3: Unable to see logs on discover page.

Thanks,
Yash

[timroes]

Hi Yash,

I think you might have hit some timezone problems. The events seem to be from "now" (in UTC time). If you are not sitting in UTC+0 time, those events might be "in the future" from your position (since actually Elasticsearch doesn't store the time zone information within the indexed time field). Since you selected "Last 7 days" the time filter will be set to 7 days ago to now, which mean your documents could be excluded.

Could you maybe use the "relative" mode in the timepicker, to set your range from "7 days ago" to "7 days in the future" and check whether that will cause the documents to show up?

If you know that your documents are from a specific timezone, i.e. you want your Kibana to behave like it's in UTC+0 and not your local timezone, you can change that in Management > Advanced Settings using the dateFormat:tz setting.

Cheers,
Tim

[yash_mangla (Yash M.)]

Hello @timroes,

I'm not sure may be you are right. Can you tell me how can i setup proper timezone in logstash configuration without going into kibana management settings. I don't want to do any changes in kibana because sometime it can break the things and this approach is not recommended.

So, I'm loooking for valid approach...

Is there any way where i can setup the timezone from the logstash end ?