Logstash : Failed parsing date from field", :field=>"timestamp"

[zoram (ThomasK)]

Dear All,

I have now again another logstash error in my configuration. I could add my index pattern ("filebeat-*") in KIbana, but I could not see any field and any data are showing.

 elk_1  | ==> /var/log/logstash/logstash.log <==
 elk_1  | {:timestamp=>"2016-11-15T13:30:57.383000+0000", :message=>"Failed parsing date from field",     :field=>"timestamp", :value=>"2016-11-15T14:30:49.302+0100", 
:exception=>"Invalid format: \"2016-11-    15T14:30:49.302+0100\"", 
:config_parsers=>"MMM d  HH:mm:ss,MMM dd HH:mm:ss", :config_locale=>"default=en_US", :level=>:warn}

.....

"type"]}>>], :response=>{"index"=>{"_index"=>"%[@metadata][filebeat]}-%{+YYYY.MM.dd", "_type"=>"%[@metadata]   [type]", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", 
"reason"=>"Invalid index name [%[@metadata][filebeat]}-%{+YYYY.MM.dd], 
must be lowercase", "index"=>"%[@metadata][filebeat]}-%{+YYYY.MM.dd"}}}, :level=>:warn}

filebeat 1.3.1 output look like here

2016/11/15 13:31:04.771680 publish.go:109: DBG  Publish: {
  "@timestamp": "2016-11-15T13:30:56.854Z",
 "beat": {
    "hostname": "computer.localhost.net",
   "name": "computer.localhost.net"
 },
"count": 1,
"fields": null,
"input_type": "log",
"message": "[#|2016-11-  15T14:30:49.302+0100|INFO|glassfish3.1.2|playermanagement.PlayerHelper|_ThreadID=218;_ThreadName=Thread-2;|Player logged out: user1|#]\n",
 "offset": 1536352,
"source": "path/runtime/glassfish-3.1/glassfish/domains/domain1/logs/server.log",
"type": "log"
}

my updated config look like here..

input {
lumberjack {
port => 5000
type => "log"
ssl_certificate => "/etc/foo.crt"
ssl_key => "/etc/foo.key"
  }
}
input {
beats {
port => 5044
ssl => false
 }
}
filter {
if [type] == "log" {
 grok {
  match => { "message" =>  "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log_level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
}
}
 syslog_pri { }
date {
  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
 }
output {
  elasticsearch {
   hosts => ["http://localhost:9200"]
   sniffing => true
    manage_template => false
    index => "%[@metadata][beat]}-%{+YYYY.MM.dd"
    document_type => "%[@metadata][type]"
  }
}

Many thanks in Advance.

Best Regards,

Thomas

[zoram (ThomasK)]

I found the error my self after reading the Documentation

index => "filebeat-%{+YYYY.MM.dd}"

I changed the index pattern name as above.