Dear All,
I have now again another logstash error in my configuration. I could add my index pattern ("filebeat-*") in KIbana, but I could not see any field and any data are showing.
elk_1 | ==> /var/log/logstash/logstash.log <==
elk_1 | {:timestamp=>"2016-11-15T13:30:57.383000+0000", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"2016-11-15T14:30:49.302+0100",
:exception=>"Invalid format: \"2016-11- 15T14:30:49.302+0100\"",
:config_parsers=>"MMM d HH:mm:ss,MMM dd HH:mm:ss", :config_locale=>"default=en_US", :level=>:warn}
.....
"type"]}>>], :response=>{"index"=>{"_index"=>"%[@metadata][filebeat]}-%{+YYYY.MM.dd", "_type"=>"%[@metadata] [type]", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception",
"reason"=>"Invalid index name [%[@metadata][filebeat]}-%{+YYYY.MM.dd],
must be lowercase", "index"=>"%[@metadata][filebeat]}-%{+YYYY.MM.dd"}}}, :level=>:warn}
filebeat 1.3.1 output look like here
2016/11/15 13:31:04.771680 publish.go:109: DBG Publish: {
"@timestamp": "2016-11-15T13:30:56.854Z",
"beat": {
"hostname": "computer.localhost.net",
"name": "computer.localhost.net"
},
"count": 1,
"fields": null,
"input_type": "log",
"message": "[#|2016-11- 15T14:30:49.302+0100|INFO|glassfish3.1.2|playermanagement.PlayerHelper|_ThreadID=218;_ThreadName=Thread-2;|Player logged out: user1|#]\n",
"offset": 1536352,
"source": "path/runtime/glassfish-3.1/glassfish/domains/domain1/logs/server.log",
"type": "log"
}
my updated config look like here..
input {
lumberjack {
port => 5000
type => "log"
ssl_certificate => "/etc/foo.crt"
ssl_key => "/etc/foo.key"
}
}
input {
beats {
port => 5044
ssl => false
}
}
filter {
if [type] == "log" {
grok {
match => { "message" => "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log_level}\|%{DATA:server_version}\|%{JAVACLASS:Class}\|%{DATA:thread}\|%{DATA:message_detail}\|\#\]" }
}
}
syslog_pri { }
date {
match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
sniffing => true
manage_template => false
index => "%[@metadata][beat]}-%{+YYYY.MM.dd"
document_type => "%[@metadata][type]"
}
}
Many thanks in Advance.
Best Regards,
Thomas