Hello,
I encounter some problems with Elk suite, and try to solve it.
For a project, we are tryring to parses with logstash this kind of log:
`2018-07-16T11:11:53 coll_bddf [3] [0] [11] [61] [9] [24]`
so, with logstash, I've created the following conf file:
`
input {
file {
path => "/var/log/logstash/SOLR_LOGS/Solr_generate_1.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:solr_timestamp} %{WORD:solr_collection_name} [%{NUMBER:solr_global_time}] [%{NUMBER:solr_num_docs}] [%{NUMBER:solr_creation_time}] [%{NUMBER:solr_commit_time}] [%{NUMBER:solr_search_time}] [%{NUMBER:solr_delete_time}]" }
overwrite => ["message"]
}
}
output {
elasticsearch{
#hosts => ["192.168.186.120:9200"]
hosts => ["192.168.1.119:9200"]
index => "solrbddf-%{+YYYY.MM.dd}"
}
`
The next step, I've created the a template for my index like below:
PUT _template/solrcoll { "index_patterns":"solrbddf*", "version": 1, "settings": { "index.refresh_interval" : "5s" }, "mappings": { "doc": { "properties": { "solr_collection-name": { "type": "text" }, "solr_commit_time": { "type": "long" }, "solr_creation_time": { "type": "long" }, "solr_delete_time": { "type": "long" }, "solr_global_time": { "type": "long" }, "solr_num_docs": { "type": "long" }, "solr_search_time": { "type": "long" }, "solr_timestamp": { "type": "date", "format": "strict_date_hour_minute_second" } } } } }
And, for a curious reason, I'm not able to use all index value through Kibana.. but I can request them properly!
{ "took": 3, "timed_out": false, "_shards": { "total": 5, "successful": 5, "skipped": 0, "failed": 0 }, "hits": { "total": 2706, "max_score": 1, "hits": [ { "_index": "solrbddf-2018.07.16", "_type": "doc", "_id": "qggWomQBq8H4N0r_aVoh", "_score": 1, "_source": { "path": "/var/log/logstash/SOLR_LOGS/Solr_generate_1.log", "solr_global_time": "44", "solr_commit_time": "298", "solr_timestamp": "2018-07-13T10:20:55", "@version": "1", "host": "centos_solr_test", "@timestamp": "2018-07-16T07:55:33.890Z", "solr_delete_time": "137", "solr_creation_time": "138", "solr_collection_name": "coll_bddf", "solr_search_time": "54", "message": "2018-07-13T10:20:55 coll_bddf [44] [0] [138] [298] [54] [137]", "solr_num_docs": "0" } },
Could you help me to solve this issue?
Did I missed some things.?
Best Regards