Logstash спорадически перестаёт отправлять логи в elastic

Elastic In Your Native Tongue Вопросы на русском языке
1

Коллеги, доброго дня. Конфигурация: elasticsearch: 6.4.2, logstash: 6.6.1

Cluster health:
{
"cluster_name" : "elastic-kibana",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 606,
"active_shards" : 606,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 605,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0412881915772
}

Конфигурация логстеша:

input {
  file {
    start_position => "beginning"
    path => "/logs/log.json"
    type => json
  }
}
filter {
mutate {
rename => ["message","log_message"]
}
grok {
match => { log_message => "{\"timestamp\":\"%{TIMESTAMP_ISO8601:date}\",\"severity\":\"%{WORD:severity}\",\"order_id\":%{GREEDYDATA:order_id},\"data\":%{GREEDY
}
mutate {
remove_field => ["log_message"]
gsub => ["client", "\"", ""]
gsub => ["friendly_order_id", "\"", ""]
gsub => ["order_id", "\"", ""]
add_field => { "newtimestamp" => "%{date}"}
}
}

output {
        elasticsearch {
                index => "cluster-blue-%{+YYYY.MM.dd}"
                hosts => ["172.17.0.1:9200"]
        }

}

Проблема: периодически перестают парситься логи с логстеша в эластик, лечится рестартом логстеша. Один из моментов остановки наступает после выполнения сервисных задач в эластике, но не всегда, проблема носит спорадический характер:

[2019-10-18T01:55:00,000][INFO ][o.e.x.m.MlDailyMaintenanceService] triggering scheduled [ML] maintenance tasks
[2019-10-18T01:55:00,000][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [Beoh7RO] Deleting expired data
[2019-10-18T01:55:00,005][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [Beoh7RO] Completed deletion of expired data
[2019-10-18T01:55:00,005][INFO ][o.e.x.m.MlDailyMaintenanceService] Successfully completed [ML] maintenance tasks

На момент прекращения записи в эластик сам лог логстеша без ошибок:

[2019-10-17T22:00:28,605][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
[2019-10-17T22:00:28,657][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
[2019-10-17T22:00:28,662][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
[2019-10-17T22:00:28,666][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-10-17T22:00:28,667][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-10-17T22:00:28,705][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://elasticsearch:9200"]}
[2019-10-17T22:00:28,737][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x626548eb run>"}
[2019-10-17T22:00:28,742][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>[]}
[2019-10-17T22:00:29,073][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Подскажите, в чём может быть проблема ?

2

Во сколько прекратилась запись? В полночь?

3

После 2 часов ночи, после выполнения сервисных задач

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.