zabbix-tn-datalake.log
[2024-12-06T00:44:49,489][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77520081][1261885] duration [2.4s], collections [1]/[2.7s], total [2.4s]/[20h], memory [8.1gb]->[6.5gb]/[31gb], all_pools {[young] [752mb]->[128mb]/[0b]}{[old] [7.2gb]->[6.4gb]/[31gb]}{[survivor] [129.6mb]->[77.7mb]/[0b]}
[2024-12-06T00:44:49,759][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77520081] overhead, spent [2.4s] collecting in the last [2.7s]
[2024-12-06T00:45:16,797][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77520108] overhead, spent [257ms] collecting in the last [1s]
[2024-12-06T01:15:51,775][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77521939] overhead, spent [259ms] collecting in the last [1s]
[2024-12-06T01:16:10,977][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77521958] overhead, spent [366ms] collecting in the last [1s]
[2024-12-06T01:16:14,990][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77521962] overhead, spent [313ms] collecting in the last [1s]
[2024-12-06T01:16:22,998][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77521970] overhead, spent [302ms] collecting in the last [1s]
[2024-12-06T01:25:20,006][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77522501][1262140] duration [5.1s], collections [1]/[5.7s], total [5.1s]/[20h], memory [8.2gb]->[6.8gb]/[31gb], all_pools {[young] [816mb]->[80mb]/[0b]}{[old] [7.3gb]->[6.6gb]/[31gb]}{[survivor] [141.2mb]->[105.3mb]/[0b]}
[2024-12-06T01:25:20,028][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77522501] overhead, spent [5.1s] collecting in the last [5.7s]
[2024-12-06T01:25:20,000][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1326913566}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [2385] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52218, profile=default}] took [5363ms] which is above the warn threshold of [5000ms]
[2024-12-06T01:25:19,995][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1326913761}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [2331] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52242, profile=default}] took [5363ms] which is above the warn threshold of [5000ms]
[2024-12-06T01:25:19,999][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1326913687}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [2385] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52202, profile=default}] took [5363ms] which is above the warn threshold of [5000ms]
[2024-12-06T01:25:20,005][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1326913454}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [2331] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [5363ms] which is above the warn threshold of [5000ms]
[2024-12-06T01:25:20,055][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1326913718}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [2385] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [5363ms] which is above the warn threshold of [5000ms]
[2024-12-06T01:25:20,004][WARN ][o.e.t.InboundHandler ] [mdata03] handling inbound transport message [InboundMessage{Header{2234}{7.13.2}{1326914125}{true}{false}{false}{false}{indices:data/read/search[phase/query]}}] took [5363ms] which is above the warn threshold of [5000ms]
[2024-12-06T01:25:49,077][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77522530] overhead, spent [298ms] collecting in the last [1s]
[2024-12-06T01:42:00,053][INFO ][o.e.x.m.MlDailyMaintenanceService] [mdata03] triggering scheduled [ML] maintenance tasks
[2024-12-06T01:42:00,160][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [mdata03] Deleting expired data
[2024-12-06T01:42:00,467][INFO ][o.e.x.m.j.r.UnusedStatsRemover] [mdata03] Successfully deleted [0] unused stats documents
[2024-12-06T01:42:00,467][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [mdata03] Completed deletion of expired ML data
[2024-12-06T01:42:00,469][INFO ][o.e.x.m.MlDailyMaintenanceService] [mdata03] Successfully completed [ML] maintenance task: triggerDeleteExpiredDataTask
[2024-12-06T01:55:40,194][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77524316][1262347] duration [1.9s], collections [1]/[2.6s], total [1.9s]/[20h], memory [7.7gb]->[6.6gb]/[31gb], all_pools {[young] [512mb]->[208mb]/[0b]}{[old] [7.1gb]->[6.4gb]/[31gb]}{[survivor] [102.9mb]->[78.3mb]/[0b]}
[2024-12-06T01:55:40,194][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77524316] overhead, spent [1.9s] collecting in the last [2.6s]
[2024-12-06T01:56:14,284][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77524350] overhead, spent [552ms] collecting in the last [1s]
[2024-12-06T01:56:52,408][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77524388] overhead, spent [314ms] collecting in the last [1s]
[2024-12-06T01:57:46,925][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77524442] overhead, spent [460ms] collecting in the last [1.3s]
[2024-12-06T02:57:58,744][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77528049] overhead, spent [588ms] collecting in the last [1s]
[2024-12-06T02:58:41,779][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77528092] overhead, spent [264ms] collecting in the last [1s]
[2024-12-06T03:00:02,092][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [.monitoring-es-7-2024.12.06] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0]
[2024-12-06T03:00:02,286][INFO ][o.e.c.r.a.AllocationService] [mdata03] updating number_of_replicas to [1] for indices [.monitoring-es-7-2024.12.06]
[2024-12-06T03:00:04,154][INFO ][o.e.c.r.a.AllocationService] [mdata03] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-7-2024.12.06][0]]]).
[2024-12-06T03:00:06,523][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [.monitoring-kibana-7-2024.12.06] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0]
[2024-12-06T03:00:06,527][INFO ][o.e.c.r.a.AllocationService] [mdata03] updating number_of_replicas to [1] for indices [.monitoring-kibana-7-2024.12.06]
[2024-12-06T03:00:07,210][INFO ][o.e.c.r.a.AllocationService] [mdata03] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-kibana-7-2024.12.06][0]]]).
[2024-12-06T03:02:13,924][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77528304] overhead, spent [273ms] collecting in the last [1s]
[2024-12-06T03:02:22,981][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77528311][1262563] duration [2.2s], collections [1]/[2.8s], total [2.2s]/[20h], memory [7.4gb]->[6.8gb]/[31gb], all_pools {[young] [80mb]->[16mb]/[0b]}{[old] [7.2gb]->[6.7gb]/[31gb]}{[survivor] [142.4mb]->[81.9mb]/[0b]}
[2024-12-06T03:02:22,982][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77528311] overhead, spent [2.2s] collecting in the last [2.8s]
[2024-12-06T03:57:26,805][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77531607][1262717] duration [3.5s], collections [2]/[4.2s], total [3.5s]/[20h], memory [23gb]->[6.6gb]/[31gb], all_pools {[young] [15.6gb]->[80mb]/[0b]}{[old] [7.2gb]->[6.5gb]/[31gb]}{[survivor] [105.9mb]->[74.3mb]/[0b]}
[2024-12-06T03:57:26,833][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77531607] overhead, spent [3.5s] collecting in the last [4.2s]
[2024-12-06T03:58:04,935][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77531645] overhead, spent [386ms] collecting in the last [1s]
[2024-12-06T04:00:00,361][INFO ][o.e.x.m.e.l.LocalExporter] [mdata03] cleaning up [2] old indices
[2024-12-06T04:00:00,749][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [.monitoring-es-7-2024.11.29/FCD9Y1-WRI-QJfYAmGmAfw] deleting index
[2024-12-06T04:00:00,752][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [.monitoring-kibana-7-2024.11.29/wUCZMwD0QO62iNplFaDHjQ] deleting index
[2024-12-06T04:23:42,612][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.11.05-001113] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_south]
[2024-12-06T04:30:00,234][INFO ][o.e.x.s.SnapshotRetentionTask] [mdata03] starting SLM retention snapshot cleanup task
[2024-12-06T04:30:00,325][INFO ][o.e.x.s.SnapshotRetentionTask] [mdata03] there are no repositories to fetch, SLM retention snapshot cleanup task complete
[2024-12-06T04:30:14,095][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.11.05-001113] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [zabbix_south]
[2024-12-06T04:30:14,156][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.11.05-001113] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [zabbix_south]
[2024-12-06T04:30:14,206][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [zabbix_south-7.12.1-2024.11.05-001113/WBtwarNjTn-0zX98SEc4hw] deleting index
[2024-12-06T04:40:14,164][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.05-000384] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [zabbix_east]
[2024-12-06T04:40:15,091][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [zabbix_ipbb-7.12.1-2024.12.06-000385] creating index, cause [rollover_index], templates [zabbix_ipbb-7.12.1], shards [1]/[1]
[2024-12-06T04:40:15,644][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.06-000385] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [zabbix_east]
[2024-12-06T04:40:16,157][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.05-000384] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [zabbix_east]
[2024-12-06T04:40:16,663][INFO ][o.e.c.m.MetadataMappingService] [mdata03] [zabbix_ipbb-7.12.1-2024.12.06-000385/NRMRckupQc6WY4WbKwFOzg] update_mapping [_doc]
[2024-12-06T04:40:17,070][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.06-000385] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [zabbix_east]
[2024-12-06T04:40:17,247][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.05-000384] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [zabbix_east]
[2024-12-06T04:40:17,251][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.05-000384] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [zabbix_east]
[2024-12-06T04:40:17,470][INFO ][o.e.c.r.a.AllocationService] [mdata03] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[zabbix_ipbb-7.12.1-2024.12.06-000385][0]]]).
[2024-12-06T04:40:17,522][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.06-000385] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [zabbix_east]
[2024-12-06T04:40:17,575][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_ipbb-7.12.1-2024.12.05-000384] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [zabbix_east]
[2024-12-06T04:57:55,991][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77535225][1262943] duration [1.8s], collections [1]/[2.6s], total [1.8s]/[20h], memory [7.8gb]->[6.7gb]/[31gb], all_pools {[young] [512mb]->[208mb]/[0b]}{[old] [7.2gb]->[6.4gb]/[31gb]}{[survivor] [92.2mb]->[74.2mb]/[0b]}
[2024-12-06T04:57:55,991][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535225] overhead, spent [1.8s] collecting in the last [2.6s]
[2024-12-06T04:58:15,587][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77535244][1262945] duration [701ms], collections [1]/[1.5s], total [701ms]/[20h], memory [21.6gb]->[6.5gb]/[31gb], all_pools {[young] [15gb]->[16mb]/[0b]}{[old] [6.4gb]->[6.4gb]/[31gb]}{[survivor] [95.6mb]->[99.8mb]/[0b]}
[2024-12-06T04:58:15,587][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535244] overhead, spent [701ms] collecting in the last [1.5s]
[2024-12-06T05:04:14,226][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535602] overhead, spent [350ms] collecting in the last [1.3s]
[2024-12-06T05:04:47,364][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535635] overhead, spent [598ms] collecting in the last [1.1s]
[2024-12-06T05:04:51,088][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77535638][1262969] duration [719ms], collections [1]/[1.7s], total [719ms]/[20.1h], memory [7.6gb]->[6.9gb]/[31gb], all_pools {[young] [880mb]->[224mb]/[0b]}{[old] [6.5gb]->[6.6gb]/[31gb]}{[survivor] [199.7mb]->[130.7mb]/[0b]}
[2024-12-06T05:04:51,088][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535638] overhead, spent [719ms] collecting in the last [1.7s]
[2024-12-06T05:05:54,590][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535699] overhead, spent [472ms] collecting in the last [1.1s]
[2024-12-06T05:06:43,081][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535747] overhead, spent [391ms] collecting in the last [1.1s]
[2024-12-06T05:07:19,592][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535783] overhead, spent [606ms] collecting in the last [1.2s]
[2024-12-06T05:10:21,589][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77535923][1262999] duration [1.4s], collections [1]/[2.2s], total [1.4s]/[20.1h], memory [22.9gb]->[6.8gb]/[31gb], all_pools {[young] [16.1gb]->[16mb]/[0b]}{[old] [6.7gb]->[6.7gb]/[31gb]}{[survivor] [56.1mb]->[128.7mb]/[0b]}
[2024-12-06T05:10:23,098][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77535923] overhead, spent [1.4s] collecting in the last [2.2s]
[2024-12-06T05:11:55,196][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536015] overhead, spent [301ms] collecting in the last [1s]
[2024-12-06T05:12:50,746][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536045] overhead, spent [568ms] collecting in the last [1.3s]
[2024-12-06T05:15:18,997][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536139] overhead, spent [320ms] collecting in the last [1s]
[2024-12-06T05:18:04,731][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448363348}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [12808ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,577][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448399820}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [14609ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,598][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448422494}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1658] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [14609ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,632][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448427994}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [14609ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,646][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448422682}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1658] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [14609ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,646][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448429500}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [14409ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,685][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448438622}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1046] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [13009ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,711][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448433700}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1047] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [13009ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:18:05,737][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1448438683}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1658] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [6606ms] which is above the warn threshold of [5000ms]
[2024-12-06T05:21:00,039][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77536204][1263048] duration [3s], collections [1]/[4s], total [3s]/[20.1h], memory [22.9gb]->[7.1gb]/[31gb], all_pools {[young] [15.9gb]->[64mb]/[0b]}{[old] [6.8gb]->[6.8gb]/[31gb]}{[survivor] [103.7mb]->[167.3mb]/[0b]}
[2024-12-06T05:21:00,342][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536204] overhead, spent [3s] collecting in the last [4s]
[2024-12-06T05:22:16,077][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536257] overhead, spent [594ms] collecting in the last [1s]
[2024-12-06T05:24:38,960][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77536399][1263073] duration [983ms], collections [1]/[1.3s], total [983ms]/[20.1h], memory [22.9gb]->[7.1gb]/[31gb], all_pools {[young] [15.9gb]->[16mb]/[0b]}{[old] [6.9gb]->[6.9gb]/[31gb]}{[survivor] [93.7mb]->[131.7mb]/[0b]}
[2024-12-06T05:24:39,122][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536399] overhead, spent [983ms] collecting in the last [1.3s]
[2024-12-06T05:25:24,200][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536444] overhead, spent [621ms] collecting in the last [1s]
[2024-12-06T05:27:09,613][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536524] overhead, spent [297ms] collecting in the last [1s]
[2024-12-06T05:30:10,355][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77536637][1263093] duration [1.3s], collections [1]/[1.7s], total [1.3s]/[20.1h], memory [22.9gb]->[7.1gb]/[31gb], all_pools {[young] [15.7gb]->[16mb]/[0b]}{[old] [7gb]->[7gb]/[31gb]}{[survivor] [134.6mb]->[153.4mb]/[0b]}
[2024-12-06T05:30:10,595][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536637] overhead, spent [1.3s] collecting in the last [1.7s]
[2024-12-06T05:30:12,618][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536639] overhead, spent [499ms] collecting in the last [1s]
[2024-12-06T05:31:03,031][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536689] overhead, spent [413ms] collecting in the last [1.4s]
[2024-12-06T05:31:05,032][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536691] overhead, spent [323ms] collecting in the last [1s]
[2024-12-06T05:31:34,436][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536720] overhead, spent [476ms] collecting in the last [1.3s]
[2024-12-06T05:33:07,538][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77536813] overhead, spent [437ms] collecting in the last [1s]
[2024-12-06T05:40:14,122][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.11.05-001116] from [{"phase":"warm","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_moscow]
[2024-12-06T05:40:14,543][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.11.05-001108] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_west]
[2024-12-06T05:46:28,220][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77537607][1263178] duration [3.1s], collections [1]/[3.9s], total [3.1s]/[20.1h], memory [8gb]->[6.5gb]/[31gb], all_pools {[young] [608mb]->[16mb]/[0b]}{[old] [7.2gb]->[6.4gb]/[31gb]}{[survivor] [191.7mb]->[98.4mb]/[0b]}
[2024-12-06T05:46:28,220][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77537607] overhead, spent [3.1s] collecting in the last [3.9s]
[2024-12-06T05:46:56,464][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77537635] overhead, spent [568ms] collecting in the last [1.2s]
[2024-12-06T05:46:57,464][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77537636] overhead, spent [289ms] collecting in the last [1s]
[2024-12-06T05:50:14,101][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.11.05-001116] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [zabbix_moscow]
[2024-12-06T05:50:14,170][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.11.05-001108] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [zabbix_west]
[2024-12-06T05:50:14,228][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.11.05-001116] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [zabbix_moscow]
[2024-12-06T05:50:14,313][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [zabbix_moscow-7.12.1-2024.11.05-001116/oFv-RtE_TGKyundFlqDILw] deleting index
[2024-12-06T05:50:15,492][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.11.05-001108] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [zabbix_west]
[2024-12-06T05:50:15,553][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [zabbix_west-7.12.1-2024.11.05-001108/AbivekW3TC-w7QrnS0MNMQ] deleting index
[2024-12-06T06:16:16,805][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77539394] overhead, spent [305ms] collecting in the last [1s]
[2024-12-06T06:16:38,820][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77539416][1263243] duration [917ms], collections [1]/[1s], total [917ms]/[20.1h], memory [22.9gb]->[7.2gb]/[31gb], all_pools {[young] [15.8gb]->[96mb]/[0b]}{[old] [6.9gb]->[6.9gb]/[31gb]}{[survivor] [231mb]->[255mb]/[0b]}
[2024-12-06T06:16:38,820][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77539416] overhead, spent [917ms] collecting in the last [1s]
[2024-12-06T06:16:44,507][WARN ][o.e.t.InboundHandler ] [mdata03] handling inbound transport message [InboundMessage{Header{1591}{7.13.2}{1466101771}{true}{false}{false}{false}{indices:data/read/search[phase/query]}}] took [5151ms] which is above the warn threshold of [5000ms]
[2024-12-06T06:16:44,511][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77539417][1263244] duration [5.1s], collections [1]/[5.6s], total [5.1s]/[20.1h], memory [7.2gb]->[7.3gb]/[31gb], all_pools {[young] [96mb]->[208mb]/[0b]}{[old] [6.9gb]->[7gb]/[31gb]}{[survivor] [255mb]->[83.4mb]/[0b]}
[2024-12-06T06:16:44,511][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77539417] overhead, spent [5.1s] collecting in the last [5.6s]
[2024-12-06T06:16:44,510][WARN ][o.e.t.InboundHandler ] [mdata03] handling inbound transport message [InboundMessage{Header{1591}{7.13.2}{1466101848}{true}{false}{false}{false}{indices:data/read/search[phase/query]}}] took [5151ms] which is above the warn threshold of [5000ms]
[2024-12-06T06:16:44,514][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1466101747}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52240, profile=default}] took [5151ms] which is above the warn threshold of [5000ms]
[2024-12-06T06:16:44,514][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1466101750}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52254, profile=default}] took [5151ms] which is above the warn threshold of [5000ms]
[2024-12-06T06:17:36,551][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77539469] overhead, spent [329ms] collecting in the last [1s]
[2024-12-06T06:19:46,782][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77539598][1263262] duration [1.2s], collections [1]/[2.1s], total [1.2s]/[20.1h], memory [7.8gb]->[7.1gb]/[31gb], all_pools {[young] [704mb]->[32mb]/[0b]}{[old] [7gb]->[7.1gb]/[31gb]}{[survivor] [122.8mb]->[68.7mb]/[0b]}
[2024-12-06T06:19:46,783][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77539598] overhead, spent [1.2s] collecting in the last [2.1s]
[2024-12-06T06:21:36,809][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77539708] overhead, spent [435ms] collecting in the last [1s]
[2024-12-06T06:32:03,234][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77540334] overhead, spent [460ms] collecting in the last [1s]
[2024-12-06T06:32:04,365][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77540335][1263290] duration [831ms], collections [1]/[1.1s], total [831ms]/[20.1h], memory [8gb]->[7.5gb]/[31gb], all_pools {[young] [656mb]->[176mb]/[0b]}{[old] [7.2gb]->[7.3gb]/[31gb]}{[survivor] [214.4mb]->[76mb]/[0b]}
[2024-12-06T06:32:04,365][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77540335] overhead, spent [831ms] collecting in the last [1.1s]
[2024-12-06T06:33:48,593][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77540437][1263298] duration [2.2s], collections [1]/[3.2s], total [2.2s]/[20.1h], memory [8gb]->[6.6gb]/[31gb], all_pools {[young] [608mb]->[0b]/[0b]}{[old] [7.3gb]->[6.5gb]/[31gb]}{[survivor] [121mb]->[90.3mb]/[0b]}
[2024-12-06T06:33:48,593][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77540437] overhead, spent [2.2s] collecting in the last [3.2s]
[2024-12-06T06:40:14,069][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.11.05-001119] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_center]
[2024-12-06T06:50:14,070][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.11.05-001119] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [zabbix_center]
[2024-12-06T06:50:14,122][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.11.05-001119] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [zabbix_center]
[2024-12-06T06:50:14,171][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [zabbix_center-7.12.1-2024.11.05-001119/dQn02xvzSqCh3CiJHTqfqw] deleting index
[2024-12-06T07:16:25,921][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77542992] overhead, spent [682ms] collecting in the last [1s]
[2024-12-06T07:16:27,924][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77542994] overhead, spent [449ms] collecting in the last [1s]
[2024-12-06T07:19:10,189][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77543156] overhead, spent [263ms] collecting in the last [1s]
[2024-12-06T07:29:25,602][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77543771] overhead, spent [374ms] collecting in the last [1s]
[2024-12-06T07:30:44,000][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77543848][1263521] duration [1.6s], collections [2]/[2.3s], total [1.6s]/[20.1h], memory [22.7gb]->[6.6gb]/[31gb], all_pools {[young] [15.2gb]->[96mb]/[0b]}{[old] [7.3gb]->[6.5gb]/[31gb]}{[survivor] [81.3mb]->[45.8mb]/[0b]}
[2024-12-06T07:30:44,000][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77543848] overhead, spent [1.6s] collecting in the last [2.3s]
[2024-12-06T07:31:12,006][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77543876][1263527] duration [763ms], collections [1]/[1s], total [763ms]/[20.1h], memory [23gb]->[6.9gb]/[31gb], all_pools {[young] [16.3gb]->[304mb]/[0b]}{[old] [6.5gb]->[6.5gb]/[31gb]}{[survivor] [86.4mb]->[85.7mb]/[0b]}
[2024-12-06T07:31:12,006][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77543876] overhead, spent [763ms] collecting in the last [1s]
[2024-12-06T07:31:56,086][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77543920] overhead, spent [538ms] collecting in the last [1s]
[2024-12-06T08:16:30,348][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546592] overhead, spent [522ms] collecting in the last [1s]
[2024-12-06T08:16:33,652][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77546593][1263632] duration [3.1s], collections [1]/[3.3s], total [3.1s]/[20.1h], memory [8.2gb]->[7.4gb]/[31gb], all_pools {[young] [928mb]->[176mb]/[0b]}{[old] [7.1gb]->[7.2gb]/[31gb]}{[survivor] [197.6mb]->[93.8mb]/[0b]}
[2024-12-06T08:16:33,652][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546593] overhead, spent [3.1s] collecting in the last [3.3s]
[2024-12-06T08:16:44,589][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546603] overhead, spent [1s] collecting in the last [1.9s]
[2024-12-06T08:17:14,613][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546633] overhead, spent [481ms] collecting in the last [1s]
[2024-12-06T08:17:15,613][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546634] overhead, spent [255ms] collecting in the last [1s]
[2024-12-06T08:17:20,115][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77546638][1263646] duration [971ms], collections [1]/[1.5s], total [971ms]/[20.1h], memory [10.4gb]->[7.5gb]/[31gb], all_pools {[young] [3.1gb]->[192mb]/[0b]}{[old] [7.2gb]->[7.2gb]/[31gb]}{[survivor] [60.3mb]->[39.6mb]/[0b]}
[2024-12-06T08:17:20,115][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546638] overhead, spent [971ms] collecting in the last [1.5s]
[2024-12-06T08:17:26,149][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546644] overhead, spent [353ms] collecting in the last [1s]
[2024-12-06T08:17:29,175][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77546647] overhead, spent [351ms] collecting in the last [1s]
[2024-12-06T08:27:49,022][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77547266][1263670] duration [1.1s], collections [1]/[1.4s], total [1.1s]/[20.1h], memory [8.2gb]->[6.6gb]/[31gb], all_pools {[young] [752mb]->[32mb]/[0b]}{[old] [7.3gb]->[6.5gb]/[31gb]}{[survivor] [166.8mb]->[100.6mb]/[0b]}
[2024-12-06T08:27:49,022][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77547266] overhead, spent [1.1s] collecting in the last [1.4s]
[2024-12-06T08:28:21,368][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77547298][1263672] duration [798ms], collections [1]/[1.3s], total [798ms]/[20.1h], memory [22.8gb]->[6.6gb]/[31gb], all_pools {[young] [16.2gb]->[16mb]/[0b]}{[old] [6.5gb]->[6.5gb]/[31gb]}{[survivor] [90.3mb]->[111.2mb]/[0b]}
[2024-12-06T08:28:21,368][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77547298] overhead, spent [798ms] collecting in the last [1.3s]
[2024-12-06T08:29:38,956][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77547375][1263678] duration [775ms], collections [1]/[1.5s], total [775ms]/[20.1h], memory [22.4gb]->[6.6gb]/[31gb], all_pools {[young] [15.8gb]->[0b]/[0b]}{[old] [6.5gb]->[6.5gb]/[31gb]}{[survivor] [109.8mb]->[147mb]/[0b]}
[2024-12-06T08:29:38,956][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77547375] overhead, spent [775ms] collecting in the last [1.5s]
[2024-12-06T09:20:14,092][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.05-001143] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [zabbix_south]
[2024-12-06T09:20:15,027][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [zabbix_south-7.12.1-2024.12.06-001144] creating index, cause [rollover_index], templates [zabbix_south-7.12.1], shards [1]/[1]
[2024-12-06T09:20:15,594][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.06-001144] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [zabbix_south]
[2024-12-06T09:20:15,857][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.05-001143] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [zabbix_south]
[2024-12-06T09:20:15,929][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.06-001144] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [zabbix_south]
[2024-12-06T09:20:16,252][INFO ][o.e.c.m.MetadataMappingService] [mdata03] [zabbix_south-7.12.1-2024.12.06-001144/1Ha_x5xdRWSwCJKMhevX0A] update_mapping [_doc]
[2024-12-06T09:20:16,448][INFO ][o.e.c.r.a.AllocationService] [mdata03] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[zabbix_south-7.12.1-2024.12.06-001144][0]]]).
[2024-12-06T09:20:16,638][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.05-001143] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [zabbix_south]
[2024-12-06T09:20:16,649][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.05-001143] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [zabbix_south]
[2024-12-06T09:20:16,911][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.06-001144] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [zabbix_south]
[2024-12-06T09:20:17,011][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_south-7.12.1-2024.12.05-001143] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [zabbix_south]
[2024-12-06T09:31:48,182][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77551101][1263897] duration [932ms], collections [1]/[1.5s], total [932ms]/[20.1h], memory [7.5gb]->[6.8gb]/[31gb], all_pools {[young] [48mb]->[192mb]/[0b]}{[old] [7.3gb]->[6.5gb]/[31gb]}{[survivor] [143.7mb]->[59.9mb]/[0b]}
[2024-12-06T09:31:48,185][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77551101] overhead, spent [932ms] collecting in the last [1.5s]
[2024-12-06T10:00:14,068][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.11.30-001141] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"warm","action":"complete","name":"complete"}] in policy [zabbix_moscow]
[2024-12-06T10:26:04,808][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77554337][1264088] duration [19.1s], collections [1]/[19.3s], total [19.1s]/[20.1h], memory [8.7gb]->[6.5gb]/[31gb], all_pools {[young] [1.2gb]->[16mb]/[0b]}{[old] [7.3gb]->[6.4gb]/[31gb]}{[survivor] [253.4mb]->[90.3mb]/[0b]}
[2024-12-06T10:26:04,808][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77554337] overhead, spent [19.1s] collecting in the last [19.3s]
[2024-12-06T10:27:27,164][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77554419][1264100] duration [1.2s], collections [1]/[1.3s], total [1.2s]/[20.1h], memory [10.5gb]->[6.5gb]/[31gb], all_pools {[young] [3.9gb]->[0b]/[0b]}{[old] [6.4gb]->[6.5gb]/[31gb]}{[survivor] [81.1mb]->[76.1mb]/[0b]}
[2024-12-06T10:27:27,164][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77554419] overhead, spent [1.2s] collecting in the last [1.3s]
[2024-12-06T10:30:14,091][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.05-001138] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [zabbix_west]
[2024-12-06T10:30:14,160][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.05-001146] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [zabbix_moscow]
[2024-12-06T10:30:14,545][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [zabbix_west-7.12.1-2024.12.06-001139] creating index, cause [rollover_index], templates [zabbix_west-7.12.1], shards [1]/[1]
[2024-12-06T10:30:15,033][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [zabbix_moscow-7.12.1-2024.12.06-001147] creating index, cause [rollover_index], templates [zabbix_moscow-7.12.1], shards [1]/[1]
[2024-12-06T10:30:15,470][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.06-001139] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [zabbix_west]
[2024-12-06T10:30:15,606][INFO ][o.e.c.m.MetadataMappingService] [mdata03] [zabbix_moscow-7.12.1-2024.12.06-001147/wVM_AbPJRTywiif_0D2XzQ] update_mapping [_doc]
[2024-12-06T10:30:15,801][INFO ][o.e.c.m.MetadataMappingService] [mdata03] [zabbix_west-7.12.1-2024.12.06-001139/omPivciXQ_Gfijvd6s3kLw] update_mapping [_doc]
[2024-12-06T10:30:15,978][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_east-7.12.1-2024.11.05-001072] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_east]
[2024-12-06T10:30:16,153][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.05-001138] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [zabbix_west]
[2024-12-06T10:30:16,205][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.06-001147] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [zabbix_moscow]
[2024-12-06T10:30:16,259][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.05-001146] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [zabbix_moscow]
[2024-12-06T10:30:16,430][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.06-001139] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [zabbix_west]
[2024-12-06T10:30:16,483][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.05-001138] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [zabbix_west]
[2024-12-06T10:30:16,487][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.05-001138] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [zabbix_west]
[2024-12-06T10:30:16,820][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.06-001147] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [zabbix_moscow]
[2024-12-06T10:30:16,883][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.05-001146] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [zabbix_moscow]
[2024-12-06T10:30:16,885][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.05-001146] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [zabbix_moscow]
[2024-12-06T10:30:16,941][INFO ][o.e.c.r.a.AllocationService] [mdata03] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[zabbix_west-7.12.1-2024.12.06-001139][0]]]).
[2024-12-06T10:30:17,106][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.06-001139] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [zabbix_west]
[2024-12-06T10:30:17,186][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_west-7.12.1-2024.12.05-001138] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [zabbix_west]
[2024-12-06T10:30:17,265][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.06-001147] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [zabbix_moscow]
[2024-12-06T10:30:17,349][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_moscow-7.12.1-2024.12.05-001146] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [zabbix_moscow]
[2024-12-06T10:32:06,232][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77554698] overhead, spent [426ms] collecting in the last [1s]
[2024-12-06T10:40:14,169][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_east-7.12.1-2024.11.05-001072] from [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] to [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] in policy [zabbix_east]
[2024-12-06T10:40:14,218][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_east-7.12.1-2024.11.05-001072] from [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}] to [{"phase":"delete","action":"delete","name":"delete"}] in policy [zabbix_east]
[2024-12-06T10:40:14,276][INFO ][o.e.c.m.MetadataDeleteIndexService] [mdata03] [zabbix_east-7.12.1-2024.11.05-001072/_sXqYkgyQbSdhWOePn-cgQ] deleting index
[2024-12-06T10:58:37,847][INFO ][o.e.x.m.a.TransportMonitoringMigrateAlertsAction] [mdata03] THREAD NAME: {}elasticsearch[mdata03][management][T#3]
[2024-12-06T11:16:53,015][WARN ][o.e.t.InboundHandler ] [mdata03] handling inbound transport message [InboundMessage{Header{459}{7.13.2}{6718349620}{true}{false}{false}{false}{internal:coordination/fault_detection/leader_check}}] took [16738ms] which is above the warn threshold of [5000ms]
[2024-12-06T11:16:53,019][WARN ][o.e.t.OutboundHandler ] [mdata03] sending transport message [MessageSerializer{Response{1612974668}{false}{false}{false}{class org.elasticsearch.search.query.QuerySearchResult}}] of size [1657] on [Netty4TcpChannel{localAddress=/10.128.149.168:9300, remoteAddress=/10.128.149.169:52240, profile=default}] took [16738ms] which is above the warn threshold of [5000ms]
[2024-12-06T11:16:53,030][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77557366][1264184] duration [16.8s], collections [2]/[17.6s], total [16.8s]/[20.1h], memory [21.8gb]->[6.5gb]/[31gb], all_pools {[young] [14.2gb]->[64mb]/[0b]}{[old] [7.2gb]->[6.4gb]/[31gb]}{[survivor] [316.5mb]->[84.2mb]/[0b]}
[2024-12-06T11:16:53,030][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77557366] overhead, spent [16.8s] collecting in the last [17.6s]
[2024-12-06T11:17:30,477][WARN ][o.e.t.InboundHandler ] [mdata03] handling inbound transport message [InboundMessage{Header{1591}{7.13.2}{1614310871}{true}{false}{false}{false}{indices:data/read/search[phase/query]}}] took [6063ms] which is above the warn threshold of [5000ms]
[2024-12-06T11:17:30,477][WARN ][o.e.t.InboundHandler ] [mdata03] handling inbound transport message [InboundMessage{Header{1591}{7.13.2}{1614310872}{true}{false}{false}{false}{indices:data/read/search[phase/query]}}] took [6063ms] which is above the warn threshold of [5000ms]
[2024-12-06T11:17:30,484][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77557398][1264197] duration [6s], collections [1]/[6.4s], total [6s]/[20.1h], memory [21.9gb]->[6.7gb]/[31gb], all_pools {[young] [15.4gb]->[192mb]/[0b]}{[old] [6.4gb]->[6.4gb]/[31gb]}{[survivor] [82.1mb]->[74mb]/[0b]}
[2024-12-06T11:17:30,485][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77557398] overhead, spent [6s] collecting in the last [6.4s]
[2024-12-06T11:18:34,499][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77557462] overhead, spent [274ms] collecting in the last [1s]
[2024-12-06T11:30:14,086][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.05-001149] from [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] to [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] in policy [zabbix_center]
[2024-12-06T11:30:14,218][INFO ][o.e.c.m.MetadataCreateIndexService] [mdata03] [zabbix_center-7.12.1-2024.12.06-001150] creating index, cause [rollover_index], templates [zabbix_center-7.12.1], shards [1]/[1]
[2024-12-06T11:30:14,467][INFO ][o.e.c.m.MetadataMappingService] [mdata03] [zabbix_center-7.12.1-2024.12.06-001150/V0YOrQ9eSNadh0drdnH0Vw] update_mapping [_doc]
[2024-12-06T11:30:14,569][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.06-001150] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [zabbix_center]
[2024-12-06T11:30:14,621][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.05-001149] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [zabbix_center]
[2024-12-06T11:30:14,835][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.06-001150] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [zabbix_center]
[2024-12-06T11:30:14,897][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.05-001149] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [zabbix_center]
[2024-12-06T11:30:14,899][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.05-001149] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [zabbix_center]
[2024-12-06T11:30:15,075][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.06-001150] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [zabbix_center]
[2024-12-06T11:30:15,132][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_center-7.12.1-2024.12.05-001149] from [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] to [{"phase":"hot","action":"complete","name":"complete"}] in policy [zabbix_center]
[2024-12-06T11:30:15,190][INFO ][o.e.c.r.a.AllocationService] [mdata03] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[zabbix_center-7.12.1-2024.12.06-001150][0]]]).
[2024-12-06T12:16:42,984][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77560947][1264293] duration [1.9s], collections [1]/[2.5s], total [1.9s]/[20.1h], memory [7.5gb]->[6.5gb]/[31gb], all_pools {[young] [0b]->[32mb]/[0b]}{[old] [7.2gb]->[6.4gb]/[31gb]}{[survivor] [276mb]->[87.7mb]/[0b]}
[2024-12-06T12:16:42,984][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77560947] overhead, spent [1.9s] collecting in the last [2.5s]
[2024-12-06T12:17:07,610][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77560971][1264297] duration [985ms], collections [1]/[1.6s], total [985ms]/[20.1h], memory [21.5gb]->[6.5gb]/[31gb], all_pools {[young] [14.9gb]->[0b]/[0b]}{[old] [6.4gb]->[6.4gb]/[31gb]}{[survivor] [104.3mb]->[114.9mb]/[0b]}
[2024-12-06T12:17:07,610][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77560971] overhead, spent [985ms] collecting in the last [1.6s]
[2024-12-06T12:23:16,746][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77561340] overhead, spent [289ms] collecting in the last [1s]
[2024-12-06T12:25:48,355][INFO ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77561491] overhead, spent [617ms] collecting in the last [1.5s]
[2024-12-06T13:09:14,797][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77564092][1264382] duration [4.4s], collections [1]/[4.9s], total [4.4s]/[20.1h], memory [8.7gb]->[6.5gb]/[31gb], all_pools {[young] [1.2gb]->[16mb]/[0b]}{[old] [7.2gb]->[6.4gb]/[31gb]}{[survivor] [222.8mb]->[89.2mb]/[0b]}
[2024-12-06T13:09:14,797][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77564092] overhead, spent [4.4s] collecting in the last [4.9s]
[2024-12-06T13:16:33,021][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77564528][1264394] duration [2.3s], collections [1]/[2.8s], total [2.3s]/[20.1h], memory [21.8gb]->[6.6gb]/[31gb], all_pools {[young] [15.2gb]->[0b]/[0b]}{[old] [6.4gb]->[6.4gb]/[31gb]}{[survivor] [202.9mb]->[206.2mb]/[0b]}
[2024-12-06T13:16:33,021][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77564528] overhead, spent [2.3s] collecting in the last [2.8s]
[2024-12-06T14:05:37,058][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77567469][1264480] duration [2s], collections [1]/[2.5s], total [2s]/[20.1h], memory [8.6gb]->[6.4gb]/[31gb], all_pools {[young] [1.1gb]->[16mb]/[0b]}{[old] [7.3gb]->[6.3gb]/[31gb]}{[survivor] [178.4mb]->[94.8mb]/[0b]}
[2024-12-06T14:05:37,058][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77567469] overhead, spent [2s] collecting in the last [2.5s]
[2024-12-06T14:07:59,851][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][young][77567611][1264484] duration [1.2s], collections [1]/[1.7s], total [1.2s]/[20.1h], memory [22.9gb]->[6.5gb]/[31gb], all_pools {[young] [16.4gb]->[0b]/[0b]}{[old] [6.3gb]->[6.3gb]/[31gb]}{[survivor] [131.3mb]->[160mb]/[0b]}
[2024-12-06T14:07:59,851][WARN ][o.e.m.j.JvmGcMonitorService] [mdata03] [gc][77567611] overhead, spent [1.2s] collecting in the last [1.7s]
[2024-12-06T14:29:21,224][INFO ][o.e.x.m.a.TransportMonitoringMigrateAlertsAction] [mdata03] THREAD NAME: {}elasticsearch[mdata03][management][T#2]
[2024-12-06T14:30:14,074][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_b2b201-2024.11.05-000031] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_b2b201]
[2024-12-06T14:30:14,149][INFO ][o.e.x.i.IndexLifecycleTransition] [mdata03] moving index [zabbix_b2b301-2024.11.05-000015] from [{"phase":"hot","action":"complete","name":"complete"}] to [{"phase":"delete","action":"delete","name":"wait-for-shard-history-leases"}] in policy [zabbix_b2b301]
