Elasticsearch version: 7.17.1 (yes, I know.)
Curator version: 7.0.1
I'm seeing an occasional error where a Curator task about allocation
fails with
Failed to complete action: allocation. <class 'KeyError'>: 'store'
I haven't been able to find any mentions of anyone else encountering
this. Most days all of the fifty odd tasks we have for
hot/cold allocation work, some days one of them fails. There is no consistency in which, if any, allocation task fails on a given day. I cannot correlate the failures with anything about cluster state. The failures appear essentially random. Nothing shows in Elasticsearch log at the time of the failure. The failures seem to only occur when Curator runs overnight. I ran all the allocation tasks on a loop with DEBUG logging during normal working hours over several days and got no failures. I've ended up having to modify the script which runs Curator overnight to run just the allocation tasks with DEBUG logging (and compress the GBs of output) to be able to see a failure happen while DEBUG logging is in effect.
Config:
---
client:
hosts:
- foo.bar.redact
port: 9200
url_prefix:
use_ssl: True
certificate: /etc/pki/tls/certs/ca-bundle.crt
client_cert:
client_key:
ssl_no_validate: False
username: "curator"
password: "redact"
timeout: 30
# master_only: True
master_only: False
logging:
# loglevel: INFO
loglevel: DEBUG
logformat: default
# blacklist: ['elasticsearch', 'urllib3']
blacklist: []
An action:
10150-l2vni-firewall-allocation:
action: allocation
description: "Apply shard allocation filtering rules to
l2vni-firewall indexes"
options:
ignore_empty_list: True
continue_if_exception: True
timeout_override: 300
allocation_type: include
key: datatype
value: cold
filters:
- filtertype: pattern
kind: prefix
value: l2vni-firewall-
- filtertype: age
source: field_stats
field: '@timestamp'
direction: older
stats_result: min_value
unit: days
unit_count: 31
DEBUG log is tens of MBs just relating to a single action so here's the
end of DEBUG logging for when that action happened to fail (having
worked on previous runs, and it's since worked on subsequent runs)
2025-02-04 01:28:32,876 DEBUG curator.indexlist
iterate_over_stats:127 Index: firepower-firewall-2024.11.11
Size: 1.4TB Docs: 6169013300 PrimarySize: 704.1GB
2025-02-04 01:28:32,876 DEBUG curator.indexlist
iterate_over_stats:127 Index: firepower-firewall-2024.12.21
Size: 309.3GB Docs: 1239412768 PrimarySize: 154.6GB
2025-02-04 01:28:32,992 DEBUG urllib3.connectionpool
_make_request:453 https://foo.bar:9200 "GET
/.kibana-sem_task_manager_7.17.1_001,.kibana-ws_7.17.1_001,.reporting-2022-09-11,.reporting-2022-12-18,.reporting-2024-01-21,.reporting-2023-05-21,radius-auth-detail-2025.01.17,radius-auth-detail-2025.02.03,radius-log-2024.12.06,radius-log-2025.01.14,httpd-access-2024-25,httpd-access-2024-46,learning-systems-httpd-access-2025-05,linux-httpd-access-2024-45,linux-syslog-2024.11.28,linux-syslog8-2024.11.06,linux-syslog8-2024.12.03,linux-syslog8-2024.12.05,linux-syslog8-2024.12.15,linux-syslog8-2025.01.03,linux-syslog8-2025.01.07,linux-syslog8-2025.02.02,dhcp-2024.11.23,dhcp-2024.12.05,dhcp-2025.01.28,dns-queries-external-2024.11.27,dns-queries-external-2024.12.21,dns-queries-external-2024.12.26,dns-queries-external-2025.01.07,dns-queries-external-2025.02.03,dns-queries-internal-2024.12.28,firepower-firewall-2024.08.17-avoidclosed-c2024.12.10c,firepower-firewall-2024.08.20-avoidclosed-c2024.12.18c,firepower-firewall-2024.08.21-avoidclosed-c2024.11.22c,firepower-firewall-2024.09.09-avoidclosed-c2024.12.10c,firepower-firewall-2024.11.08,firepower-firewall-2024.12.15,firepower-firewall-2024.12.19,firepower-firewall-2025.01.10,firepower-firewall-2025.01.12,firewall-2024.11.14,firewall-2024.12.06,security-firewall-2024.12.28,firewall-2025.02.03,netflow-2024.12.21,netflow-2024.12.22,netflow-2024.12.29,vpn-2025.01.08,vpn-2025.01.24,vpn-2025.01.29,eventlog-application-2024.09-avoidclosed-c2024.12c,eventlog-2024.11.27,eventlog-2024.12.04,server-provisioning-eventlog-2024.12.12,eventlog-2024.12.16,eventlog-2025.01.09,eventlog-2025.01.30,firewall-prod-2024.11.07,firewall-prod-2024.11.24,firewall-prod-2025.02.04,iis-2024.12.27,sql-2024.11.27,sql-2024.12.09,sql-2024.12.10,sits-httpd-error-2025.02,storage-eventlog-2024.08.24-avoidclosed-c2024.12.19c,storage-eventlog-2024.09.30-avoidclosed-c2025.01.23c,storage-eventlog-2024.11.25,storage-eventlog-2025.01.04,storage-eventlog-2025.01.14,storage-freenas-2024.11.11,storage-freenas-2025.01.23,unix-mail-relays-2024-48,unix-mail-relays-2025-03,access-2025.01.30,application-2024.11.06,application-2024.11.29,application-2025.01.16,application-2025.01.30,application-2025.02.03,audit-2023-47,audit-2024-02,audit-2024-21,reports-2025-03/_stats/store,docs
HTTP/1.1" 200 31532
2025-02-04 01:28:32,992 INFO elasticsearch
log_request_success:265 GET
https://foo.bar:9200/.kibana-sem-_task_manager_7.17.1_001,.kibana-ws_7.17.1_001,.reporting-2022-09-11,.reporting-2022-12-18,.reporting-2024-01-21,.reporting-2023-05-21,radius-auth-detail-2025.01.17,radius-auth-detail-2025.02.03,radius-log-2024.12.06,radius-log-2025.01.14,httpd-access-2024-25,httpd-access-2024-46,httpd-access-2025-05,linux-httpd-access-2024-45,linux-hosting-syslog-2024.11.28,linux-syslog8-2024.11.06,linux-syslog8-2024.12.03,linux-syslog8-2024.12.05,linux-syslog8-2024.12.15,linux-syslog8-2025.01.03,linux-syslog8-2025.01.07,linux-syslog8-2025.02.02,dhcp-2024.11.23,dhcp-2024.12.05,dhcp-2025.01.28,dns-queries-external-2024.11.27,dns-queries-external-2024.12.21,dns-queries-external-2024.12.26,dns-queries-external-2025.01.07,dns-queries-external-2025.02.03,dns-queries-internal-2024.12.28,firepower-firewall-2024.08.17-avoidclosed-c2024.12.10c,firepower-firewall-2024.08.20-avoidclosed-c2024.12.18c,firepower-firewall-2024.08.21-avoidclosed-c2024.11.22c,firepower-firewall-2024.09.09-avoidclosed-c2024.12.10c,firepower-firewall-2024.11.08,firepower-firewall-2024.12.15,firepower-firewall-2024.12.19,firepower-firewall-2025.01.10,firepower-firewall-2025.01.12,firewall-2024.11.14,firewall-2024.12.06,firewall-2024.12.28,firewall-2025.02.03,netflow-2024.12.21,netflow-2024.12.22,netflow-2024.12.29,vpn-2025.01.08,vpn-2025.01.24,vpn-2025.01.29,eventlog-application-2024.09-avoidclosed-c2024.12c,eventlog-2024.11.27,eventlog-2024.12.04,eventlog-2024.12.12,eventlog-2024.12.16,eventlog-2025.01.09,eventlog-2025.01.30,firewall-prod-2024.11.07,firewall-prod-2024.11.24,firewall-prod-2025.02.04,iis-2024.12.27,sql-2024.11.27,sql-2024.12.09,sql-2024.12.10,sits-httpd-error-2025.02,storage-eventlog-2024.08.24-avoidclosed-c2024.12.19c,storage-eventlog-2024.09.30-avoidclosed-c2025.01.23c,storage-eventlog-2024.11.25,storage-eventlog-2025.01.04,storage-eventlog-2025.01.14,storage-freenas-2024.11.11,storage-freenas-2025.01.23,unix-mail-relays-2024-48,unix-mail-relays-2025-03,access-2025.01.30,application-2024.11.06,application-2024.11.29,application-2025.01.16,application-2025.01.30,application-2025.02.03,audit-2023-47,audit-2024-02,audit-2024-21,reports-2025-03/_stats/store,docs
[status:200 request:0.116s]
2025-02-04 01:28:32,993 DEBUG elasticsearch
log_request_success:268 > None
2025-02-04 01:28:32,993 DEBUG elasticsearch
log_request_success:269 <
{"_shards":{"total":656,"successful":655,"failed":0},"_all":{"primaries":{"docs":{"count":12682354001,"deleted":49242},"store":{"size_in_bytes":3704130106765,"total_data_set_size_in_bytes":3704130106765,"reserved_in_bytes":0}},"total":{"docs":{"count":25381384841,"deleted":98484},"store":{"size_in_bytes":7412248285492,"total_data_set_size_in_bytes":7412248285492,"reserved_in_bytes":0}}},"indices":{"linux-syslog8-2025.02.02":{"uuid":"w72QM4GdQkOEKmbRPh129w","primaries":{"docs":{"count":49327713,"deleted":0},"store":{"size_in_bytes":28056625471,"total_data_set_size_in_bytes":28056625471,"reserved_in_bytes":0}},"total":{"docs":{"count":98655426,"deleted":0},"store":{"size_in_bytes":56103177653,"total_data_set_size_in_bytes":56103177653,"reserved_in_bytes":0}}},"application-2024.11.29":{"uuid":"DKGO5vdZRmuhTJvNSChMGw","primaries":{"docs":{"count":40378898,"deleted":0},"store":{"size_in_bytes":7727311572,"total_data_set_size_in_bytes":7727311572,"reserved_in_bytes":0}},"total":{"docs":{"count":80757796,"deleted":0},"store":{"size_in_bytes":15454623144,"total_data_set_size_in_bytes":15454623144,"reserved_in_bytes":0}}},"storage-eventlog-2024.09.30-avoidclosed-c2025.01.23c":{"uuid":"Nh7novuwR32h-i1IhPgVZA","primaries":{"docs":{"count":1129,"deleted":0},"store":{"size_in_bytes":1189146,"total_data_set_size_in_bytes":1189146,"reserved_in_bytes":0}},"total":{"docs":{"count":2258,"deleted":0},"store":{"size_in_bytes":2378292,"total_data_set_size_in_bytes":2378292,"reserved_in_bytes":0}}},"dns-queries-external-2024.12.21":{"uuid":"x2WRtV2cQuy6U3kGmV2kpw","primaries":{"docs":{"count":12390378,"deleted":0},"store":{"size_in_bytes":2482864203,"total_data_set_size_in_bytes":2482864203,"reserved_in_bytes":0}},"total":{"docs":{"count":24780756,"deleted":0},"store":{"size_in_bytes":4965728406,"total_data_set_size_in_bytes":4965728406,"reserved_in_bytes":0}}},"dns-queries-external-2024.12.26":{"uuid":"KooJInZ7SiqChB1MNcjhlQ","primaries":{"docs":{"count":12735255,"deleted":0},"store":{"size_in_bytes":2583612578,"total_data_set_size_in_bytes":2583612578,"reserved_in_bytes":0}},"total":{"docs":{"count":25470510,"deleted":0},"store":{"size_in_bytes":5167225156,"total_data_set_size_in_bytes":5167225156,"reserved_in_bytes":0}}},"storage-eventlog-2024.11.25":{"uuid":"lhaTF84VSPevzF9i5XsWfg","primaries":{"docs":{"count":13816250,"deleted":0},"store":{"size_in_bytes":12332663561,"total_data_set_size_in_bytes":12332663561,"reserved_in_bytes":0}},"total":{"docs":{"count":27632500,"deleted":0},"store":{"size_in_bytes":24665213933,"total_data_set_size_in_bytes":24665213933,"reserved_in_bytes":0}}},"firewall-2024.12.06":{"uuid":"I1djGafZR66GokxD6xFgpA","primaries":{"docs":{"count":225331121,"deleted":0},"store":{"size_in_bytes":76414585675,"total_data_set_size_in_bytes":76414585675,"reserved_in_bytes":0}},"total":{"docs":{"count":450662242,"deleted":0},"store":{"size_in_bytes":152829171350,"total_data_set_size_in_bytes":152829171350,"reserved_in_bytes":0}}},"linux-syslog8-2024.12.03":{"uuid":"Md-A1YE7T_uI0a3uIU0HLg","primaries":{"docs":{"count":132643873,"deleted":0},"store":{"size_in_bytes":46510233093,"total_data_set_size_in_bytes":46510233093,"reserved_in_bytes":0}},"total":{"docs":{"count":265287746,"deleted":0},"store":{"size_in_bytes":93024496563,"total_data_set_size_in_bytes":93024496563,"reserved_in_bytes":0}}},"linux-syslog8-2024.12.05":{"uuid":"8M5L1xmMTk6rglj9lxFsqA","primaries":{"docs":{"count":110861492,"deleted":0},"store":{"size_in_bytes":40701162299,"total_data_set_size_in_bytes":40701162299,"reserved_in_bytes":0}},"total":{"docs":{"count":221722984,"deleted":0},"store":{"size_in_bytes":81402324598,"total_data_set_size_in_bytes":81402324598,"reserved_in_bytes":0}}},"sql-2024.12.10":{"uuid":"OorHx5NMRZKJUGJRJqYwPQ","primaries":{"docs":{"count":23276709,"deleted":0},"store":{"size_in_bytes":14773977649,"total_data_set_size_in_bytes":14773977649,"reserved_in_bytes":0}},"total":{"docs":{"count":46553418,"deleted":0},"store":{"size_in_bytes":29547910091,"total_data_set_size_in_bytes":29547910091,"reserved_in_bytes":0}}},".kibana-ws_7.17.1_001":{"uuid":"7mYHpnZURsG30xmCNjABqw","primaries":{"docs":{"count":33,"deleted":12},"store":{"size_in_bytes":2496635,"total_data_set_size_in_bytes":2496635,"reserved_in_bytes":0}},"total":{"docs":{"count":66,"deleted":24},"store":{"size_in_bytes":4993270,"total_data_set_size_in_bytes":4993270,"reserved_in_bytes":0}}},"eventlog-2024.12.16":{"uuid":"77P0WJqNS-6kfqh-2beLMg","primaries":{"docs":{"count":18565768,"deleted":0},"store":{"size_in_bytes":22218941731,"total_data_set_size_in_bytes":22218941731,"reserved_in_bytes":0}},"total":{"docs":{"count":37131536,"deleted":0},"store":{"size_in_bytes":44437848119,"total_data_set_size_in_bytes":44437848119,"reserved_in_bytes":0}}},"eventlog-2024.12.12":{"uuid":"XKG9J-gkQT-ykGGJynZzkw","primaries":{"docs":{"count":21468658,"deleted":0},"store":{"size_in_bytes":25524194757,"total_data_set_size_in_bytes":25524194757,"reserved_in_bytes":0}},"total":{"docs":{"count":42937316,"deleted":0},"store":{"size_in_bytes":51048093522,"total_data_set_size_in_bytes":51048093522,"reserved_in_bytes":0}}},"dhcp-2025.01.28":{"uuid":"9gfXdgBTStOvoY3wRMj6mA","primaries":{"docs":{"count":27496884,"deleted":0},"store":{"size_in_bytes":5170463829,"total_data_set_size_in_bytes":5170463829,"reserved_in_bytes":0}},"total":{"docs":{"count":54993768,"deleted":0},"store":{"size_in_bytes":10340927658,"total_data_set_size_in_bytes":10340927658,"reserved_in_bytes":0}}},"dhcp-2024.12.05":{"uuid":"jlwjIXfLT7GSTXD-Xu4dLg","primaries":{"docs":{"count":28841834,"deleted":0},"store":{"size_in_bytes":5401786038,"total_data_set_size_in_bytes":5401786038,"reserved_in_bytes":0}},"total":{"docs":{"count":57683668,"deleted":0},"store":{"size_in_bytes":10803572076,"total_data_set_size_in_bytes":10803572076,"reserved_in_bytes":0}}},"dns-queries-external-2025.01.07":{"uuid":"vuUOw3dJSDWrOqjSXKU5KA","primaries":{"docs":{"count":16839565,"deleted":0},"store":{"size_in_bytes":3228113243,"total_data_set_size_in_bytes":3228113243,"reserved_in_bytes":0}},"total":{"docs":{"count":33679130,"deleted":0},"store":{"size_in_bytes":6456226486,"total_data_set_size_in_bytes":6456226486,"reserved_in_bytes":0}}},"radius-auth-detail-2025.01.17":{"uuid":"HaiW9GTqT6a3PK7l26dbJg","primaries":{"docs":{"count":1071924,"deleted":0},"store":{"size_in_bytes":871721297,"total_data_set_size_in_bytes":871721297,"reserved_in_bytes":0}},"total":{"docs":{"count":2143848,"deleted":0},"store":{"size_in_bytes":1743442594,"total_data_set_size_in_bytes":1743442594,"reserved_in_bytes":0}}},"audit-2024-02":{"uuid":"dDEuo3T_SPmJq3tjFwtVnA","primaries":{"docs":{"count":4668423,"deleted":0},"store":{"size_in_bytes":2382098068,"total_data_set_size_in_bytes":2382098068,"reserved_in_bytes":0}},"total":{"docs":{"count":9336846,"deleted":0},"store":{"size_in_bytes":4764196136,"total_data_set_size_in_bytes":4764196136,"reserved_in_bytes":0}}},"vpn-2025.01.29":{"uuid":"qYOcLbFhR9CUp0JBq66yzA","primaries":{"docs":{"count":122102,"deleted":0},"store":{"size_in_bytes":20287587,"total_data_set_size_in_bytes":20287587,"reserved_in_bytes":0}},"total":{"docs":{"count":244204,"deleted":0},"store":{"size_in_bytes":40575174,"total_data_set_size_in_bytes":40575174,"reserved_in_bytes":0}}},"radius-log-2024.12.06":{"uuid":"cIGUF08qQviQVXiFFPSHFw","primaries":{"docs":{"count":130582,"deleted":0},"store":{"size_in_bytes":39952993,"total_data_set_size_in_bytes":39952993,"reserved_in_bytes":0}},"total":{"docs":{"count":261164,"deleted":0},"store":{"size_in_bytes":79905986,"total_data_set_size_in_bytes":79905986,"reserved_in_bytes":0}}},"application-2025.01.16":{"uuid":"O6atelcoTTCZHHibUZlB1Q","primaries":{"docs":{"count":44569894,"deleted":0},"store":{"size_in_bytes":8398850268,"total_data_set_size_in_bytes":8398850268,"reserved_in_bytes":0}},"total":{"docs":{"count":89139788,"deleted":0},"store":{"size_in_bytes":16797700536,"total_data_set_size_in_bytes":16797700536,"reserved_in_bytes":0}}},"firewall-prod-2024.11.24":{"uuid":"e1p3XLydTmSl6C1RUPiCbA","primaries":{"docs":{"count":247047772,"deleted":0},"store":{"size_in_bytes":40173441183,"total_data_set_size_in_bytes":40173441183,"reserved_in_bytes":0}},"total":{"docs":{"count":494095544,"deleted":0},"store":{"size_in_bytes":80346882366,"total_data_set_size_in_bytes":80346882366,"reserved_in_bytes":0}}},"vpn-2025.01.24":{"uuid":"i9hrQe0nRIWGuOOym8JkEg","primaries":{"docs":{"count":140323,"deleted":0},"store":{"size_in_bytes":23252628,"total_data_set_size_in_bytes":23252628,"reserved_in_bytes":0}},"total":{"docs":{"count":280646,"deleted":0},"store":{"size_in_bytes":46505256,"total_data_set_size_in_bytes":46505256,"reserved_in_bytes":0}}},"eventlog-2025.01.09":{"uuid":"dti8RiX2R1yFHu-nZE7jcw","primaries":{"docs":{"count":20149090,"deleted":0},"store":{"size_in_bytes":23087008979,"total_data_set_size_in_bytes":23087008979,"reserved_in_bytes":0}},"total":{"docs":{"count":40298180,"deleted":0},"store":{"size_in_bytes":46174255268,"total_data_set_size_in_bytes":46174255268,"reserved_in_bytes":0}}},".reporting-2022-09-11":{"uuid":"QrKTNeb4SPS50e9qth_Cgw","primaries":{"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":18001,"total_data_set_size_in_bytes":18001,"reserved_in_bytes":0}},"total":{"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":36002,"total_data_set_size_in_bytes":36002,"reserved_in_bytes":0}}},"httpd-access-2024-25":{"uuid":"np5d3wS3T863ChgHMPnHpw","primaries":{"docs":{"count":3507655,"deleted":0},"store":{"size_in_bytes":1858710707,"total_data_set_size_in_bytes":1858710707,"reserved_in_bytes":0}},"total":{"docs":{"count":7015310,"deleted":0},"store":{"size_in_bytes":3717421414,"total_data_set_size_in_bytes":3717421414,"reserved_in_bytes":0}}},".reporting-2022-12-18":{"uuid":"EhKmNG6hTGyglsrBgaK8oQ","primaries":{"docs":{"count":2,"deleted":0},"store":{"size_in_bytes":10420155,"total_data_set_size_in_bytes":10420155,"reserved_in_bytes":0}},"total":{"docs":{"count":4,"deleted":0},"store":{"size_in_bytes":20840310,"total_data_set_size_in_bytes":20840310,"reserved_in_bytes":0}}},"reports-2025-03":{"uuid":"kOp3ekdCQQWkLgTSxJSqoA","primaries":{"docs":{"count":212494,"deleted":0},"store":{"size_in_bytes":164167678,"total_data_set_size_in_bytes":164167678,"reserved_in_bytes":0}},"total":{"docs":{"count":424988,"deleted":0},"store":{"size_in_bytes":328335356,"total_data_set_size_in_bytes":328335356,"reserved_in_bytes":0}}},"eventlog-2024.12.04":{"uuid":"FlyVpg11RPCFNBimXtapQA","primaries":{"docs":{"count":21012551,"deleted":0},"store":{"size_in_bytes":24665210237,"total_data_set_size_in_bytes":24665210237,"reserved_in_bytes":0}},"total":{"docs":{"count":42025102,"deleted":0},"store":{"size_in_bytes":49330420474,"total_data_set_size_in_bytes":49330420474,"reserved_in_bytes":0}}},"storage-eventlog-2024.08.24-avoidclosed-c2024.12.19c":{"uuid":"miYqAyHyRqeYbAW8dnU3ww","primaries":{"docs":{"count":194,"deleted":0},"store":{"size_in_bytes":554191,"total_data_set_size_in_bytes":554191,"reserved_in_bytes":0}},"total":{"docs":{"count":388,"deleted":0},"store":{"size_in_bytes":1108382,"total_data_set_size_in_bytes":1108382,"reserved_in_bytes":0}}},"unix-mail-relays-2024-48":{"uuid":"3sFNgI2HQBeVy8st176dmg","primaries":{"docs":{"count":28451369,"deleted":0},"store":{"size_in_bytes":8437005890,"total_data_set_size_in_bytes":8437005890,"reserved_in_bytes":0}},"total":{"docs":{"count":56902738,"deleted":0},"store":{"size_in_bytes":16874011780,"total_data_set_size_in_bytes":16874011780,"reserved_in_bytes":0}}},"storage-freenas-2025.01.23":{"uuid":"v1wIZZ_LRAuAqTSUOvSqOQ","primaries":{"docs":{"count":828100,"deleted":0},"store":{"size_in_bytes":151195596,"total_data_set_size_in_bytes":151195596,"reserved_in_bytes":0}},"total":{"docs":{"count":1656200,"deleted":0},"store":{"size_in_bytes":302391192,"total_data_set_size_in_bytes":302391192,"reserved_in_bytes":0}}},"application-2024.11.06":{"uuid":"QMQmWRVKRUS0HaTpMQW08Q","primaries":{"docs":{"count":44969878,"deleted":0},"store":{"size_in_bytes":12128686296,"total_data_set_size_in_bytes":12128686296,"reserved_in_bytes":0}},"total":{"docs":{"count":89939756,"deleted":0},"store":{"size_in_bytes":24257372592,"total_data_set_size_in_bytes":24257372592,"reserved_in_bytes":0}}},"dhcp-2024.11.23":{"uuid":"Fij3E-sXR66M-SlqTDPbew","primaries":{"docs":{"count":15861966,"deleted":0},"store":{"size_in_bytes":3002573056,"total_data_set_size_in_bytes":3002573056,"reserved_in_bytes":0}},"total":{"docs":{"count":31723932,"deleted":0},"store":{"size_in_bytes":6005146112,"total_data_set_size_in_bytes":6005146112,"reserved_in_bytes":0}}},"sql-2024.12.09":{"uuid":"UfgoXpuVRfiz4Rvyc2a0NA","primaries":{"docs":{"count":19560062,"deleted":0},"store":{"size_in_bytes":11637284455,"total_data_set_size_in_bytes":11637284455,"reserved_in_bytes":0}},"total":{"docs":{"count":39120124,"deleted":0},"store":{"size_in_bytes":23274551891,"total_data_set_size_in_bytes":23274551891,"reserved_in_bytes":0}}},"linux-httpd-access-2024-45":{"uuid":"4OoOq3dZTDKa0l-T8D9-hw","primaries":{"docs":{"count":1958042,"deleted":0},"store":{"size_in_bytes":961024741,"total_data_set_size_in_bytes":961024741,"reserved_in_bytes":0}},"total":{"docs":{"count":3916084,"deleted":0},"store":{"size_in_bytes":1922049482,"total_data_set_size_in_bytes":1922049482,"reserved_in_bytes":0}}},"storage-eventlog-2025.01.04":{"uuid":"liCE9o9mQJmmvA5P6QzRJQ","primaries":{"docs":{"count":7612094,"deleted":0},"store":{"size_in_bytes":5720416748,"total_data_set_size_in_bytes":5720416748,"reserved_in_bytes":0}},"total":{"docs":{"count":15224188,"deleted":0},"store":{"size_in_bytes":11440965033,"total_data_set_size_in_bytes":11440965033,"reserved_in_bytes":0}}},"firewall-2024.11.14":{"uuid":"iaDxsqT9StOqRnm-N37gGw","primaries":{"docs":{"count":231206579,"deleted":0},"store":{"size_in_bytes":75652258020,"total_data_set_size_in_bytes":75652258020,"reserved_in_bytes":0}},"total":{"docs":{"count":462413158,"deleted":0},"store":{"size_in_bytes":151304516040,"total_data_set_size_in_bytes":151304516040,"reserved_in_bytes":0}}},".reporting-2023-05-21":{"uuid":"Vq8_5e0US02Bny1rsQ3uIw","primaries":{"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":58772,"total_data_set_size_in_bytes":58772,"reserved_in_bytes":0}},"total":{"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":117544,"total_data_set_size_in_bytes":117544,"reserved_in_bytes":0}}},"firepower-firewall-2024.11.08":{"uuid":"j4UIKQhnTq2YiV0AtDT1nw","primaries":{"docs":{"count":2308017193,"deleted":0},"store":{"size_in_bytes":573686282868,"total_data_set_size_in_bytes":573686282868,"reserved_in_bytes":0}},"total":{"docs":{"count":4616034386,"deleted":0},"store":{"size_in_bytes":1147372565736,"total_data_set_size_in_bytes":1147372565736,"reserved_in_bytes":0}}},"firepower-firewall-2025.01.12":{"uuid":"Y5uR83-uRguKWmq-LGzAdA","primaries":{"docs":{"count":2025831192,"deleted":0},"store":{"size_in_bytes":506038432760,"total_data_set_size_in_bytes":506038432760,"reserved_in_bytes":0}},"total":{"docs":{"count":4051662384,"deleted":0},"store":{"size_in_bytes":1012076906605,"total_data_set_size_in_bytes":1012076906605,"reserved_in_bytes":0}}},"firewall-2024.12.28":{"uuid":"OIgUX2vLSfiiuoDbA0F2ww","primaries":{"docs":{"count":135323197,"deleted":0},"store":{"size_in_bytes":46861243994,"total_data_set_size_in_bytes":46861243994,"reserved_in_bytes":0}},"total":{"docs":{"count":270646394,"deleted":0},"store":{"size_in_bytes":93722487988,"total_data_set_size_in_bytes":93722487988,"reserved_in_bytes":0}}},"dns-queries-external-2025.02.03":{"uuid":"oZQJcgPuSASJUva74NMr4A","primaries":{},"total":{"docs":{"count":16476359,"deleted":0},"store":{"size_in_bytes":3214684365,"total_data_set_size_in_bytes":3214684365,"reserved_in_bytes":0}}},"firepower-firewall-2024.08.20-avoidclosed-c2024.12.18c":{"uuid":"gAP8qmyIRRKM6iGaRas0Qg","primaries":{"docs":{"count":531,"deleted":0},"store":{"size_in_bytes":826411,"total_data_set_size_in_bytes":826411,"reserved_in_bytes":0}},"total":{"docs":{"count":1062,"deleted":0},"store":{"size_in_bytes":1652822,"total_data_set_size_in_bytes":1652822,"reserved_in_bytes":0}}},"firepower-firewall-2024.08.17-avoidclosed-c2024.12.10c":{"uuid":"ZCDQRjS4TtGd3KYRq7RCjw","primaries":{"docs":{"count":2,"deleted":0},"store":{"size_in_bytes":54796,"total_data_set_size_in_bytes":54796,"reserved_in_bytes":0}},"total":{"docs":{"count":4,"deleted":0},"store":{"size_in_bytes":109592,"total_data_set_size_in_bytes":109592,"reserved_in_bytes":0}}},"audit-2023-47":{"uuid":"vUkDIO0hQq-Wl_9OFyJNyQ","primaries":{"docs":{"count":3999217,"deleted":0},"store":{"size_in_bytes":2082084539,"total_data_set_size_in_bytes":2082084539,"reserved_in_bytes":0}},"total":{"docs":{"count":7998434,"deleted":0},"store":{"size_in_bytes":4164169078,"total_data_set_size_in_bytes":4164169078,"reserved_in_bytes":0}}},"firepower-firewall-2025.01.10":{"uuid":"Wy0YCRt6S-C9V4qALYPaAA","primaries":{"docs":{"count":2499750832,"deleted":0},"store":{"size_in_bytes":622107457808,"total_data_set_size_in_bytes":622107457808,"reserved_in_bytes":0}},"total":{"docs":{"count":4999501664,"deleted":0},"store":{"size_in_bytes":1244214895500,"total_data_set_size_in_bytes":1244214895500,"reserved_in_bytes":0}}},"firepower-firewall-2024.12.19":{"uuid":"ObdJGb89QPmEI2MebENXNA","primaries":{"docs":{"count":856203874,"deleted":0},"store":{"size_in_bytes":231915911789,"total_data_set_size_in_bytes":231915911789,"reserved_in_bytes":0}},"total":{"docs":{"count":1712407748,"deleted":0},"store":{"size_in_bytes":463831823578,"total_data_set_size_in_bytes":463831823578,"reserved_in_bytes":0}}},"firepower-firewall-2024.08.21-avoidclosed-c2024.11.22c":{"uuid":"UyKDVdSaQdmSqi4sDi_LWw","primaries":{"docs":{"count":1,"deleted":0},"store":{"size_in_bytes":29263,"total_data_set_size_in_bytes":29263,"reserved_in_bytes":0}},"total":{"docs":{"count":2,"deleted":0},"store":{"size_in_bytes":58526,"total_data_set_size_in_bytes":58526,"reserved_in_bytes":0}}},"iis-2024.12.27":{"uuid":"qkUrVuc1QQezaqhrJcuHVQ","primaries":{"docs":{"count":10192020,"deleted":0},"store":{"size_in_bytes":1551616085,"total_data_set_size_in_bytes":1551616085,"reserved_in_bytes":0}},"total":{"docs":{"count":20384040,"deleted":0},"store":{"size_in_bytes":3103232170,"total_data_set_size_in_bytes":3103232170,"reserved_in_bytes":0}}},"storage-freenas-2024.11.11":{"uuid":"nPnL_Nc7SP6JKE9fAJ8zDQ","primaries":{"docs":{"count":184103,"deleted":0},"store":{"size_in_bytes":44743346,"total_data_set_size_in_bytes":44743346,"reserved_in_bytes":0}},"total":{"docs":{"count":368206,"deleted":0},"store":{"size_in_bytes":89486692,"total_data_set_size_in_bytes":89486692,"reserved_in_bytes":0}}},"firepower-firewall-2024.12.15":{"uuid":"E8izDNKTST-YZi6IxnCk_g","primaries":{"docs":{"count":751678609,"deleted":0},"store":{"size_in_bytes":199225556387,"total_data_set_size_in_bytes":199225556387,"reserved_in_bytes":0}},"total":{"docs":{"count":1503357218,"deleted":0},"store":{"size_in_bytes":398451112486,"total_data_set_size_in_bytes":398451112486,"reserved_in_bytes":0}}},"httpd-access-2025-05":{"uuid":"CmYIrd4hRUyUhoduXpurpw","primaries":{"docs":{"count":10684818,"deleted":0},"store":{"size_in_bytes":6036389912,"total_data_set_size_in_bytes":6036389912,"reserved_in_bytes":0}},"total":{"docs":{"count":21369636,"deleted":0},"store":{"size_in_bytes":12072779824,"total_data_set_size_in_bytes":12072779824,"reserved_in_bytes":0}}},".kibana-sem-_task_manager_7.17.1_001":{"uuid":"WRAQXtNeTYeT1W4i1F4Zxg","primaries":{"docs":{"count":11,"deleted":49230},"store":{"size_in_bytes":4885950,"total_data_set_size_in_bytes":4885950,"reserved_in_bytes":0}},"total":{"docs":{"count":22,"deleted":98460},"store":{"size_in_bytes":9771900,"total_data_set_size_in_bytes":9771900,"reserved_in_bytes":0}}},"radius-log-2025.01.14":{"uuid":"VO7r1b3JRNOy9EYdep4pag","primaries":{"docs":{"count":116741,"deleted":0},"store":{"size_in_bytes":35962940,"total_data_set_size_in_bytes":35962940,"reserved_in_bytes":0}},"total":{"docs":{"count":233482,"deleted":0},"store":{"size_in_bytes":71925880,"total_data_set_size_in_bytes":71925880,"reserved_in_bytes":0}}},"httpd-access-2024-46":{"uuid":"RV-HPWJmTDCJIm9iRSACBQ","primaries":{"docs":{"count":14180640,"deleted":0},"store":{"size_in_bytes":7767873384,"total_data_set_size_in_bytes":7767873384,"reserved_in_bytes":0}},"total":{"docs":{"count":28361280,"deleted":0},"store":{"size_in_bytes":15535746768,"total_data_set_size_in_bytes":15535746768,"reserved_in_bytes":0}}},"eventlog-application-2024.09-avoidclosed-c2024.12c":{"uuid":"PGPajv4BQ9mLzSyfz8Ys3w","primaries":{"docs":{"count":208,"deleted":0},"store":{"size_in_bytes":130674,"total_data_set_size_in_bytes":130674,"reserved_in_bytes":0}},"total":{"docs":{"count":416,"deleted":0},"store":{"size_in_bytes":261348,"total_data_set_size_in_bytes":261348,"reserved_in_bytes":0}}},"eventlog-2025.01.30":{"uuid":"Y6IO6ofxScalsKjlLD1XDA","primaries":{"docs":{"count":26533029,"deleted":0},"store":{"size_in_bytes":28248170862,"total_data_set_size_in_bytes":28248170862,"reserved_in_bytes":0}},"total":{"docs":{"count":53066058,"deleted":0},"store":{"size_in_bytes":56496341724,"total_data_set_size_in_bytes":56496341724,"reserved_in_bytes":0}}},"eventlog-2024.11.27":{"uuid":"APayPIY5SHuJ0WYSWD10Ug","primaries":{"docs":{"count":21229970,"deleted":0},"store":{"size_in_bytes":24892017689,"total_data_set_size_in_bytes":24892017689,"reserved_in_bytes":0}},"total":{"docs":{"count":42459940,"deleted":0},"store":{"size_in_bytes":49784035378,"total_data_set_size_in_bytes":49784035378,"reserved_in_bytes":0}}},"linux-syslog8-2025.01.03":{"uuid":"wPWujghHS8StJyb73rygBg","primaries":{"docs":{"count":103976189,"deleted":0},"store":{"size_in_bytes":38535835001,"total_data_set_size_in_bytes":38535835001,"reserved_in_bytes":0}},"total":{"docs":{"count":207952378,"deleted":0},"store":{"size_in_bytes":77071670002,"total_data_set_size_in_bytes":77071670002,"reserved_in_bytes":0}}},"unix-mail-relays-2025-03":{"uuid":"czifaLHST9iR9JLG3w5GMQ","primaries":{"docs":{"count":28285574,"deleted":0},"store":{"size_in_bytes":7995510472,"total_data_set_size_in_bytes":7995510472,"reserved_in_bytes":0}},"total":{"docs":{"count":56571148,"deleted":0},"store":{"size_in_bytes":15991020944,"total_data_set_size_in_bytes":15991020944,"reserved_in_bytes":0}}},"dns-queries-internal-2024.12.28":{"uuid":"7YwjWq89TZOmMceVKIkcZQ","primaries":{"docs":{"count":119025500,"deleted":0},"store":{"size_in_bytes":23616249204,"total_data_set_size_in_bytes":23616249204,"reserved_in_bytes":0}},"total":{"docs":{"count":238051000,"deleted":0},"store":{"size_in_bytes":47232498408,"total_data_set_size_in_bytes":47232498408,"reserved_in_bytes":0}}},"netflow-2024.12.29":{"uuid":"kx-sQtjXQf2WcZjm9DSSdQ","primaries":{"docs":{"count":445524647,"deleted":0},"store":{"size_in_bytes":178905504486,"total_data_set_size_in_bytes":178905504486,"reserved_in_bytes":0}},"total":{"docs":{"count":891049294,"deleted":0},"store":{"size_in_bytes":357811008972,"total_data_set_size_in_bytes":357811008972,"reserved_in_bytes":0}}},"firepower-firewall-2024.09.09-avoidclosed-c2024.12.10c":{"uuid":"O2WdCX2-RPWyzoB_sSmATg","primaries":{"docs":{"count":14,"deleted":0},"store":{"size_in_bytes":375630,"total_data_set_size_in_bytes":375630,"reserved_in_bytes":0}},"total":{"docs":{"count":28,"deleted":0},"store":{"size_in_bytes":751260,"total_data_set_size_in_bytes":751260,"reserved_in_bytes":0}}},"linux-syslog8-2025.01.07":{"uuid":"0VGgUyrnQmWC0-IRYzIALA","primaries":{"docs":{"count":112191412,"deleted":0},"store":{"size_in_bytes":40542661766,"total_data_set_size_in_bytes":40542661766,"reserved_in_bytes":0}},"total":{"docs":{"count":224382824,"deleted":0},"store":{"size_in_bytes":81085396097,"total_data_set_size_in_bytes":81085396097,"reserved_in_bytes":0}}},"radius-auth-detail-2025.02.03":{"uuid":"qqZF4mCeTAuFE2LQpTaIPQ","primaries":{"docs":{"count":1104844,"deleted":0},"store":{"size_in_bytes":878282485,"total_data_set_size_in_bytes":878282485,"reserved_in_bytes":0}},"total":{"docs":{"count":2209688,"deleted":0},"store":{"size_in_bytes":1764211253,"total_data_set_size_in_bytes":1764211253,"reserved_in_bytes":0}}},"sql-2024.11.27":{"uuid":"nBeYWWoIQyybjyTM8XH26Q","primaries":{"docs":{"count":24020392,"deleted":0},"store":{"size_in_bytes":14544372862,"total_data_set_size_in_bytes":14544372862,"reserved_in_bytes":0}},"total":{"docs":{"count":48040784,"deleted":0},"store":{"size_in_bytes":29088808966,"total_data_set_size_in_bytes":29088808966,"reserved_in_bytes":0}}},"firewall-prod-2025.02.04":{"uuid":"S9ruSkkXQ1uxc3mNe5-QdQ","primaries":{"docs":{"count":14510997,"deleted":0},"store":{"size_in_bytes":2478047676,"total_data_set_size_in_bytes":2478047676,"reserved_in_bytes":0}},"total":{"docs":{"count":29223050,"deleted":0},"store":{"size_in_bytes":5535264048,"total_data_set_size_in_bytes":5535264048,"reserved_in_bytes":0}}},"firewall-2025.02.03":{"
10150-l2vni-firewall-allocation:
action: allocation
description: "Apply shard allocation filtering rules to
l2vni-firewall indexes"
options:
ignore_empty_list: True
continue_if_exception: True
timeout_override: 300
allocation_type: include
key: datatype
value: cold
filters:
- filtertype: pattern
kind: prefix
value: l2vni-firewall-
- filtertype: age
source: field_stats
field: '@timestamp'
direction: older
stats_result: min_value
unit: days
unit_count: 31
uuid":"BbIP26MdQiSPAz82HSx-pA","primaries":{"docs":{"count":235527293,"deleted":0},"store":{"size_in_bytes":78574288061,"total_data_set_size_in_bytes":78574288061,"reserved_in_bytes":0}},"total":{"docs":{"count":471054586,"deleted":0},"store":{"size_in_bytes":157311318883,"total_data_set_size_in_bytes":157311318883,"reserved_in_bytes":0}}},"audit-2024-21":{"uuid":"_rci31NlTDaEIuDqpf05og","primaries":{"docs":{"count":6343606,"deleted":0},"store":{"size_in_bytes":3468646052,"total_data_set_size_in_bytes":3468646052,"reserved_in_bytes":0}},"total":{"docs":{"count":12687212,"deleted":0},"store":{"size_in_bytes":6937292104,"total_data_set_size_in_bytes":6937292104,"reserved_in_bytes":0}}},"storage-eventlog-2025.01.14":{"uuid":"pPSCYXvOSfiNVb3cCxyhhw","primaries":{"docs":{"count":13249695,"deleted":0},"store":{"size_in_bytes":12163144063,"total_data_set_size_in_bytes":12163144063,"reserved_in_bytes":0}},"total":{"docs":{"count":26499390,"deleted":0},"store":{"size_in_bytes":24326589286,"total_data_set_size_in_bytes":24326589286,"reserved_in_bytes":0}}},"dns-queries-external-2024.11.27":{"uuid":"lmPZxZlFR9KuQwOT5zWKTQ","primaries":{"docs":{"count":15729937,"deleted":0},"store":{"size_in_bytes":2957562022,"total_data_set_size_in_bytes":2957562022,"reserved_in_bytes":0}},"total":{"docs":{"count":31459874,"deleted":0},"store":{"size_in_bytes":5915124044,"total_data_set_size_in_bytes":5915124044,"reserved_in_bytes":0}}},"vpn-2025.01.08":{"uuid":"zLZ8e5NYRwunw99v6SfB6A","primaries":{"docs":{"count":124377,"deleted":0},"store":{"size_in_bytes":20691227,"total_data_set_size_in_bytes":20691227,"reserved_in_bytes":0}},"total":{"docs":{"count":248754,"deleted":0},"store":{"size_in_bytes":41382454,"total_data_set_size_in_bytes":41382454,"reserved_in_bytes":0}}},"firewall-prod-2024.11.07":{"uuid":"GMTuz_SpQjuuOta6Ac8zwQ","primaries":{"docs":{"count":230849767,"deleted":0},"store":{"size_in_bytes":38382817344,"total_data_set_size_in_bytes":38382817344,"reserved_in_bytes":0}},"total":{"docs":{"count":461699534,"deleted":0},"store":{"size_in_bytes":76765634688,"total_data_set_size_in_bytes":76765634688,"reserved_in_bytes":0}}},"application-2025.02.03":{"uuid":"tBKFQ4_0SOuT6vNVPsHbkw","primaries":{"docs":{"count":47634238,"deleted":0},"store":{"size_in_bytes":9336789319,"total_data_set_size_in_bytes":9336789319,"reserved_in_bytes":0}},"total":{"docs":{"count":95268476,"deleted":0},"store":{"size_in_bytes":18703040423,"total_data_set_size_in_bytes":18703040423,"reserved_in_bytes":0}}},"linux-syslog8-2024.12.15":{"uuid":"bQbTRu96TlSGOZa9zhJjIQ","primaries":{"docs":{"count":134796437,"deleted":0},"store":{"size_in_bytes":45274721155,"total_data_set_size_in_bytes":45274721155,"reserved_in_bytes":0}},"total":{"docs":{"count":269592874,"deleted":0},"store":{"size_in_bytes":90549442310,"total_data_set_size_in_bytes":90549442310,"reserved_in_bytes":0}}},"application-2025.01.30":{"uuid":"_Gz8-74lTM-RwYdaax3BzA","primaries":{"docs":{"count":44534761,"deleted":0},"store":{"size_in_bytes":9240545268,"total_data_set_size_in_bytes":9240545268,"reserved_in_bytes":0}},"total":{"docs":{"count":89069522,"deleted":0},"store":{"size_in_bytes":18481090536,"total_data_set_size_in_bytes":18481090536,"reserved_in_bytes":0}}},"access-2025.01.30":{"uuid":"zze2CZYxRxOoGyzXs4MiMA","primaries":{"docs":{"count":74898450,"deleted":0},"store":{"size_in_bytes":66981289283,"total_data_set_size_in_bytes":66981289283,"reserved_in_bytes":0}},"total":{"docs":{"count":149796900,"deleted":0},"store":{"size_in_bytes":133962578566,"total_data_set_size_in_bytes":133962578566,"reserved_in_bytes":0}}},".reporting-2024-01-21":{"uuid":"NJZ3JBL0TVSYjLJATQmTwQ","primaries":{"docs":{"count":1,"deleted":0},"store":{"size_in_bytes":1649381,"total_data_set_size_in_bytes":1649381,"reserved_in_bytes":0}},"total":{"docs":{"count":2,"deleted":0},"store":{"size_in_bytes":3298762,"total_data_set_size_in_bytes":3298762,"reserved_in_bytes":0}}},"netflow-2024.12.22":{"uuid":"2Zfh2hKyS0OCWH7TjRL8tQ","primaries":{"docs":{"count":425206060,"deleted":0},"store":{"size_in_bytes":167169449358,"total_data_set_size_in_bytes":167169449358,"reserved_in_bytes":0}},"total":{"docs":{"count":850412120,"deleted":0},"store":{"size_in_bytes":334338898716,"total_data_set_size_in_bytes":334338898716,"reserved_in_bytes":0}}},"linux-syslog8-2024.11.06":{"uuid":"MZWskopMQYCMEubBEPHzZQ","primaries":{"docs":{"count":73749834,"deleted":0},"store":{"size_in_bytes":28955141534,"total_data_set_size_in_bytes":28955141534,"reserved_in_bytes":0}},"total":{"docs":{"count":147499668,"deleted":0},"store":{"size_in_bytes":57910283068,"total_data_set_size_in_bytes":57910283068,"reserved_in_bytes":0}}},"sits-httpd-error-2025.02":{"uuid":"mYnPfW1YR12liDrd3kRBbA","primaries":{"docs":{"count":4567,"deleted":0},"store":{"size_in_bytes":908574,"total_data_set_size_in_bytes":908574,"reserved_in_bytes":0}},"total":{"docs":{"count":8558,"deleted":0},"store":{"size_in_bytes":1908387,"total_data_set_size_in_bytes":1908387,"reserved_in_bytes":0}}},"linux-syslog-2024.11.28":{"uuid":"NxASNTa6QpetEKVNmQpsqQ","primaries":{"docs":{"count":9927384,"deleted":0},"store":{"size_in_bytes":1624762797,"total_data_set_size_in_bytes":1624762797,"reserved_in_bytes":0}},"total":{"docs":{"count":19854768,"deleted":0},"store":{"size_in_bytes":3249525594,"total_data_set_size_in_bytes":3249525594,"reserved_in_bytes":0}}},"netflow-2024.12.21":{"uuid":"Zb0ET8KNRq2K47Oix-ZCxg","primaries":{"docs":{"count":430155181,"deleted":0},"store":{"size_in_bytes":169464771262,"total_data_set_size_in_bytes":169464771262,"reserved_in_bytes":0}},"total":{"docs":{"count":860310362,"deleted":0},"store":{"size_in_bytes":338929542524,"total_data_set_size_in_bytes":338929542524,"reserved_in_bytes":0}}}}}
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index: linux-syslog8-2025.02.02 Size: 52.3GB
Docs: 98655426 PrimarySize: 26.1GB
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index: application-2024.11.29
Size: 14.4GB Docs: 80757796 PrimarySize: 7.2GB
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index:
storage-eventlog-2024.09.30-avoidclosed-c2025.01.23c Size: 2.3MB Docs:
2258 PrimarySize: 1.1MB
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index: dns-queries-external-2024.12.21
Size: 4.6GB Docs: 24780756 PrimarySize: 2.3GB
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index: dns-queries-external-2024.12.26
Size: 4.8GB Docs: 25470510 PrimarySize: 2.4GB
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index: storage-eventlog-2024.11.25 Size: 23.0GB
Docs: 27632500 PrimarySize: 11.5GB
2025-02-04 01:28:32,993 DEBUG curator.indexlist
iterate_over_stats:127 Index: firewall-2024.12.06 Size:
142.3GB Docs: 450662242 PrimarySize: 71.2GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: linux-syslog8-2024.12.03 Size: 86.6GB
Docs: 265287746 PrimarySize: 43.3GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: linux-syslog8-2024.12.05 Size: 75.8GB
Docs: 221722984 PrimarySize: 37.9GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: sql-2024.12.10 Size:
27.5GB Docs: 46553418 PrimarySize: 13.8GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: .kibana-ws_7.17.1_001 Size: 4.8MB Docs:
66 PrimarySize: 2.4MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index:
eventlog-2024.12.16 Size: 41.4GB Docs:
37131536 PrimarySize: 20.7GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index:
eventlog-2024.12.12 Size: 47.5GB Docs:
42937316 PrimarySize: 23.8GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: dhcp-2025.01.28 Size: 9.6GB
Docs: 54993768 PrimarySize: 4.8GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: dhcp-2024.12.05 Size: 10.1GB
Docs: 57683668 PrimarySize: 5.0GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: dns-queries-external-2025.01.07
Size: 6.0GB Docs: 33679130 PrimarySize: 3.0GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: radius-auth-detail-2025.01.17 Size:
1.6GB Docs: 2143848 PrimarySize: 831.3MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: audit-2024-02 Size:
4.4GB Docs: 9336846 PrimarySize: 2.2GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: vpn-2025.01.29 Size: 38.7MB
Docs: 244204 PrimarySize: 19.3MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: radius-log-2024.12.06 Size: 76.2MB
Docs: 261164 PrimarySize: 38.1MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: application-2025.01.16
Size: 15.6GB Docs: 89139788 PrimarySize: 7.8GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index:
firewall-prod-2024.11.24 Size: 74.8GB Docs:
494095544 PrimarySize: 37.4GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: vpn-2025.01.24 Size: 44.4MB
Docs: 280646 PrimarySize: 22.2MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index:
eventlog-2025.01.09 Size: 43.0GB Docs:
40298180 PrimarySize: 21.5GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: .reporting-2022-09-11 Size: 35.2KB
Docs: 0 PrimarySize: 17.6KB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: httpd-access-2024-25
Size: 3.5GB Docs: 7015310 PrimarySize: 1.7GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: .reporting-2022-12-18 Size:
19.9MB Docs: 4 PrimarySize: 9.9MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: reports-2025-03
Size: 313.1MB Docs: 424988 PrimarySize: 156.6MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index:
eventlog-2024.12.04 Size: 45.9GB Docs:
42025102 PrimarySize: 23.0GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index:
storage-eventlog-2024.08.24-avoidclosed-c2024.12.19c Size: 1.1MB Docs:
388 PrimarySize: 541.2KB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: unix-mail-relays-2024-48 Size: 15.7GB
Docs: 56902738 PrimarySize: 7.9GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: storage-freenas-2025.01.23 Size: 288.4MB
Docs: 1656200 PrimarySize: 144.2MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: applicationBasically Sweng -2024.11.06
Size: 22.6GB Docs: 89939756 PrimarySize: 11.3GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: dhcp-2024.11.23 Size: 5.6GB
Docs: 31723932 PrimarySize: 2.8GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: sql-2024.12.09 Size:
21.7GB Docs: 39120124 PrimarySize: 10.8GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: linux-httpd-access-2024-45 Size: 1.8GB
Docs: 3916084 PrimarySize: 916.5MB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: storage-eventlog-2025.01.04 Size: 10.7GB
Docs: 15224188 PrimarySize: 5.3GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: firewall-2024.11.14 Size:
140.9GB Docs: 462413158 PrimarySize: 70.5GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: .reporting-2023-05-21
Size: 114.8KB Docs: 0 PrimarySize: 57.4KB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: firepower-firewall-2024.11.08
Size: 1.0TB Docs: 4616034386 PrimarySize: 534.3GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: firepower-firewall-2025.01.12
Size: 942.6GB Docs: 4051662384 PrimarySize: 471.3GB
2025-02-04 01:28:32,994 DEBUG curator.indexlist
iterate_over_stats:127 Index: firewall-2024.12.28 Size:
87.3GB Docs: 270646394 PrimarySize: 43.6GB
2025-02-04 01:28:32,994 ERROR curator.cli
run:211 Failed to complete action: allocation. <class 'KeyError'>:
'store'
2025-02-04 01:28:32,994 INFO curator.cli
run:216 Continuing execution with next action because
"continue_if_exception" is set to True for action allocation
2025-02-04 01:28:33,004 INFO curator.cli
run:223 Action ID: 10150-l2vni-firewall-allocation,
"allocation" completed.
I see no explanation there for why it failed other than '<class
'KeyError'>: 'store'' from which I am unable to derive anything meaningful.
Unlike with
there are no instances of a GET request Curator makes failing with
log_request_fail in the DEBUG log.
[curator@foo.bar:production:~]$ zgrep -c log_request_success
debugrun_1738629001.log.gz
22620
[curator@foo.bar:production:~]$ zgrep -c failing debugrun_1738629001.log.gz
0
[curator@foo.bar:production:~]$
Has anyone seen this happen and/or can offer can insight on why it happens?