Hi,
I have an ELK stack with a hook to slack
The logstash filter is the following
if [message] =~ "Reallocated_Sector_Count" or [message] =~ "Reallocated_Sector_Ct"
or [message] =~ "Reported_Uncorrectable_Errors" or [message] =~ "Reported_Uncorrect"
or [message] =~ "Command_Timeout"
or [message] =~ "Current_Pending_Sector_Count" or [message] =~ "Current_Pending_Sector"
or [message] =~ "Offline_Uncorrectable" {
# https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/
# SMART 5 - Reallocated_Sector_Count / Reallocated_Sector_Ct
# SMART 187 - Reported_Uncorrectable_Errors / Reported_Uncorrect
# SMART 188 - Command_Timeout
# SMART 197 - Current_Pending_Sector_Count / Current_Pending_Sector
# SMART 198 - Offline_Uncorrectable
mutate{
replace => { "log_level" => "WARNING" }
}
}
and in elastalert I have
filter:
- query_string:
query: "log_level:CRIT OR log_level: CRITICAL OR log_level:WARN OR log_level: WARNING OR log_level: ERROR OR log_level: FATAL"
alert:
- "slack"
- "command"
command: "curl -s -H 'content-type: application/json' -XPOST 'http://logstash:8080/' -d '{ \"@timestamp\": \"%(@timestamp)s\", \"host\": \"%(host)s\", \"container_name\": \"%(container_name)s\", \"log_level\": \"%(log_level)s\", \"message\": \"%(message)s\" }' > /dev/null"
alert_text: "{0} -- {1} -- {2} -- {3} -- {4}"
alert_text_args: ["@timestamp", "host", "container_name", "log_level", "message"]
alert_text_type: alert_text_only
slack:
slack_webhook_url: something
slack_channel_override: "somechanel"
slack_emoji_override: "space_invader"
slack_username_override: "Log - Alerts (ELK)"
I then do
echo "20171129 THIS IS MY OWN TEST for Reallocated_Sector_Count" >> /var/log/syslog
echo "20171129 THIS IS MY OWN TEST for Reported_Uncorrectable_Errors" >> /var/log/syslog
echo "20171129 THIS IS MY OWN TEST for Current_Pending_Sector_Count" >> /var/log/syslog
echo "20171129 THIS IS MY OWN TEST for Offline_Uncorrectable" >> /var/log/syslog
...and a few more
this all hits the syslog fine and I can also see it in kibana, but slack just shows randomly 1-3 of those messages sent. Any ideas what it could be?
Thanks