Hi everyone
Logstash keeps crashing with 403 error, I know this error its caused by no available space on disk, on that moment I had 30 Gb, now I attached 100 Gb on disk and keeps crashing
I tried removing Logstash, installing another versions but keeps appering same error
Can someone help this poor blind men?
I have:
4 CPU
16 VCPU
32 GB RAM
130 GB STORAGE
CentOS 7
What does your configuration file look like, and exactly what error message are you getting?
I have installed ElasticStack since a few months and ieverything was run perfectly
On this week im with those errors
Error
Feb 11 14:00:27 elastic-stack logstash[4197]: [2019-02-11T14:00:27,920][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})
Configuration Files
/etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
/etc/logstash/conf.d/10-syslog-filter.conf
filter {
if [fileset][module] == "system" {
if [fileset][name] == "auth" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DAT$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive iden$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: \s*%{DATA:[system][a$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name=
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid]
pattern_definitions => {
"GREEDYMULTILINE"=> "(.|\n)*"
}
remove_field => "message"
}
date {
match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
geoip {
source => "[system][auth][ssh][ip]"
target => "[system][auth][ssh][geoip]"
}
}
else if [fileset][name] == "syslog" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSIN
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
remove_field => "message"
}
date {
match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
}
/etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
Thanks for the support
OK, so you ran out of disk space and elasticsearch put the indexes into read-only mode. Once you have added or freed up disk space you need to tell elasticsearch to enable read-write. See this post for how to do that.
Hello again, im running next command but im receiving the next error:
Error:
curl: (3) [globbing] nested braces not supported at pos 27
Command:
curl -X PUT "localhost:9200/filebeat-%{[@metadata][version]}-2019.02.07/_settings" -H 'Content-Type: application/json' -d'
{
"index.blocks.read_only_allow_delete": null
}
'
Can you please tell me what im doing wrong?
Thanks bro
In this context, there is no metadata to do a substition from. Check your index name, it will be something like filebeat-6.6.0-2019.02.07 and use that name in the curl command.
Oh, and if filebeat-%{[@metadata][version]}-2019.02.07 really is the literal name of your index then you may need to use backslash to escape some of the characters ({} and/or [ ]).
In the event that you have to do date math, you should cast the field esteems to date types in Painless. Regularly students ask me to do my homework for me. Who interested, welcome