Logstash Error 403

(Daniel Gutierrez) #1

Hi everyone
Logstash keeps crashing with 403 error, I know this error its caused by no available space on disk, on that moment I had 30 Gb, now I attached 100 Gb on disk and keeps crashing
I tried removing Logstash, installing another versions but keeps appering same error
Can someone help this poor blind men?

I have:
4 CPU
16 VCPU
32 GB RAM
130 GB STORAGE
CentOS 7

#2

What does your configuration file look like, and exactly what error message are you getting?

(Daniel Gutierrez) #3

I have installed ElasticStack since a few months and ieverything was run perfectly
On this week im with those errors

Error
Feb 11 14:00:27 elastic-stack logstash[4197]: [2019-02-11T14:00:27,920][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

Configuration Files
/etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

/etc/logstash/conf.d/10-syslog-filter.conf

filter {
if [fileset][module] == "system" {
if [fileset][name] == "auth" {
grok {
match => { "message" => ["%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DAT$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: %{DATA:[system][auth$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sshd(?:[%{POSINT:[system][auth][pid]}])?: Did not receive iden$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} sudo(?:[%{POSINT:[system][auth][pid]}])?: \s*%{DATA:[system][a$
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} groupadd(?:[%{POSINT:[system][auth][pid]}])?: new group: name= "%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} useradd(?:\[%{POSINT:[system][auth][pid]}\])?: new user: name=%{
"%{SYSLOGTIMESTAMP:[system][auth][timestamp]} %{SYSLOGHOST:[system][auth][hostname]} %{DATA:[system][auth][program]}(?:[%{POSINT:[system][auth][pid] pattern_definitions => { "GREEDYMULTILINE"=> "(.|\n)*" } remove_field => "message" } date { match => [ "[system][auth][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } geoip { source => "[system][auth][ssh][ip]" target => "[system][auth][ssh][geoip]" } } else if [fileset][name] == "syslog" { grok { match => { "message" => ["%{SYSLOGTIMESTAMP:[system][syslog][timestamp]} %{SYSLOGHOST:[system][syslog][hostname]} %{DATA:[system][syslog][program]}(?:\[%{POSIN
pattern_definitions => { "GREEDYMULTILINE" => "(.|\n)*" }
remove_field => "message"
}
date {
match => [ "[system][syslog][timestamp]", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
}

/etc/logstash/conf.d/30-elasticsearch-output.conf

output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}

Thanks for the support

#4

OK, so you ran out of disk space and elasticsearch put the indexes into read-only mode. Once you have added or freed up disk space you need to tell elasticsearch to enable read-write. See this post for how to do that.

(Daniel Gutierrez) #5

Oh dude! Thanks, Im on it!
THANKS

(Daniel Gutierrez) #6

Hello again, im running next command but im receiving the next error:

Error:
curl: (3) [globbing] nested braces not supported at pos 27

Command:
curl -X PUT "localhost:9200/filebeat-%{[@metadata][version]}-2019.02.07/_settings" -H 'Content-Type: application/json' -d'
{
"index.blocks.read_only_allow_delete": null
}
'

Can you please tell me what im doing wrong?
Thanks bro

#7

In this context, there is no metadata to do a substition from. Check your index name, it will be something like filebeat-6.6.0-2019.02.07 and use that name in the curl command.

Oh, and if filebeat-%{[@metadata][version]}-2019.02.07 really is the literal name of your index then you may need to use backslash to escape some of the characters ({} and/or [ ]).

(dave) #8

In the event that you have to do date math, you should cast the field esteems to date types in Painless. Regularly students ask me to do my homework for me. Who interested, welcome

(Daniel Gutierrez) #9

Its done! Thanks bro, a lot!!!!!! <3

(system) closed #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.