Pipeline error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `close' for nil:NilClass>

Elastic Stack Logstash
1

image
image1679×452 72.3 KB

I have that error in logstash logs when logstash starting. Here my config file for logstash:

input {
  beats {
    port => 5044
  }
}
filter {
    grok {
      patterns_dir => ["/etc/logstash/pattern"]
      match => { "message" => "%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response}" }
    }
}
output {
  elasticsearch {
      hosts => ["127.0.0.1:9200"]
      index => "elk_fb_logstash-%{+YYYY.MM.dd}"
  }
}

Pls, can you help me to solve this problem.

2

That exception occurs if the grok filter is unable to open a pattern file. Check the patterns_dir option and the permissions on the files there.

3

I gave the permission to the pattern file by command chmod -R 777, but this error doesn't dissapear. And where can I check the patterns_dir option?

4

I am saying to check that this is the correct directory. It is a directory, right?

5

Yes, this is my pattern directory.

6

By the way also I used such command as chown logstash: /etc/logstash/pattern to to my pattern file in this directory.

7

Set the log.level to trace and grok will tell you which file it is trying to read.

9

Please do not post pictures of text, they are hard to read, impossible to search, and some folk may not be able to see them at all.

11

That is log.level INFO, not log.level TRACE.

12

But I changed it to trace here in logstash.yml:

# ------------ Debugging Settings --------------
#
# Options for log.level:
#   * fatal
#   * error
#   * warn
#   * info (default)
#   * debug
#   * trace
#
 log.level: trace
path.logs: /var/log/logstash
#
13

Hey Dark_Man

try removing the space before log.level in your config, remember this is YAML configuration files they are sensible to spaces and tabulations.

15

How do you start logstash ?

16

I use such kind of commands:

systemctl restart logstash
systemctl status logstash
cat /var/log/logstash/logstash-plain.log
17

If logstash started without an exception when you had a single space in front of that logstash.yml entry then it seems to me unlikely that it was reading the file you edited, and was in fact using a different copy.

18

What do you have inside of /etc/logstash/patterns, one file? multiple files?

Can you share the content of the files inside the pattern dir?

19

@leandrojmp at pattern directory I've created one file named as nginx, here is content of this file:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
20

@Badger Do you mean that I need to put a single space before log.level: trace at logstash.yml?

21

No, I am saying that if you edited the logstash.yml in that way and it had no effect on logstash then it seems likely that logstash is not using the logstash.yml that you think it is.

22

@Badger Ok, I understand what you mean, but then what file is logstash using instead of logstash.yml? Maybe you know, how can I check it?

23

logstash.yml is loaded from path.settings, which could use the default, could be set on the command line using --path.settings, or could be set in the environment using $LS_PATH_SETTINGS.

If I remember correctly then --config.debug will print the value of path.settings that logstash is using.