Pipeline error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `close' for nil:NilClass>

Elastic Stack Logstash
24

@Badger I found the parameters what you say about, they're in file named as startup.options:


# Set a home directory
LS_HOME=/usr/share/logstash

# logstash settings directory, the path which contains logstash.yml
LS_SETTINGS_DIR=/etc/logstash

# Arguments to pass to logstash
LS_OPTS="--path.settings ${LS_SETTINGS_DIR}"

Do you mean that I need to change them?

25

No, you need to find out what actual values are being used.

26

Hello @Badger , can you tell me please, where can I find --config.debug parameter for print the value of path.settings that logstash is using?

27

It is a command line option.

29

-f (path.config) should point to the logstash configuration file, not the yml.

30

@Badger which file do you mean by logstash configuration file? Is it .conf file or another one?

31

Yes, the .conf file that defines the inputs, filters, and outputs.

32

@Badger I checked this command in my nginx.conf file, the output was:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-02-17 10:16:03.234 [main] runner - Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[WARN ] 2022-02-17 10:16:03.503 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-02-17 10:16:04.041 [LogStash::Runner] Reflections - Reflections took 57 ms to scan 1 urls, producing 119 keys and 417 values
[WARN ] 2022-02-17 10:16:04.940 [LogStash::Runner] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-02-17 10:16:04.985 [LogStash::Runner] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-02-17 10:16:05.138 [LogStash::Runner] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-02-17 10:16:05.174 [LogStash::Runner] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
Configuration OK
[INFO ] 2022-02-17 10:16:05.195 [LogStash::Runner] runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

I didn't see that I have any critical or syntax errors here. I only saw that it was ignoring pipelines.yml file, does it main problem? Or I have another problem that I couldn't see.

33

No, that's not a problem if you want to use the -f option. I suggested using --config.debug --log.level debug, not --config.test_and_exit.

34

@Badger I used the command --config.debug --log.level debug as you said, and do I need to wait until this command finish process? And what is this command do? What is purpose of it?

35

I have the same issue. Any tips for solution?

36

When you run logstash with --config.debug --log.level debug it will print the actual value of path.settings that it is using, which I think may be different to the value you expect, meaning it is not loading the logstash.yml you are editing.

38

One of the debug messages will contain the value of path.settings that it is using. You need to see if that is set to the directory that contains logstash.yml

40

If you used the options I suggested (--config.debug --log.level debug) then you should get dozens of messages from the logstash.runner class

[2022-02-20T22:38:48,414][DEBUG][logstash.runner ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2022-02-20T22:38:48,414][DEBUG][logstash.runner ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2022-02-20T22:38:48,414][DEBUG][logstash.runner ] xpack.monitoring.enabled: false

42

@Badger I found it:

[DEBUG] 2022-02-21 09:21:58.631 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"
[DEBUG] 2022-02-21 09:21:58.631 [LogStash::Runner] runner - path.logs: "/usr/share/logstash/logs"
[DEBUG] 2022-02-21 09:21:58.631 [LogStash::Runner] runner - xpack.monitoring.enabled: false

After I checked it by this path:

cat /usr/share/logstash/config
cat: /usr/share/logstash/config: No such file or directory

And there is no such config file.
so see that path.settings are placed here: /usr/share/logstash/config. So what I need to do next?

43

OK, let's go back to the beginning ... you were trying to change the log.level in logstash.yml, and you say you edited a file called that.

logstash is trying to read logstash.yml from /usr/share/logstash/config, which does not exist. On the command line you can set --path.settings to the directory in which you were editing your logstash.yml.

44

@Badger where can I set on the command line --path.settings to the directory in which you were editing your logstash.yml? Sorry, I didn't understand clearly what you mean.

45

You said you edited logstash.yml, if the path to that is /home/user/foo/logstash.yml then add --path.settings /home/user/foo to the command line. If it is /tmp/logstash.yml then add --path.settings /tmp

46

Ok, my logstash.yml placed at /etc/logstash, so at which command that I need to add to command line --path.settings /etc/logstash/logstash.yml?

47

No, it points to a directory. Try --path.settings /etc/logstash.