Upgrade from 1.5 to 2.0: Pattern not defined

Elastic Stack Logstash
1

I am upgrading from the 1.5 branch to 2.0. I am getting the following errors on startup:

"Default settings used: Filter workers: 1
The error reported is:
pattern %{HOST:thehost} not defined
"

I think %{HOST} is a default pattern included with logstash, so it is not a case of having a bad patterns directory. No problems with 1.5.

2

What's your config look like?

3

The host configuration option is now hosts in logstash 2.0
https://www.elastic.co/guide/en/logstash/current/breaking-changes.html

4

The config is fairly large to copy and paste and it isn't giving me a line number. And if I try it with --configtest, it surprisingly passes.

I did remove the entire filter {} section and it worked so it is not happy with something there.

I added this to the beginning of every grok stanza, to be safe, following an issue I found on github:

patterns_dir => "/etc/logstash/patterns"

I've already updated to change "host" to "hosts" since I saw that in the incompatible changes. That's not the problem here, but thanks for the suggestion.

Might it have anything to do with this? Grok -> patterns_dir not working in logstash 1.5.0 Do I need to install a plugin to access the patterns included with logstash by default?

Edit: confirmed that %{HOST} is a built-in pattern (or at least grok debugger says it is): https://grokdebug.herokuapp.com/patterns# (look under grok-patterns)

Edit2: I am not confident that it is related but the logstash log is filling up with permission denied errors like this one for local files:

{:timestamp=>"2015-11-11T06:28:13.624000-0500", :message=>"failed to open /var/log/syslog: Permission denied - /var/log/syslog", :level=>:warn}
5

If you start Logstash with --debug it'll tell you about all pattern files and patterns that it reads. That might give you some clues.

6

Thanks for the tip.

I used --debug but it was a bit overwhelming and still did not display a line number or greater information for the error. I did manage to find that it was actually loading the default patterns from somewhere in /opt/logstash, making the error that much more strange. Is there anyway to force a line number for this error? It doesn't seem like a normal error, especially since I have encountered grok errors before and they are pretty self-explanatory to debug.

7

Were you able to figure it out? I am having the same issue.

8

%{HOST:domain} => %{IPORHOST:domain} fix my problem.