Upgrade from 1.5 to 2.0: Pattern not defined

I am upgrading from the 1.5 branch to 2.0. I am getting the following errors on startup:

"Default settings used: Filter workers: 1
The error reported is:
pattern %{HOST:thehost} not defined

I think %{HOST} is a default pattern included with logstash, so it is not a case of having a bad patterns directory. No problems with 1.5.

What's your config look like?

The host configuration option is now hosts in logstash 2.0

The config is fairly large to copy and paste and it isn't giving me a line number. And if I try it with --configtest, it surprisingly passes.

I did remove the entire filter {} section and it worked so it is not happy with something there.

I added this to the beginning of every grok stanza, to be safe, following an issue I found on github:

patterns_dir => "/etc/logstash/patterns"

I've already updated to change "host" to "hosts" since I saw that in the incompatible changes. That's not the problem here, but thanks for the suggestion.

Might it have anything to do with this? Grok -> patterns_dir not working in logstash 1.5.0 Do I need to install a plugin to access the patterns included with logstash by default?

Edit: confirmed that %{HOST} is a built-in pattern (or at least grok debugger says it is): https://grokdebug.herokuapp.com/patterns# (look under grok-patterns)

Edit2: I am not confident that it is related but the logstash log is filling up with permission denied errors like this one for local files:

{:timestamp=>"2015-11-11T06:28:13.624000-0500", :message=>"failed to open /var/log/syslog: Permission denied - /var/log/syslog", :level=>:warn}

If you start Logstash with --debug it'll tell you about all pattern files and patterns that it reads. That might give you some clues.

Thanks for the tip.

I used --debug but it was a bit overwhelming and still did not display a line number or greater information for the error. I did manage to find that it was actually loading the default patterns from somewhere in /opt/logstash, making the error that much more strange. Is there anyway to force a line number for this error? It doesn't seem like a normal error, especially since I have encountered grok errors before and they are pretty self-explanatory to debug.

Were you able to figure it out? I am having the same issue.

%{HOST:domain} => %{IPORHOST:domain} fix my problem.