Invalid configuration "undefined method terminal"

Elastic Stack Logstash
1

I've been debugging my logstash config, and end up running into this:

root@indexerNode01:/opt/logstash/bin# ./logstash -t -f /etc/logstash/conf.d/filter.conf 
The given configuration is invalid. Reason: undefined method `terminal?' for #<LogStash::Config::AST::Attribute:0x13015762> {:level=>:fatal}

Googled turn up nothing on undefined method `terminal?' that I could find.

service logstash configtest gives me this:

root@indexerNode01:/opt/logstash/bin# service logstash configtest
The given configuration is invalid. Reason: undefined method `recursive_inject' for #<Treetop::Runtime::SyntaxNode:0x4baebf67> {:level=>:fatal}

To further confuse things, I have a second node configured identically to the first one. Both config test commands return "Configuration OK" when run on that node.

I use ansible to template the config files. I've even copied the filter.conf file from both locally and used Kompare to diff them. They were identical.

Both nodes are running Logstash 2.3.4 installed from the apt repos. The OS is Ubuntu 14.04.

My config is here: http://pastebin.com/f9sbthGU

Any suggestions?

2

And what does the configuration look like?

3

Oops... I totally forgot to link my pastebin... I just added it to the first post.

4

That's quite a lot so I didn't read it in detail, but nothing struck me as totally bonkers. I suggest you comment out pieces until you at least can narrow things down to a small number of lines.

Two things I did see that you should correct even if they're not the cause of your weird error message:

  • In a few places there's a space missing after if (e.g. if[main_message_two]).
  • In regexp match expressions in conditionals, the regexp is supposed to be delimited by slashes, not regular quotes.
5

I fixed the missing spaces.

For the regexp:

if [forwardedip] !~ /(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)/ {

instead of:

if [forwardedip] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)" {

Right?

Erm, the version with the quotes does work. It's been working for months. Do the regex's just need some kind of delimiter at either end, and it doesn't much matter what?

Neither change helped.

I did spend a while removing filters by [type] and the trying to start it. I got a bunch of different error messages. Most something about various undefined methods. But when I reached the point of filter { } I couldn't really do much else...

I also tried purging logstash and java-common, then reinstalling. It did not help.

The part that has me the most confused is that my other indexer node works just fine. And it is configured exactly the same as the one that is not working. I even tried copying the /etc/logstash directory from it to the broken node. It didn't help.

At this point I'm sure the issue is not my configuration. I think something on the vm got messed up. Any ideas what I could look for?

I'd just delete the vm and rebuild it, but I always prefer knowing what broke so I can fix it if it happens again.

6

Right?

Yes.

Erm, the version with the quotes does work. It's been working for months. Do the regex's just need some kind of delimiter at either end, and it doesn't much matter what?

It might totally work, but I don't know if it's supported or if it just happens to work.

But this one is weird indeed. I downloaded your config and my Logstash 2.3.4 found it acceptable.

7

Ah, ok.

Definitely weird. "undefined method" makes me think that the libraries or something are corrupted. But purging and reinstalling both Logstash and OpenJDK 7 should have fixed that. If it was something corrupted in RAM, then restarting should have helped...

Oh well, I'll poke around a bit more Monday, then I'll just delete the vm and build a new one.

8

Well, I deleted the vm, built a new one, and everything works just fine. No idea what what in the world was going on....

Thanks for the help! @magnusbaeck