Can't load an invalid configuration (:reason=>"Expected one of #

Elastic Stack Logstash
1

Hi,

I have now figured out a little bit more about my problems.

I get the following error in: /var/log/logstash:

Error message:
[2017-03-05T00:01:22,530][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, { at line 20, column 14 (byte 303) after filter { \n if [type] == "syslog" {\n #change to pfSense ip address\n if [host] =~ /192\.168\.0\.1/ {\n mutate "}

So I trying to get this blog entry to work at home. http://pfelk.3ilson.com/2016/11/kibanaelasticsearchlogstash-plugin-x.html#gpluscomments

I have copy pasted all configuration files. I have pasted here my 10-syslog.conf I have added there the adress of my Pfsense.
10-syslog.conf
http://pastebin.com/xecU0LY9

Maybe someone can help me?

2

Can you pastebin the entire config?

3

Hi,

Yes of course. In the following pastebin you can find all other files i created.
File 4 is the grok file.

http://pastebin.com/62FQkNKr

4

I can't see anything obviously wrong. Comment out parts of the files to narrow things down. Use a tool like hexdump to make sure you don't have any unexpected non-printable characters in there.

5

Hmm there i have the problem. I don't really understand the grokking patterns and the config files from Logstash. So I don't know how I can do that what you want me to do.

So if someone has some hints for me concerning commenting out which parts and test, I'm open to do it.

In Kibana i have the following sentence: unable to fetch mapping. Do you have indices matching the pattern?

6

I found a mistake in a configuration file. Now i get still an error:

[2017-03-08T14:57:19,685][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-03-08T14:59:49,986][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, input, filter, output at line 119, column 1 (byte 2914) after "}

How can I find line 119? Which file?

7

It merges all the config files into a single one.

8

So i have identyfied the following line in the middle of those 3 lines in the grok file as the bad line:

PFSENSE_IP_SPECIFIC_DATA (%{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA})
PFSENSE_IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),
PFSENSE_IPv4_SPECIFIC_DATA_ECN (%{BASE16NUM:tos}),(%{INT:ecn}),(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}),

Does someone can check this?

9

You may want to look at the --config.debug option as well.

10

Are you using your grok file into conf.d directory ? Probably this is causing this error.

Try to copy your grok file to /etc/logstash/conf.d/patterns and leave only conf files into conf.d directory.

Hope it helps.

Thanks

11

The grok file is in the directory patterns alone. All config files are in the conf.d folder.

I probably have to watch with the --config.debug option. At the moment i don't know how to do that exactly on centos 7.

12

You problably need something like

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ --config.debug

But maybe you are adding your config into some editor that change quotes for example and break your config.

Hope it helps.

Regards,

13

I receive the following anser:

[root@logs logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ --config.debug
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
19:30:48.608 [LogStash::Runner] ERROR logstash.agent - Cannot load an invalid configuration {:reason=>"Expected one of #, input, filter, output at line 115, column 1 (byte 2909) after "}

A little bit strange it looks like it worked well but he didn't find logstash.yml. In the ende it shows the same error. I use vi to create the files.

14

19:30:48.608 [LogStash::Runner] ERROR logstash.agent - Cannot load an invalid configuration {:reason=>"Expected one of #, input, filter, output at line 115, column 1 (byte 2909) after "}

Debug this by commenting out lines near line 115 (unfortunately this probably means line 115 in the in-memory concatenation of all config files) until the problem disappears. If you have multiple files you can run them through --configtest individually. Be systematic.

15

Line 115 is in the grok file. So i comment out and startet logstash a few times and i also started the server new. I tried that with different lines. It always shows me the same error in: sudo vi logstash-plain.log

[2017-03-09T20:03:22,558][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, input, filter, output at line 115, column 1 (byte 2909) after "}

Can it be something other than the files?

Or where does he starts to pick the files and set them together?
That's how i see it in ssh: 01-inputs.conf 10-syslog.conf 11-pfsense.conf 30-outputs.conf patterns
So for me the first file is: 01-inputs.conf but maybe it's different for logstash.

16

Line 115 is in the grok file.

You have a grok pattern file in /etc/logstash/conf.d. Don't do that. Logstash reads all files in that directory. Put the file in e.g. /etc/logstash instead.

17

In most cases we see this error due to some syntax issue with the config file. This is what happened in my case. The blog below talks about it in more details and also has information about troubleshooting few other common issues.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.