I have a requirement to parse different type of log files (own Java application's log file) and catalina log files. Both of these have a different type of logging system, for which I have to use different type of multiline regex string.
I'm using filebeat on the client side to send these log files and I have setup a graylog filebeat-listener on the server side to receive these log files.
I have tested my filebeat configurations with several configuration files (trial and error editing of configurational parameters) but have not been able to figure out how to send these log files with a different multiline pattern and a different out put port (if possible).
this and it gave me "line 37: mapping values are not allowed in this context." error. Next I tried with
this , it gives me "line 31: found a tab character that violate indentation. Exiting." I've set the indentation as it is for the first prospectus. Yet, it gives me the errror . It works fine when I commented the whole of second prospectors.
Files from the first prospectors is being delivered as expected. But then my requirement is not fulfilled.
I would really be very greatful if somebdy could help me with this. I'm not so familiar with yaml or json.
beats is not supporting any kind of event routing yet. Output is one global option and can not be configured per prospector. All you configs are somewhat invalid.
Users normally go with a) run two filebeat instances with different config or b) forward to logstash, using logstash for routing events to different endpoints based on custom fields set in prospectors fields section. I don't know about graylog, though.
this , it gives me "line 31: found a tab character that violate indentation. Exiting." I've set the indentation as it is for the first prospectus. Yet, it gives me the errror . It works fine when I commented the whole of second prospectors.
From looks it might seem to be ok, but some editors mix spaces and tabs (some editors can display these in order to check indentation being correct).
After I read your reply, I tried different editors like vim and eclipse (with a proper plugin installed). Although, the error, yet, persists.
Its been two days and I have hard time figuring this out. It would really be helpful if somebody could help me with this or It would also be ok if someone could guide me or point me towards the configuring the second instance of filebeat on the same host.
I had a look at both your config files. In the first one you mix prospectors and outputs which is not possible. In addition in all files you have multiple registry_files. There is only one global registry file.
In the second file you have tabs on the lines indicated. You do not need any special editor, just get an editor that shows you spaces and tabs and fix it manually.
when using vim type :set list. This will display spaces, tabs and newline characters.
Maybe it's a good idea to start with sample config provided with filebeat (If you're using 5.0-alpha3 have a look at filebeat.full.yml) and make it work for your first prospector case first. The sample config contains all configs in right places including space, but no tabs. Just remove the # from configs you want to change. Note, the sample config contains default values for all config options. Once it works, copy your config and make it work for your second prospector type (just one prospector). Having 2 configs, you can configure 2 filebeat instances forwarding the logs to the correct endpoints.
Thank you both very much for your support. Got this working.
Removed the extra tab space using "vim type :set list" and set just one common registry file for both the prospectors.
Sorry about the late response. Was caught up with some other stuff.
If I do not specify the registry file, will it have its own default registry file to keep track of the file pointers?
This is my config-file
Now catalina logs are completely being ignored and only the module logs are being parsed?
Is it cause of the registry file?
How do I get this to work?
The module logs, line, starts with the date in the format "yyyy-mm-dd"
and catalina logs, line, starts with the date in format "dd-mon-yy".
I am able to parse either the catalina logs or the module logs but not both together.
Edit : Tried the same config without the registry parameter entry in it. Still doesn't parse the catalina logs. First prospector is completely being ignored.
logging should be at the top level and can only be defined once
Registry file must be indented to be under filebeat
Could it be that your multiline pattern is wrong? Have you tried without multiline?
As soon as the registry file is written, next time you start it will read from the old position. So make sure to stop filebeat and remove the registry file in case you want to start from scratch again.
If you run with the option -e -d "*" you get the debug output and see in much more detail on what is happening.
I'm afraid the catalina logs are not ignored, but combined into one very very big multiline event. In YAML always put your regexes into single quotes. ' in order to disable escaping done by YAML parser (YAML has multiple types of strings ).
Here is a small script testing multiline regexes: https://play.golang.org/p/ABEJaX_lsK
Lines starting with true will be merged into line before/after (depending on config). Put in some of your own logs + regexes and see if they can be matched correctly.
Your config still seems incorrect in some places. See notes by @ruflin
After doing a lot of trial and error, I realised I had been doing a silly mistake in my config file since the beginning. I ran filebeat with -e -d "*" option and saw that both the prospectors were being successfully loaded. crawler.go:52: INFO Loading Prospectors completed. Number of prospectors: 2
But the catalina logs were not being parsed. Then I realised that there was a typo in complete file path.
Earlier file path was /apps/azuga-web/log/catalina.out which was incorrect. When I tried to cat it manually, I got the error "No such file exists.
The correct path is /apps/azuga-web/logs/catalina.out.
Everything seems to be working fine now.
Although, logging is the only thing not working right now.
I started file beat using filebeat.sh -e -d "*" >> /var/log/filebeat/filebeat.log & at the moment.
Need to figure out the logging part.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
. It works fine when I commented the whole of second prospectors.
. It works fine when I commented the whole of second prospectors.
).
