Good afternoon,
We are currently using Filebeat to send pre-formatted JSON messages, saved in a logfile through to our processing tool.
JSON looks as follows:
{
"generated_at": "2018-02-06T13:56:04+00:00",
"profile": {
"title": "CIS Ubuntu 16.04 Level 1 Benchmark",
"version": "0.0.1"
},
"control": {
"title": "Check That Users Are Assigned Valid Home Directories",
"description": "Users can be defined in /etc/passwd without a home directory or with a home directory that does not actually exist.",
"impact": 1.0,
"cis_id": "xccdf_org.cisecurity.benchmarks_rule_13.12_Check_That_Users_Are_Assigned_Valid_Home_Directories"
},
"results": {
"status": "failed",
"code_desc": "Bash command cat /etc/passwd | awk -F: '{ print $1 \" \" $3 \" \" $6 }' | while read user uid dir; do if [ $uid -ge 500 -a ! -d \"$dir\" -a $user != \"nfsnobody\" -a $user != \"nobody\" ]; then echo \"The home directory ($dir) of user $user does not exist.\"; fi; done stdout should not match /.+/",
"run_time": 0.0200421,
"start_time": "2018-02-06 13:56:03 +0000",
"message": "expected \"The home directory (/opt/splunkforwarder) of user splunk does not exist.\\nThe home directory (/var/o...exist.\\nThe home directory (/var/opt/microsoft/omsagent/run) of user nxautomation does not exist.\\n\" not to match /.+/\nDiff:\n@@ -1,2 +1,5 @@\n-/.+/\n+The home directory (/opt/splunkforwarder) of user splunk does not exist.\n+The home directory (/var/opt/microsoft/omsagent/run) of user omsagent does not exist.\n+The home directory (/home/omi) of user omi does not exist.\n+The home directory (/var/opt/microsoft/omsagent/run) of user nxautomation does not exist.\n"
}
}
Every now and then we get the following error: Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}
or other errors: Error decoding JSON: invalid character '"' in literal null (expecting 'u')
and Error decoding JSON: EOF
Step by step process is as follows:
- Ruby script removes old version of the log file.
- Ruby script executes a command that outputs to JSON. That output is parsed and send to a log file that will ultimately contain 196 lines.
- Filebeat is used to harvest that specific file and send the data through to our log processor
This process goes right once every four times. The problem here is that the Ruby script only runs once a week and is used to fill some monitoring dashboards.
I've tried the following settings already:
- Flushing Ruby after every line thats outputted to the log file
- Use multilines based on a regex (just to test, there are no multilines)
- Change backoff time
I've yet to find a root-cause to this problem but my gut feeling tells me the Harvester is reading the file too quickly and sometimes causes the line to be parsed before Ruby is actually done writing it completely.
I've seen many posts on this website from other people having issues with the exact same command and the post that comes closest to my problem is: Filebeat is not able to parse json from files where \n separated json lines (events) are written. It sometimes misses the records and sometimes gives error when traffic is high
Hope anyone can help me with this issue.
Kind regards,
Teun Roefs