Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

[Triv1an (Teun Roefs)]

Good afternoon,

We are currently using Filebeat to send pre-formatted JSON messages, saved in a logfile through to our processing tool.

JSON looks as follows:

{
	"generated_at": "2018-02-06T13:56:04+00:00",
	"profile": {
		"title": "CIS Ubuntu 16.04 Level 1 Benchmark",
		"version": "0.0.1"
	},
	"control": {
		"title": "Check That Users Are Assigned Valid Home Directories",
		"description": "Users can be defined in /etc/passwd without a home directory or with a home directory that does not actually exist.",
		"impact": 1.0,
		"cis_id": "xccdf_org.cisecurity.benchmarks_rule_13.12_Check_That_Users_Are_Assigned_Valid_Home_Directories"
	},
	"results": {
		"status": "failed",
		"code_desc": "Bash command  cat /etc/passwd | awk -F: '{ print $1 \" \" $3 \" \" $6 }' | while read user uid dir; do if [ $uid -ge 500 -a ! -d \"$dir\" -a $user != \"nfsnobody\" -a $user != \"nobody\" ]; then echo \"The home directory ($dir) of user $user does not exist.\"; fi; done stdout should not match /.+/",
		"run_time": 0.0200421,
		"start_time": "2018-02-06 13:56:03 +0000",
		"message": "expected \"The home directory (/opt/splunkforwarder) of user splunk does not exist.\\nThe home directory (/var/o...exist.\\nThe home directory (/var/opt/microsoft/omsagent/run) of user nxautomation does not exist.\\n\" not to match /.+/\nDiff:\n@@ -1,2 +1,5 @@\n-/.+/\n+The home directory (/opt/splunkforwarder) of user splunk does not exist.\n+The home directory (/var/opt/microsoft/omsagent/run) of user omsagent does not exist.\n+The home directory (/home/omi) of user omi does not exist.\n+The home directory (/var/opt/microsoft/omsagent/run) of user nxautomation does not exist.\n"
	}
}

Every now and then we get the following error: Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {} or other errors: Error decoding JSON: invalid character '"' in literal null (expecting 'u') and Error decoding JSON: EOF

Step by step process is as follows:

1. Ruby script removes old version of the log file.
2. Ruby script executes a command that outputs to JSON. That output is parsed and send to a log file that will ultimately contain 196 lines.
3. Filebeat is used to harvest that specific file and send the data through to our log processor

This process goes right once every four times. The problem here is that the Ruby script only runs once a week and is used to fill some monitoring dashboards.

I've tried the following settings already:

• Flushing Ruby after every line thats outputted to the log file
• Use multilines based on a regex (just to test, there are no multilines)
• Change backoff time

I've yet to find a root-cause to this problem but my gut feeling tells me the Harvester is reading the file too quickly and sometimes causes the line to be parsed before Ruby is actually done writing it completely.

I've seen many posts on this website from other people having issues with the exact same command and the post that comes closest to my problem is: Filebeat is not able to parse json from files where \n separated json lines (events) are written. It sometimes misses the records and sometimes gives error when traffic is high

Hope anyone can help me with this issue.

Kind regards,

Teun Roefs

[exekias (Carlos Pérez Aradros)]

Hi @Triv1an,

Could you please share the settings you are using?

Best regards

[Triv1an (Teun Roefs)]

Hi @exekias,

The settings are as follows:

###################### Filebeat Configuration  ################################

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

- type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
        - /var/log/inspec/*.log

  json.keys_under_root: true

#================================ General =====================================

fields:
   uuid: 5ca3590a-1316-b741-9efa-94f79b9d1092

#================================ Processors ===================================
processors:
- drop_fields:
    fields: ["beat", "offset", "type"] # Cannot remove more fields as of now.

#================================ Outputs =====================================

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ['10.0.0.8:5044']

#================================ Logging =====================================

We are currently not using any other settings, just the defaults.

Kind regards

[dcroonen (Davy Croonen)]

Hi @Triv1an

I don't now if this helps but as I remember myself correctly, we also had some issues with filebeat and in our case rotating log files. We managed to solve this by using the option copytruncate in logrotate. In essence this comes down to fact, that instead of removing the old log file and creating a new one (which caused problems with filebeat), copy the log file and empty the current one. In your case you just want to empty your current logfile instead of removing it and creating a new one.

Regards
Davy

[Triv1an (Teun Roefs)]

Hi @dcroonen

What we noticed was that when we did not empty the file but just let it be overwritten, the error occurred more often. We let the Ruby script itself remove the file before running and then create a new file. I will test your solution by removing the removal of the file in the script and let logrotate copytruncate the file.

I will let you know the results.

Regards
Teun

[Triv1an (Teun Roefs)]

Hi @dcroonen @exekias,

Unfortunately this did not resolve the issue. Any other suggestions regarding this issue?

Kind regards,

Teun Roefs

[Triv1an (Teun Roefs)]

Hi @exekias,

Were my settings of any use? I've tested the copytruncate thoroughly but it caused the Harvester to give the following error everytime: ERR Error decoding JSON: EOF

Would it be a possibility to rewrite the script to Python instead of Ruby?

Are there any other options regarding Filebeat I can try?

Kind regards