Filebeat fails? file.go:84: Fail to convert the event to JSON - reflect.Value.IsNil on zero Value


I am trying to ingest some large log files into ELK. Recently setup Filebeat to monitor a watch folder that I drop logs into. This has been working great on my sample data. As I copy over a directory with a days worth of real data the filebeat will process a few hundred thousand lines and then it stops.

I can see that process is still running. I notice the following error in /var/log/messages:

/usr/bin/filebeat[27070]: file.go:84: Fail to convert the event to JSON: reflect: call of reflect.Value.IsNil on zero Value

A 'service filebeat restart' will repair and it continue process logs (but I'm uncertain on if there are gaps). This has failed about half dozen times today and I know this isn't stable enough for what at we need in long term.

My filebeat.yml is relatively simple. Here is the prospector being used:

        - /data/filebeat/watch/*/sn.log.*
         logsrc-name: name
         logsrc-site: site
         logsrc-cust: tenant
         logsrc-tz: MST
      document_type: logstash_type
      fields_under_root: true
      input_type: log
      exclude_files: [".gz$"]

    hosts: ["mylogstashhost:5044"]

    path: "/data/filebeat/diags"
    number_of_files: 20 
    rotate_every_kb: 10000

(Magnus B├Ąck) #2

Interesting. I wonder what that event looks like. Could you compile Filebeat yourself, modifying the "Fail to convert the event to JSON" line in libbeat/outputs/fileout/file.go to this:

logp.Err("Fail to convert the event to JSON: %s: %#v", err, event)

(Steffen Siering) #3

first hunch: looks like some bug in the standard library. Definitely worth debugging. This happens regularly?

@dromrell Which filebeat version are you using? Which operating system? the tip by @magnusbaeck is very helpfull. You need any help building filebeat for your OS?


I used yum to install the standard packages as explained here:
filebeat version 1.1.1 (amd64)
CentOS Linux release 7.2.1511 (Core)

I will work on creating the custom build. Any quick tips/links to setup golang build env (starting google search now) or specific versions of filebeat I should modify?


(Steffen Siering) #5 has quite helpful docs. See go getting started guide.

I'd say to test first with main branch if issue (most likely) still persists. You can get nightlies (v 5.0.0 build I think) here (no need to install go dev environment):

If problem still occurs (most likely) modifications can be done in main branch (grep for json.Unmarshal for possible occurences). I created a github issue for adding error output, but didn't manage to add these yet.

(system) #6