Filebeat fails? file.go:84: Fail to convert the event to JSON - reflect.Value.IsNil on zero Value

Elastic Stack Beats
1

I am trying to ingest some large log files into ELK. Recently setup Filebeat to monitor a watch folder that I drop logs into. This has been working great on my sample data. As I copy over a directory with a days worth of real data the filebeat will process a few hundred thousand lines and then it stops.

I can see that process is still running. I notice the following error in /var/log/messages:

/usr/bin/filebeat[27070]: file.go:84: Fail to convert the event to JSON: reflect: call of reflect.Value.IsNil on zero Value

A 'service filebeat restart' will repair and it continue process logs (but I'm uncertain on if there are gaps). This has failed about half dozen times today and I know this isn't stable enough for what at we need in long term.

My filebeat.yml is relatively simple. Here is the prospector being used:

filebeat:
  prospectors:
    - 
      paths:
        - /data/filebeat/watch/*/sn.log.*
      fields:
         logsrc-addr: 1.2.3.4
         logsrc-name: name
         logsrc-site: site
         logsrc-cust: tenant
         logsrc-tz: MST
      document_type: logstash_type
      fields_under_root: true
      input_type: log
      exclude_files: [".gz$"]
...

output:
  logstash:
    hosts: ["mylogstashhost:5044"]

  file:
    path: "/data/filebeat/diags"
    number_of_files: 20 
    rotate_every_kb: 10000
2

Interesting. I wonder what that event looks like. Could you compile Filebeat yourself, modifying the "Fail to convert the event to JSON" line in libbeat/outputs/fileout/file.go to this:

logp.Err("Fail to convert the event to JSON: %s: %#v", err, event)
3

first hunch: looks like some bug in the standard library. Definitely worth debugging. This happens regularly?

@dromrell Which filebeat version are you using? Which operating system? the tip by @magnusbaeck is very helpfull. You need any help building filebeat for your OS?

4

I used yum to install the standard packages as explained here:
https://www.elastic.co/guide/en/beats/libbeat/current/setup-repositories.html
filebeat version 1.1.1 (amd64)
CentOS Linux release 7.2.1511 (Core)

I will work on creating the custom build. Any quick tips/links to setup golang build env (starting google search now) or specific versions of filebeat I should modify?

Thanks

5

golang.org has quite helpful docs. See go getting started guide.

I'd say to test first with main branch if issue (most likely) still persists. You can get nightlies (v 5.0.0 build I think) here (no need to install go dev environment): https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

If problem still occurs (most likely) modifications can be done in main branch (grep for json.Unmarshal for possible occurences). I created a github issue for adding error output, but didn't manage to add these yet.