Parse / ship JSON file with filebeat

Is it possible to directly parse a json file from filebeats into elasticsearch?

I have a small json file and I would like to have filebeats read in and ship directly to elasticsearch in json. json file -> filebeats => elasticsearch.

No, filebeat will just forward lines from files. For parsing it must be used with logstash. You can use json_lines codec in logstash to parse.

In case you have one complete json-object per line you can try in logstash

input {
    beats {
        ...
        codec => "json_lines"
     }
}

See codec documentation

filebeat will follow lines being written. If you want to send your file only once you can try the tcp input plugin in logstash with netcat.

edit: added the missing 's' on json_lines

Hi Steffens,

I just tried the json_line codec you talk about here. I copied it from the post into my beats plugin on my input config.

    beats {
     port => 5000
     tags => "beats"
     codec => "json_line"
     #ssl  => true
     #ssl_certificate => "/opt/filebeats/logs.example.com.crt"
     #ssl_key => "/opt/filebeats/logs.example.com.key"
     }
[root@logs:/etc/logstash/conf.d] #/opt/logstash/bin/logstash --configtest --config 10-logstash-input.conf
The error reported is:
  Couldn't find any codec plugin named 'json_line'. Are you sure this is correct? Trying to load the json_line codec plugin resulted in this error: no such file to load -- logstash/codecs/json_line

And I'm on the latest version:

[root@logs:/etc/logstash/conf.d] #/opt/logstash/bin/logstash --version
logstash 2.2.0

This is my whole input config:

input {
   lumberjack {
       # The port to listen on
       port => 2541

       # The paths to your ssl cert and key
       ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
       ssl_key => "/etc/pki/tls/private/logstash.key"

         # Set this to whatever you want.
         type => "logstash"
         codec => "json"
       }

     beats {
     port => 5000
     tags => "beats"
     codec => "json_line"
     #ssl  => true
     #ssl_certificate => "/opt/filebeats/logs.example.com.crt"
     #ssl_key => "/opt/filebeats/logs.example.com.key"
     }


     syslog {
        type => "syslog"
        port => "5514"

    }

    redis {
     host => "216.xxx.xxx.98"
     type => "redis-input"
     data_type => "list"
     key => "logstash"
    }
}

What's going wrong here?

Thanks

Oh, it should've been codec => "json_lines" instead of codec => "json_line"!!! Wah wah wah wahhhhh. lol :joy:

is "json_lines" working for you? You can also try the "json" codec.

Hi Steffens,

Yeah they both seem to work really great!

I've run into another issue where filebeat stopped working however.

Filebeat Stopped working

It was nice having it work while it lasted !