Parse / ship JSON file with filebeat

Elastic Stack Beats
1

Is it possible to directly parse a json file from filebeats into elasticsearch?

I have a small json file and I would like to have filebeats read in and ship directly to elasticsearch in json. json file -> filebeats => elasticsearch.

2

No, filebeat will just forward lines from files. For parsing it must be used with logstash. You can use json_lines codec in logstash to parse.

In case you have one complete json-object per line you can try in logstash

input {
    beats {
        ...
        codec => "json_lines"
     }
}

See codec documentation

filebeat will follow lines being written. If you want to send your file only once you can try the tcp input plugin in logstash with netcat.

edit: added the missing 's' on json_lines

3

Hi Steffens,

I just tried the json_line codec you talk about here. I copied it from the post into my beats plugin on my input config.

    beats {
     port => 5000
     tags => "beats"
     codec => "json_line"
     #ssl  => true
     #ssl_certificate => "/opt/filebeats/logs.example.com.crt"
     #ssl_key => "/opt/filebeats/logs.example.com.key"
     }

[root@logs:/etc/logstash/conf.d] #/opt/logstash/bin/logstash --configtest --config 10-logstash-input.conf
The error reported is:
  Couldn't find any codec plugin named 'json_line'. Are you sure this is correct? Trying to load the json_line codec plugin resulted in this error: no such file to load -- logstash/codecs/json_line

And I'm on the latest version:

[root@logs:/etc/logstash/conf.d] #/opt/logstash/bin/logstash --version
logstash 2.2.0

This is my whole input config:

input {
   lumberjack {
       # The port to listen on
       port => 2541

       # The paths to your ssl cert and key
       ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
       ssl_key => "/etc/pki/tls/private/logstash.key"

         # Set this to whatever you want.
         type => "logstash"
         codec => "json"
       }

     beats {
     port => 5000
     tags => "beats"
     codec => "json_line"
     #ssl  => true
     #ssl_certificate => "/opt/filebeats/logs.example.com.crt"
     #ssl_key => "/opt/filebeats/logs.example.com.key"
     }


     syslog {
        type => "syslog"
        port => "5514"

    }

    redis {
     host => "216.xxx.xxx.98"
     type => "redis-input"
     data_type => "list"
     key => "logstash"
    }
}

What's going wrong here?

Thanks

4

Oh, it should've been codec => "json_lines" instead of codec => "json_line"!!! Wah wah wah wahhhhh. lol :joy:

5

is "json_lines" working for you? You can also try the "json" codec.

6

Hi Steffens,

Yeah they both seem to work really great!

I've run into another issue where filebeat stopped working however.

Filebeat Stopped working

It was nice having it work while it lasted !