Need configuration to send JSON string to ELK server through Filebeat

[Honda_fred_elk (Fredrick Yessaian)]

Hello,

I am trying to export log message (JSON) from client side log to ELK Server through FileBeat.

1. My custom log (log4j) will produce JSON output in a file through log4j.RollingFileAppender. I know, I have to have only one JSON String (Object) per line.
2. I want Filebeat to read JSON line by line from the log file and send to Remote ELK Server.
3. File beat should not miss any JSON line, even though log4j rotates the log file
4. I have installed Filebeat in my client machine.

I just need to know, a concreate settings that I have to set in filebeat.yml file.

Thanks..

[andrewkroh]

See https://www.elastic.co/guide/en/beats/filebeat/5.1/configuration-filebeat-options.html#config-json for documentation of all the options. Here's a brief example.

filebeat.prospectors:
- paths:
    - /var/log/path/to/input.json
  document_type: myapp
  json.keys_under_root: true
  json.add_error_key: true

output.elasticsearch:
  hosts: ['http://localhost:9200']

[Honda_fred_elk (Fredrick Yessaian)]

Thank you so much Andrew.. Instead of elastic search I send the JSON string to logstash

I have following settings in filebeat.yml

filebeat.prospectors:
- paths:
    - <Windows path>jsonoutput.log
  document_type: log
  json.keys_under_root: true
  json.add_error_key: true

output.logstash:
  hosts: ['http://<logstashserver-ip>:6000']

below is the configuration settings in logstash and logstash starts without any issues.

input {
  beats {
     port => 6000
     tags => "beats"
     codec => "json_lines"
  }	
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout{}  
  file{
      path => "/appl/log/TestLogsOutput/LogStash_output.log"
  }
}

When I start Filebeat, it starts without any error. But once .log file generated JSON string as log.
It tries to establish connection to logstash.. and throughs following error..

WARN DNS lookup failure "http://<logstashserver-ip>:6000": lookup http://<logstashserver-ip>:6000: getaddrinfow: No such host is known. ERR Connecting error publishing events (retrying): lookup http://<logstashserver-ip>:6000: getaddrinfow: No such host is known.

I am able telnet from my window laptop (where I run the filebeat) to logstash server..

Could you help me, what could be the reason?

Thanks

[andrewkroh]

Checkout the instructions for configuring Filebeat and Logstash. Let us know if you have problems after following the config examples and instructions there.

Configuring Filebeat to Use Logstash
Setting Up Logstash for Beats

[Honda_fred_elk (Fredrick Yessaian)]

Thank you Andrew.

Everything fine except the URL.. instead of giving like below

output.logstash:
  hosts: ['http://logstashserver-ip:6000']

I changed to

output.logstash:
  hosts: ['logstashserver-ip:6000']

Now it works absolutely fine..

Thanks

[andrewkroh]

@Honda_fred_elk

codec => "json_lines"

You shouldn't' need that line either since you are doing JSON decoding in Filebeat.

[Honda_fred_elk (Fredrick Yessaian)]

I am facing another problem now.
I have given specific file name to read JSON string. so below the config.

filebeat.prospectors:
- paths:
    -  C:\appl\log\GALCLogs\Server-jsonoutput.log

This file written by Log4j framework.. and it rotates as soon as it attains the specified file size.
because of this, file beat looses bottom 20 lines of log messages (JSON String) on every rotation.

Has anyone faced this problem? How this could be fixed?

Thanks

[andrewkroh]

Well if you use a glob pattern that also matches the rotated files, then filebeat can follow the rotated file and continue reading it to the end. Try this if your rotation moves x.log to x.log.1.

filebeat.prospectors:
- paths:
    -  'C:\appl\log\GALCLogs\Server-jsonoutput.log*'