Sending Json Logs from Filebeat 5.0.0_alpha5-1 to Graylog 2.2.0

I am trying to send my json logs to Graylog. I've configured my filebeat.yml config file that similar to following post. However, I am not able to send my logs to Graylog properly and I get "null" message in Graylog that means that Filebeat does not parse my json formatted log. How can I parse json log properly?

Note: I am sending plain text logs smoothly from Filebeat to Graylog.

My Filebeat.yml config file:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/testfilebeat.log
  document_type: json
  fields:
    type: testfilebeatjson
    codec: json
  json.message_key: log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: "request"
  document_type: test_request
  tail_files: true

output.logstash:
  hosts: ["10.0.1.10:12201"]
  tls.insecure: true

logging.to_files: true
logging.files:
  path: /var/log/filebeatlogs
  name: filebeatlog
  rotateeverybytes: 104857600 # = 100MB
  keepfiles: 10

My Json formatted Log file; /var/log/testfilebeat.log :

{"_message":{"took":"0","_httpParams":{},"httpUrl":"/v1/json/ping","httpClientIp":"127.0.0.1","httpCFIp":"","httpMethod":"GET","_httpHeaders":{"Accept":"*/*","X-ELB-IP":"10.0.0.0","Connection":"close","User-Agent":"ELB-HealthChecker/1.0","Host":"localhost:9000"},"logTime":"1478044910569","status":"200 OK"},"timestamp":"1477977078"}

Log is shown in Graylog output as follow:

Note: I installed Graylog Filebeat plugin from following link.

Graylog version : 2.2.0
Filebeat version: 5.0.0_alpha5-1

You defined json.message_key: log but in your example entry there is no entry log.

I tried json.message_key entry as below, but unfortunately still getting "null" message in Graylog. Is there anyway to parse json?

json.message_key: _message

In following config file, messages are shipping in plain text format, and Graylog doesn't understand json formatted entries(especially timestamp is important)

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/testfilebeat.log
  multiline.pattern: ^\{"
  multiline.negate: true
  multiline.match: after
  tags: "request"
  document_type: test_request
  tail_files: false

output.logstash:
  hosts: ["10.0.26.10:12205"]
  tls.insecure: true

logging.to_files: true
logging.files:
  path: /var/log/opsgenieFileBeatLogs
  name: opsgenieFileBeat
  rotateeverybytes: 104857600 # = 100MB
  keepfiles: 10

I'm not aware of the inner details of Graylog. Can you try to just use file output and check if you get the expected result?

This topic was automatically closed after 21 days. New replies are no longer allowed.