Sending Json Logs from Filebeat 5.0.0_alpha5-1 to Graylog 2.2.0

Elastic Stack Beats
1

I am trying to send my json logs to Graylog. I've configured my filebeat.yml config file that similar to following post. However, I am not able to send my logs to Graylog properly and I get "null" message in Graylog that means that Filebeat does not parse my json formatted log. How can I parse json log properly?

Note: I am sending plain text logs smoothly from Filebeat to Graylog.

My Filebeat.yml config file:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/testfilebeat.log
  document_type: json
  fields:
    type: testfilebeatjson
    codec: json
  json.message_key: log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: "request"
  document_type: test_request
  tail_files: true

output.logstash:
  hosts: ["10.0.1.10:12201"]
  tls.insecure: true

logging.to_files: true
logging.files:
  path: /var/log/filebeatlogs
  name: filebeatlog
  rotateeverybytes: 104857600 # = 100MB
  keepfiles: 10

My Json formatted Log file; /var/log/testfilebeat.log :

{"_message":{"took":"0","_httpParams":{},"httpUrl":"/v1/json/ping","httpClientIp":"127.0.0.1","httpCFIp":"","httpMethod":"GET","_httpHeaders":{"Accept":"*/*","X-ELB-IP":"10.0.0.0","Connection":"close","User-Agent":"ELB-HealthChecker/1.0","Host":"localhost:9000"},"logTime":"1478044910569","status":"200 OK"},"timestamp":"1477977078"}

Log is shown in Graylog output as follow:

filebeat.png1128×598 40 KB

Note: I installed Graylog Filebeat plugin from following link.

Graylog version : 2.2.0
Filebeat version: 5.0.0_alpha5-1

2

You defined json.message_key: log but in your example entry there is no entry log.

3

I tried json.message_key entry as below, but unfortunately still getting "null" message in Graylog. Is there anyway to parse json?

json.message_key: _message

In following config file, messages are shipping in plain text format, and Graylog doesn't understand json formatted entries(especially timestamp is important)

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/testfilebeat.log
  multiline.pattern: ^\{"
  multiline.negate: true
  multiline.match: after
  tags: "request"
  document_type: test_request
  tail_files: false

output.logstash:
  hosts: ["10.0.26.10:12205"]
  tls.insecure: true

logging.to_files: true
logging.files:
  path: /var/log/opsgenieFileBeatLogs
  name: opsgenieFileBeat
  rotateeverybytes: 104857600 # = 100MB
  keepfiles: 10
4

I'm not aware of the inner details of Graylog. Can you try to just use file output and check if you get the expected result?

5

This topic was automatically closed after 21 days. New replies are no longer allowed.