Getting filebeat to process JSON logs from osquery

Elastic Stack Beats
1

I'm trying to use filebeat instead of logstash to aggregate osquery logs but the logs seem to be loaded as plain text instead.

Here are the instructions I am referring to:
http://lowlyadmin.com/mac/2016/06/02/deploying-filebeat-on-macos/

And my pipeline is as follows:
osquery logs --> filebeat --> logstash --> es

Each line in the osquery logs is a JSON object. However, the logs seem to be parsed as plaintext instead and put inside the "messages" field. Is there a way to parse the logs as JSON?

Here is my filebeat config:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/osquery/osqueryd.results.log
      document_type: json
      fields:
        type: osquery_json
        codec: json
      input_type: log
output:
  logstash:
    hosts: ["domain.com:8600"]
    tls:
      certificate_authorities: ["/etc/pki/tls/certs/filebeat.crt"]
shipper:
  name: "shauvik@domain.com"
  tags: ["macbook"]
logging:
  to_files: true
  files:
    path: /var/log
    name: filebeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  level: info

And here is my logstash config:

input {
  beats {
    port => 8600
    ssl => true
    type => "JSON"
    ssl_certificate => "/etc/logstash/filebeat.crt"
    ssl_key => "/etc/logstash/filebeat.key"
    congestion_threshold => 1000
  }
}
output {
  stdout {}
  elasticsearch {
    hosts => ["localhost"]
  }
}

This is the record from elasticsearch, which has the entire record in "message" and a tag of "beats_input_codec_plain_applied"

{
  "_index": "logstash-2016.07.05",
  "_type": "json",
  "_id": "AVW8l5bfGrkTbS2VxQgg",
  "_score": null,
  "_source": {
    "message": "{\"name\":\"pack_osquery-monitoring_osquery_info\",\"hostIdentifier\":\"Shauviks-MacBook-Pro.local\",\"calendarTime\":\"Tue Jul  5 19:44:50 2016 UTC\",\"unixTime\":\"1467747890\",\"columns\":{\"build_distro\":\"10.11\",\"build_platform\":\"darwin\",\"config_hash\":\"7ffd8492aa551068b24dc444109a8d83\",\"config_valid\":\"1\",\"counter\":\"44\",\"extensions\":\"active\",\"pid\":\"41936\",\"resident_size\":\"9768960\",\"start_time\":\"1467747804\",\"system_time\":\"52\",\"user_time\":\"138\",\"version\":\"1.7.4\"},\"action\":\"added\"}",
    "@version": "1",
    "@timestamp": "2016-07-05T19:44:56.746Z",
    "offset": 2503721,
    "input_type": "log",
    "beat": {
      "hostname": "Shauviks-MacBook-Pro.local",
      "name": "shauvik@domain.com"
    },
    "tags": [
      "macbook",
      "beats_input_codec_plain_applied"
    ],
    "source": "/var/log/osquery/osqueryd.results.log",
    "count": 1,
    "fields": {
      "codec": "json",
      "type": "osquery_json"
    },
    "type": "json",
    "host": "Shauviks-MacBook-Pro.local"
  },
  "fields": {
    "@timestamp": [
      1467747896746
    ]
  },
  "sort": [
    1467747896746
  ]
}
2

have you tried json support in filebeat 5.0alpha4?

3

Hi Steffens, I tried filebeat 5.0alpha4 but couldn't get it to work.

Here is my new config:

filebeat.prospectors:

# Input from osquery logs
- input_type: log
  paths:
    - /var/log/osquery/osqueryd.results.log
  document_type: json
  fields:
    type: osquery_json
    codec: json

# Parse each line as JSON
json.message_key: log

# Output to logstash
output.logstash:
  hosts: ["domain.com:8600"]
  tls.certificate_authorities: ["/etc/pki/tls/certs/filebeat.crt"]

# Shipper details  
name: "shauvik@domain.com"
tags: ["macbook"]

# Filebeat logging
logging.level: debug

The new elastic search record still contains the entire text "message" and a tag of "beats_input_codec_plain_applied":

{
  "_index": "logstash-2016.07.18",
  "_type": "json",
  "_id": "AVX8dSsgGrkTbS2VxfGD",
  "_score": null,
  "_source": {
    "message": "{\"name\":\"pack_osquery-monitoring_schedule\",\"hostIdentifier\":\"Shauviks-MacBook-Pro.local\",\"calendarTime\":\"Mon Jul 18 05:22:53 2016 UTC\",\"unixTime\":\"1468819373\",\"columns\":{\"average_memory\":\"0\",\"avg_system_time\":\"\",\"avg_user_time\":\"\",\"executions\":\"0\",\"interval\":\"86400\",\"last_executed\":\"0\",\"name\":\"pack_it-compliance_alf_services\",\"output_size\":\"0\",\"wall_time\":\"0\"},\"action\":\"added\"}",
    "@version": "1",
    "@timestamp": "2016-07-18T05:23:00.865Z",
    "offset": 4351706,
    "type": "json",
    "tags": [
      "macbook",
      "beats_input_codec_plain_applied"
    ],
    "source": "/var/log/osquery/osqueryd.results.log",
    "input_type": "log",
    "beat": {
      "name": "shauvik@domain.com",
      "hostname": "Shauviks-MacBook-Pro.local"
    },
    "fields": {
      "type": "osquery_json",
      "codec": "json"
    },
    "host": "Shauviks-MacBook-Pro.local"
  },
  "fields": {
    "@timestamp": [
      1468819380865
    ]
  },
  "sort": [
    1468819380865
  ]
}

I have tried several values for the json.message_key option including empty string, message, log, json but none of them seem to work.

Any insights or help are appreciated. Thanks!

4

@shauvik The indentation of your config file looks off. json.message_key should be on the same level like paths or document_type. Or is this just a copy / paste issue?

5

Hi Ruflin, That was it. Thank you for pointing that out.

6

This topic was automatically closed after 21 days. New replies are no longer allowed.