I'm trying to use filebeat instead of logstash to aggregate osquery logs but the logs seem to be loaded as plain text instead.
Here are the instructions I am referring to:
http://lowlyadmin.com/mac/2016/06/02/deploying-filebeat-on-macos/
And my pipeline is as follows:
osquery logs --> filebeat --> logstash --> es
Each line in the osquery logs is a JSON object. However, the logs seem to be parsed as plaintext instead and put inside the "messages" field. Is there a way to parse the logs as JSON?
Here is my filebeat config:
filebeat:
prospectors:
-
paths:
- /var/log/osquery/osqueryd.results.log
document_type: json
fields:
type: osquery_json
codec: json
input_type: log
output:
logstash:
hosts: ["domain.com:8600"]
tls:
certificate_authorities: ["/etc/pki/tls/certs/filebeat.crt"]
shipper:
name: "shauvik@domain.com"
tags: ["macbook"]
logging:
to_files: true
files:
path: /var/log
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
level: info
And here is my logstash config:
input {
beats {
port => 8600
ssl => true
type => "JSON"
ssl_certificate => "/etc/logstash/filebeat.crt"
ssl_key => "/etc/logstash/filebeat.key"
congestion_threshold => 1000
}
}
output {
stdout {}
elasticsearch {
hosts => ["localhost"]
}
}
This is the record from elasticsearch, which has the entire record in "message" and a tag of "beats_input_codec_plain_applied"
{
"_index": "logstash-2016.07.05",
"_type": "json",
"_id": "AVW8l5bfGrkTbS2VxQgg",
"_score": null,
"_source": {
"message": "{\"name\":\"pack_osquery-monitoring_osquery_info\",\"hostIdentifier\":\"Shauviks-MacBook-Pro.local\",\"calendarTime\":\"Tue Jul 5 19:44:50 2016 UTC\",\"unixTime\":\"1467747890\",\"columns\":{\"build_distro\":\"10.11\",\"build_platform\":\"darwin\",\"config_hash\":\"7ffd8492aa551068b24dc444109a8d83\",\"config_valid\":\"1\",\"counter\":\"44\",\"extensions\":\"active\",\"pid\":\"41936\",\"resident_size\":\"9768960\",\"start_time\":\"1467747804\",\"system_time\":\"52\",\"user_time\":\"138\",\"version\":\"1.7.4\"},\"action\":\"added\"}",
"@version": "1",
"@timestamp": "2016-07-05T19:44:56.746Z",
"offset": 2503721,
"input_type": "log",
"beat": {
"hostname": "Shauviks-MacBook-Pro.local",
"name": "shauvik@domain.com"
},
"tags": [
"macbook",
"beats_input_codec_plain_applied"
],
"source": "/var/log/osquery/osqueryd.results.log",
"count": 1,
"fields": {
"codec": "json",
"type": "osquery_json"
},
"type": "json",
"host": "Shauviks-MacBook-Pro.local"
},
"fields": {
"@timestamp": [
1467747896746
]
},
"sort": [
1467747896746
]
}