Json Parsing to Elastic Search using filebeat

Elastic Stack Beats
1

Hi Team,

I am trying to push some audit logs which are in json format to ES from a docker container. Could you please let us know how can we perform it using filebeat (without logstash)?

Filebeat Version: 6.0.0
Elastic Search: 5.6

Sample Data
{"time":"2017-12-05T19:29:57Z","type":"request","auth":{"client_token":"hmac-sha256:88838df166cba1c6c5918d0156fcea93e2fdaed57529f83be8a90747995a382e","accessor":"hmac-sha256:55d6b114f6754c45ee33768b3d4b36a1fbbaee1c60fa737fc0d4b18efda68412","display_name":"ldap-user1","policies":["default","r_ldap_wer","ldapadmin","w_use_bin","welcome"],"metadata":{"policies":"default,r_shared_21334,w_use_bin","username":"welcome"}},"request":{"id":"f0057990-9811-6b90-c7cd-73e3fc737acf","operation":"list","client_token":"hmac-sha256:88838df166cba345345435435431c6c5918d0156fcea93e2fdaed57529f83be8a90747995a382e","client_token_accessor":"hmac-sha256:55d6b114f64534543534754c45ee33768b3d4b36a1fbbaee1c60fa737fc0d4b18efda68412","path":"metadata/users/","data":null,"remote_address":"172.17.0.1","wrap_ttl":0,"headers":{}},"error":""}

2

I got the solution and the solution is

filebeat configuration 
    filebeat.prospectors:
- input_type: log
  paths:
    - /root/vaultlogs/tvault-vault_audit.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log
output.elasticsearch:
  hosts: ["10.65.228.156:9200"]
  template:
    name: "app"
    path: "app.template.json"
  indices:
    - index: "app-audit-%{+MM.yyyy}"

There should be a template file in in the /etc/filebeat/ localtion

{
  "index_patterns": [
    "app-audit-*"
  ],
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "time": {
          "format": "YYYY-MM-dd'T'HH:mm:ssZ",
          "type": "date"
        }
      }
    },
    "aliases": {
      "team-app-auditlog": {

      }
    }
  },
  "order": 1,
  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": 10000
        }
      },
      "refresh_interval": "5s"
    }
  }
}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.