I am very excited about the new dashboards feature for filebeat. I am trying to use custom indicies to differentiate between vpc flow logs, elb access logs, and cloudtrail logs. The issue is that the dashboards do not load properly. I am referencing the dashboards in my filebeat config like so
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana.host: "localhost:5601"
setup.kibana.protocol: "https"
setup.kibana.username: "username"
setup.kibana.password: "password"
setup.dashboards.enabled: true
setup.dashboards.directory: ${path.config}/kibana
setup.ilm.enabled: false
logging.level: debug
output.elasticsearch:
hosts: ["localhost:9243"]
protocol: "https"
username: "username"
password: "password"
bulk_max_size: 3200
worker: 16
indices:
- index: "cloudtrail-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
event.dataset: "aws.cloudtrail"
- index: "elb-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
event.dataset: "aws.elb"
- index: "vpc-%{[agent.version]}-%{+yyyy.MM.dd}"
when.contains:
event.dataset: "aws.vpc"
processors:
- add_fields:
target: SIEM
fields:
environment: ${ENVIRONMENT}
In the setup.dashboards.directory: ${path.config}/kibana directory I have custom json files for building the dashboards. Here is an example of a vpc flow log visualization.
{
"attributes": {
"description": "",
"kibanaSavedObjectMeta": {
"searchSourceJSON": "{\"filter\": [], \"query\": {\"query\": \"\", \"language\": \"kuery\"}}"
},
"title": "VPC Flow Top IP Addresses [Filebeat AWS]",
"uiStateJSON": "{}",
"version": 1,
"visState": "{\"aggs\": [], \"type\": \"metrics\", \"title\": \"VPC Flow Top IP Addresses [Filebeat AWS]\", \"params\": {\"show_legend\": 1, \"gauge_style\": \"half\", \"gauge_width\": 10, \"default_index_pattern\": \"vpc-*\", \"id\": \"61ca57f0-469d-11e7-af02-69e470af7417\", \"series\": [{\"type\": \"timeseries\", \"split_mode\": \"terms\", \"label\": \"IP address\", \"stacked\": \"none\", \"series_index_pattern\": \"vpc-*\", \"override_index_pattern\": 1, \"chart_type\": \"line\", \"axis_position\": \"right\", \"terms_field\": \"source.ip\", \"filter\": {\"query\": \"fileset.name : \\\"vpcflow\\\" \", \"language\": \"kuery\"}, \"fill\": 0.5, \"id\": \"61ca57f1-469d-11e7-af02-69e470af7417\", \"series_drop_last_bucket\": 0, \"color\": \"rgba(115,216,255,1)\", \"separate_axis\": 0, \"point_size\": 1, \"metrics\": [{\"type\": \"count\", \"id\": \"61ca57f2-469d-11e7-af02-69e470af7417\"}, {\"field\": \"61ca57f2-469d-11e7-af02-69e470af7417\", \"id\": \"40c52370-3e87-11ea-9067-cf383a4ea3b3\", \"type\": \"cumulative_sum\"}], \"formatter\": \"number\", \"line_width\": 1, \"color_rules\": [{\"id\": \"42e14220-3e86-11ea-9067-cf383a4ea3b3\"}], \"terms_order_by\": \"61ca57f2-469d-11e7-af02-69e470af7417\"}], \"default_timefield\": \"@timestamp\", \"axis_formatter\": \"number\", \"gauge_color_rules\": [{\"id\": \"2b29c940-3e86-11ea-9067-cf383a4ea3b3\"}], \"interval\": \"\", \"isModelInvalid\": false, \"bar_color_rules\": [{\"id\": \"cc6d5070-3e85-11ea-9067-cf383a4ea3b3\"}], \"legend_position\": \"bottom\", \"axis_scale\": \"normal\", \"axis_min\": \"0\", \"time_field\": \"\", \"gauge_inner_width\": 10, \"type\": \"top_n\", \"background_color_rules\": [{\"id\": \"29527130-3e86-11ea-9067-cf383a4ea3b3\"}], \"axis_position\": \"left\", \"pivot_id\": \"user_agent.original\", \"show_grid\": 1, \"index_pattern\": \"vpc-*\", \"pivot_type\": \"string\"}}"
},
"id": "75853f20-4484-11ea-ad63-791a5dc86f10",
"migrationVersion": {
"visualization": "7.3.1"
},
"references": [],
"type": "visualization",
"updated_at": "2020-02-03T16:42:58.806Z",
"version": "WzYzNzcsMV0="
}
The issue is that when the dashboards are generated I run into the following issue
I would love to know how to point the dashboards to the custom indicies. Any help would be greatly appreciated.