And Now for the Complete Solution with Client Names and Data Streams
PUT _index_template/applog
{
"name": "applog",
"index_template": {
"index_patterns": [
"applog-*"
],
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "applog"
}
}
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "text"
}
}
}
},
"composed_of": [],
"data_stream": {
"hidden": false,
"allow_custom_routing": true <!--- NOTE this for later
}
}
}
And the Logstash this is my stub the filter and output are important..
input {
stdin {
}
}
filter {
# Assume you have a the fields you want...
mutate {
add_field => {
"client_name" => "beta-corp"
}
}
# Assume you have a client name
# Set the datastream namespace name to your client
mutate {
add_field => {
"[data_stream][namespace]" => "%{client_name}"
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
data_stream => true
data_stream_auto_routing => true
data_stream_dataset => "applogs"
# Some reason this does not work I think it should...
# data_stream_namespace => "%{client_name}"
}
stdout {
codec => rubydebug
}
}
And the output when I set the client to different name Note the 2 data streams with the common suffix and then the client name... the logs prefix is pretty much hard coded...
And a Single Data View for All
And Discover One Data View (or you could create one per client) with all the data
Event test rollover
POST logs-applogs-acme-corp/_rollover
{
"acknowledged" : true,
"shards_acknowledged" : true,
"old_index" : ".ds-logs-applogs-acme-corp-2022.07.26-000001",
"new_index" : ".ds-logs-applogs-acme-corp-2022.07.26-000002",
"rolled_over" : true,
"dry_run" : false,
"conditions" : { }
}
This was good for me to go all through too!!