Hi Team,
I am deploying elasticsearch cluster with latest version i.e 7.14 through automation tool. The current config (v 7.4) is creating daily indices and it is not having data_stream, ILM policy, index_template etc..
I am trying to use latest features like data stream, ILM policy and mention them in logstash pipeline config file to apply these to new indices when it is created.
beats ---> logstash --> elasticsearch.
Currently i have logstash pipeline file like below.
input {
beats {
port => 5044
}
}
filter {
if [log_type] == "app_server" and [app_id] == "app"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TraceID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "app_server"
}
}
}
if [log_type] == "access_server" and [app_id] == "as"
{
grok { match => { "message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}(?::?%{SECOND})\| %{USERNAME:exchangeId}\| %{DATA:trackingId}\| %{NUMBER:RoundTrip:int}%{SPACE}ms\| %{NUMBER:ProxyRoundTrip:int}%{SPACE}ms\| %{NUMBER:UserInfoRoundTrip:int}%{SPACE}ms\| %{DATA:Resource}\| %{DATA:subject}\| %{DATA:authmech}\| %{DATA:scopes}\| %{IPV4:Client}\| %{WORD:method}\| %{DATA:Request_URI}\| %{INT:response_code}\| %{DATA:failedRuleType}\| %{DATA:failedRuleName}\| %{DATA:APP_Name}\| %{DATA:Resource_Name}\| %{DATA:Path_Prefix}" } }
mutate {
replace => {
"[type]" => "access_server"
}
}
}
output {
if [log_type] == "app_server" {
elasticsearch {
hosts => ['http://es_ip:9200']
index => "%{type}-%{+YYYY.MM.dd}"
user => elastic
password => xxx
}
}
if [log_type] == "access_server" {
elasticsearch {
hosts => ['http://es_ip:9200']
index => "%{type}-%{+YYYY.MM.dd}"
user => elastic
password => xxx
}
}
elasticsearch {
hosts => ['http://es_ip:9200']
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => elastic
password => xxx
}
}
So the index i am actually going to run queries against is currently getting created daily as,
access_server-2021.08.25
access_server-2021.08.26
access_server-2021.08.27
Before actually using index_template, ILM policy, it must first be created.
Below is the index_template i am planning to use,
PUT _index_template/access_template
{
"version": 1,
"priority": 500,
"template": {
"settings": {
"index.number_of_shards": 1,
"index.number_of_replicas": 0,
"index.lifecycle.name": "testpolicy",
"index.lifecycle.rollover_alias": "access_server-alias"
},
"mappings": {
"dynamic": true,
"numeric_detection": true,
"date_detection": true,
"dynamic_date_formats": [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"_source": {
"enabled": true,
"includes": [],
"excludes": []
},
"_routing": {
"required": false
},
"dynamic_templates": []
}
},
"index_patterns": [
"test-data-stream*"
],
"data_stream": {}
}
Below is the ILM policy,
So this will perform rollover when either of the condition meets and then keep index in delete phase for 2 days and then performs delete.
PUT _ilm/policy/testpolicy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "2d",
"max_primary_shard_size": "900mb"
},
"set_priority": {
"priority": 100
}
},
"min_age": "0ms"
},
"delete": {
"min_age": "2d",
"actions": {
"delete": {}
}
}
}
}
}
Bootstrap the initial time series index with a write index alias.
curl -X PUT "localhost:9200/access-000001?pretty" -H 'Content-Type: application/json' -d'
{
"aliases": {
"access-alias":{
"is_write_index": true
}
}
}'
Create the data stream
the link says, You can also manually create the stream using the create data stream API. The stream’s name must still match one of your template’s index patterns.
curl -X PUT "localhost:9200/_data_stream/test-data-stream?pretty"
In logstash pipeline config file, mentioning all above created.
template => "access_template.json"
template_name => "access_template"
template_overwrite => "false"
i.e
output {
if [log_type] == "access_server" {
elasticsearch {
hosts => ['http://es_ip:9200']
index => "%{type}-%{+YYYY.MM.dd}"
user => elastic
password => xxx
template => "access_template.json"
template_name => "access_template"
template_overwrite => "false"
}
}
(removed index => "%{type}-%{+YYYY.MM.dd}" in above, so that it will not create daily indices)
Q. i am confused, what data stream will name the index so somewhere above i am doing mistake in mentioning index_pattern, data_stream name so name mismatch might happen.
Q. Can i use it like above in logstash config to apply data_stream, ILM policy to apply to new indices?
Thanks,