Hi,
I have built an ES with architect such as picture. And deploy it on AWS EKS.
My expected is filebeat will collect logs all pods on EKS. And send it to Logstash.
And Logstash will send this to Elasticsearch.
At Logstash, I want use Index Lifecycle Policies feature to management Index.
I want to create each Index to Deployment. (Have than 100 Deployment)
I cannot dynamic index follow to %{[fields][log_type]}.
I am using if else at output and specific ilm_rollover_alias and ilm_policy to each service.
logstash.conf
input {
beats {
port => 5044
}
}
filter {
mutate { remove_field => ["input_type", "[event][original]", "[tags]", "[fields][fields_under_root]" ] }
if [kubernetes][labels][logstyle] == "nginx" {
#Nginx
grok {
match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]}( \"%{DATA:[nginx][access][referrer]}\")?( \"%{DATA:[nginx][access][agent]}\")?",
"%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \\[%{HTTPDATE:[nginx][access][time]}\\] \"-\" %{NUMBER:[nginx][access][response_code]} -" ] }
}
# date {
# match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
# remove_field => "[nginx][access][time]"
# }
useragent {
source => "[nginx][access][agent]"
target => "[nginx][access][user_agent]"
remove_field => "[nginx][access][agent]"
}
geoip {
source => "[nginx][access][remote_ip]"
target => "[nginx][access][geoip]"
}
}
else if [kubernetes][pod][labels][app] == "filebeat" {
#filebeat
grok {
match => [ "message", "(?<timestamp>%{TIMESTAMP_ISO8601})\s+%{LOGLEVEL:level}\s+%{DATA}\s+%{GREEDYDATA:logmessage}" ]
}
}
else {
#HTD java
grok {
match => [ "message", "(?<timestamp>%{TIMESTAMP_ISO8601}) - \[(?<thread>[A-Za-z0-9-]+)\] %{LOGLEVEL:level}\s+(?<class>[A-Za-z0-9.]*\.[A-Za-z0-9#_]+)\s* - %{GREEDYDATA:logmessage}" ]
}
}
}
output {
if [event][module] == "nginx" {
elasticsearch {
hosts => ["https://es.example.com:443"]
user => 'logstash-test'
password => '${LOGSTASH_PW}'
index => "testenv-%{[event][module]}-%{+yyyy.MM.dd}"
ilm_enabled => true
ilm_rollover_alias => "testenv-nginx"
ilm_pattern => "{now/d}-000001"
ilm_policy => "testenv-nginx-policy"
#cacert => "/usr/share/logstash/config/elasticsearch-certs/ca.crt"
}
}
else if [fields][log_type] == "service-1" {
elasticsearch {
hosts => ["https://es.example.com:443"]
user => 'logstash-test'
password => '${LOGSTASH_PW}'
index => "testenv-%{[fields][log_type]}-%{+yyyy.MM.dd}"
ilm_enabled => true
ilm_rollover_alias => "testenv-service-1"
ilm_pattern => "{now/d}-000001"
ilm_policy => "testenv-service-1-policy"
#cacert => "/usr/share/logstash/config/elasticsearch-certs/ca.crt"
}
}
else if [fields][log_type] == "service-2" {
elasticsearch {
hosts => ["https://es.example.com:443"]
user => 'logstash-test'
password => '${LOGSTASH_PW}'
index => "testenv-%{[fields][log_type]}-%{+yyyy.MM.dd}"
ilm_enabled => true
ilm_rollover_alias => "testenv-service-2"
ilm_pattern => "{now/d}-000001"
ilm_policy => "testenv-service-2-policy"
#cacert => "/usr/share/logstash/config/elasticsearch-certs/ca.crt"
}
}
else {
elasticsearch {
ilm_enabled => false
hosts => ["https://es.example.com:443"]
user => 'logstash-test'
password => '${LOGSTASH_PW}'
index => "testenv-%{[fields][log_type]}-%{+yyyy.MM.dd}"
#cacert => "/usr/share/logstash/config/elasticsearch-certs/ca.crt"
}
}
}
But when use this solution, time to start Logstash takes a long time (~= 20 minutes with 130 service). Because have than 100 output.
Log went start Logstash
[INFO ] 2023-08-22 07:36:00.050 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://es.example.com:443"]}
[INFO ] 2023-08-22 07:36:00.062 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash-stag:xxxxxx@es.example.com:443/]}}
[WARN ] 2023-08-22 07:36:00.104 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash-stag:xxxxxx@es.example.com:443/"}
[INFO ] 2023-08-22 07:36:00.109 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.7.1) {:es_version=>8}
[WARN ] 2023-08-22 07:36:00.109 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2023-08-22 07:36:00.115 [[main]-pipeline-manager] elasticsearch - Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"ilm_enabled"=>"true", "ilm_rollover_alias"=>"stagenv-truemoney", "ilm_policy"=>"stagenv-truemoney-policy", "ilm_pattern"=>"{now/d}-000001"}
[INFO ] 2023-08-22 07:36:00.115 [[main]-pipeline-manager] elasticsearch - Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[WARN ] 2023-08-22 07:36:00.116 [[main]-pipeline-manager] elasticsearch - Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[INFO ] 2023-08-22 07:36:00.121 [Ruby-0-Thread-64: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:161] elasticsearch - Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[INFO ] 2023-08-22 07:36:20.584 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://es.example.com:443"]}
[INFO ] 2023-08-22 07:36:20.587 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash-stag:xxxxxx@es.example.com:443/]}}
[WARN ] 2023-08-22 07:36:20.622 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash-stag:xxxxxx@es.example.com:443/"}
[INFO ] 2023-08-22 07:36:20.628 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.7.1) {:es_version=>8}
[WARN ] 2023-08-22 07:36:20.628 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2023-08-22 07:36:20.639 [[main]-pipeline-manager] elasticsearch - Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"ilm_enabled"=>"true", "ilm_rollover_alias"=>"stagenv-gametokencredentialservice", "ilm_policy"=>"stagenv-gametokencredentialservice-policy", "ilm_pattern"=>"{now/d}-000001"}
[INFO ] 2023-08-22 07:36:20.639 [[main]-pipeline-manager] elasticsearch - Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[WARN ] 2023-08-22 07:36:20.640 [[main]-pipeline-manager] elasticsearch - Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[INFO ] 2023-08-22 07:36:20.640 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://es.example.com:443"]}
[INFO ] 2023-08-22 07:36:20.643 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash-stag:xxxxxx@es.example.com:443/]}}
[INFO ] 2023-08-22 07:36:20.699 [Ruby-0-Thread-66: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:161] elasticsearch - Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[WARN ] 2023-08-22 07:36:20.706 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash-stag:xxxxxx@es.example.com:443/"}
[INFO ] 2023-08-22 07:36:20.711 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (8.7.1) {:es_version=>8}
[WARN ] 2023-08-22 07:36:20.711 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[INFO ] 2023-08-22 07:36:20.718 [[main]-pipeline-manager] elasticsearch - Not eligible for data streams because config contains one or more settings that are not compatible with data streams: {"ilm_enabled"=>"true", "ilm_rollover_alias"=>"stagenv-nginx", "ilm_policy"=>"stagenv-nginx-policy", "ilm_pattern"=>"{now/d}-000001"}
[INFO ] 2023-08-22 07:36:20.718 [[main]-pipeline-manager] elasticsearch - Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[WARN ] 2023-08-22 07:36:20.728 [[main]-pipeline-manager] elasticsearch - Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[INFO ] 2023-08-22 07:36:20.747 [Ruby-0-Thread-68: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.13.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:161] elasticsearch - Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
Please support for optimize time start Logstash or different solution to output Logstash!!!
Thanks!