Description
I'm using fluentd to send logs to Elasticsearch 7.10.0 / Kibana 7.10.0
Now without applying any IndexTemplate and LifecyclePolicy I face no issues. However, After applying an IndexTemplate and LifecyclePolicy to a specific index I can directly see all the new logs in the Discover tab. So far so good. After this if I try to add/edit some fields from the fluentd side with new values, I can't see the changes in the Discover tab directly. For some reason it'll be displayed after a while minimum one hour.
Used configurations
Fluentd configs:
<source>
@type sample
sample {"hello":"world"}
tag sample
</source>
<filter sample.**>
@type record_transformer
<record>
hostname "${hostname}"
environment "testenv"
Year "21"
month "3"
day "1"
</record>
</filter>
<match sample.**>
@type elasticsearch
with_transporter_log true
@log_level info
host 1.2.3.4
port 443
scheme https
logstash_format false
index_name new_index-1
rollover_index true
reconnect_on_error true
reload_connections false
reload_on_failure true
client_cert /path/to/cert
client_key /path/to/key
<buffer>
@type file
path /path/to/buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 8
flush_interval 5s
retry_forever
retry_max_interval 30
overflow_action block
</buffer>
</match>
On Kibana:
The lifecycle Policy
{
"iPolicy-test" : {
"version" : 5,
"modified_date" : "2021-02-24T10:28:15.104Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50kb",
"max_age" : "12h"
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : {
"min_age" : "1d",
"actions" : { }
}
}
}
}
}
The index template with dynamic mapping
{
"index_templates" : [
{
"name" : "testing_template",
"index_template" : {
"index_patterns" : [
"new_index*"
],
"template" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "iPolicy-test",
"rollover_alias" : "rollover_alias_new"
},
"mapping" : {
"ignore_malformed" : "true"
},
"search" : {
"slowlog" : {
"threshold" : {
"fetch" : {
"debug" : "0ms",
"info" : "1s"
},
"query" : {
"debug" : "0ms",
"info" : "1s"
}
}
}
},
"refresh_interval" : "10s",
"translog" : {
"sync_interval" : "1m"
},
"query" : {
"default_field" : "time"
},
"priority" : "99",
"number_of_replicas" : "1"
}
},
"mappings" : {
"_source" : {
"enabled" : true
},
"dynamic_templates" : [
{
"no_index_fields_past_depth_2" : {
"path_match" : "*.*",
"mapping" : {
"type" : "object",
"enabled" : false
},
"match_mapping_type" : "object"
}
},
{
"create_keyword_index_for_all_string_fields" : {
"mapping" : {
"normalizer" : "lowercase_normalizer",
"ignore_above" : 1000,
"type" : "keyword"
},
"match_mapping_type" : "string",
"match" : "*"
}
}
]
}
},
"composed_of" : [ ],
"priority" : 99
}
}
]
}
With theses configs the rolling_over feature is working fine with the index alias rollover_alias_new
GET _cat/aliases/rollover_alias_new\?v
alias index filter routing.index routing.search is_write_index
rollover_alias_new new_index-000007 - - - false
rollover_alias_new new_index-000005 - - - false
rollover_alias_new new_index-000008 - - - false
rollover_alias_new new_index-000002 - - - false
rollover_alias_new new_index-000003 - - - false
rollover_alias_new new_index-000004 - - - false
rollover_alias_new new_index-000009 - - - true
rollover_alias_new new_index-000006 - - - false
rollover_alias_new new_index-1 - - - false
I created an Index pattern using the alias name.
If I edit/add fields in the fluentd configs I can see the changes after refreshing the field list on the index pattern directly. However, under the discover tab these changes aren't shown there.
To Reproduce
-
Setting up fluentd client with the above mentioned configs without starting fluentd service yet.
-
Create the lifecycle policy as above.
-
Create the the index template as above.
-
Create a new index with an alias as follow:
PUT new_index-1 { "aliases": { "rollover_alias_new": { "is_write_index": true } } } -
Start the fluentd service.
Expected behavior
The new field's information and the new logs should be displayed in the Discover tab up-to-date without any delay, knowing that there is no filters applyed.
Actual behavior
The Discover tab keep showing the old fields/logs for a couple of hours before finally the updates are showed up.
Surely, I can make a new created field displayed in the Discover tab if I deactivate the "Hide missing fields" option. However, no data will be exposed.
I did try also using the static mapping instead of dynamic mapping with the same fields that are in the fluentd configs but still the same behaviour.