Fluentd performance is degraded

[Voula_Mikr (Voula Mikromastora)]

In a deployment with elasticsearch 8.x, fluentd pod performance has been degraded and following log is printed whenever a new message is received and it is about to be delivered in elastic.

warning: 299 Elasticsearch-7.8.0-8e340a0952a52f4feb290854ae1051e9a83ca9de "[types removal] Specifying types in bulk requests is deprecated."

Is this log relevant to the loss of the performance?
What should I check in order to increase performance?
Thank you in advance.

[John_Xanthopoulos]

Hi, we also see these kind of errors..
2021-07-30 10:12:16 +0000 [warn]: #1 dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="pmdata" time=2021-07-30 10:12:14.688068202 +0000 record={"vnf_name"=>"UDM", "vnf_type"=>"UDM", "distinguished_name"=>"NTF3", "counter_name"=>"-counter-name-15", "counter_group_name"=>"counterGroupUDM1", "measurement_time"=>"2021-07-23T10:12:14.687369Z", "counter_value"=>3, "counter_display_name"=>"UDM-NTF3-counterGroupUDM1-counter-name-2", "is_dynamic"=>"True", "date"=>"2021-07-23", "_hash"=>"MjUyMzRhNDctOWFjMC00YTQ0LTlkZTktNGI4YjAyNzQ3M2Vj"}

[abrx]

Hi,
@Voula_Mikr The type in indices, that allowed to group different kind of documents in the same index, is deprecated since Elasticsearch 6.x and removed with 7.x (see here for details, but in short the idea is to keep only the data that look similar in same indices, and to use custom fields to distinguish them if needed).
So you should remove types occurence in fluend configuration (as it's now useless), but I don't think it could cause performance issues, unless you try to index/query huge amount of data.
However you can check:

• if there's any pending tasks / relocating shards on your cluster with _cluster/health endpoint
• if there is rejected messages (on Elasticsearch logs) that can make your fluentd to buffer unsent data and fill its memory (and more generally if there's any resource saturation on both side)

@John_Xanthopoulos I think that your issue is different, as the data is rejected by Elasticsearch and no mention of type appears on your logs.
You may consider looking:

• if your cluster is healthy and no proxy prevent you from writing in it
• if there's some mismatch on data types you try to index vs your mapping (for instance, string vs date or so)
• if your index is read-only due to a previous saturation, via the <index>/_settings endpoint, there should be no field like index.blocks.read_only_allow_delete (if there is, you can PUT a null value to it to remove the constraint)

[John_Xanthopoulos]

@abrx

_cluster/health

Thanks abrx, with regards the log I posted the issue was under the category --> * if there's some mismatch on data types you try to index vs your mapping (for instance, string vs date or so)

With regards this log --> warning: 299 Elasticsearch-7.8.0-8e340a0952a52f4feb290854ae1051e9a83ca9de "[types removal] Specifying types in bulk requests is deprecated. We managed to suppress it by configuring fluentd with "suppress_type_name true" and log was vanished.
Performance issue is still there though, output of cluster health is
'''
{
"active_primary_shards": 53,
"active_shards": 106,
"active_shards_percent_as_number": 100.0,
"cluster_name": "nc1296-admin-ns-zts-lms",
"delayed_unassigned_shards": 0,
"initializing_shards": 0,
"number_of_data_nodes": 2,
"number_of_in_flight_fetch": 0,
"number_of_nodes": 8,
"number_of_pending_tasks": 0,
"relocating_shards": 0,
"status": "green",
"task_max_waiting_in_queue_millis": 0,
"timed_out": false,
"unassigned_shards": 0
}
'''
So it looks that there is no issue there right? Additionally I dont see any rejected messages neither in fluentd nor in elasticsearch.