Hi, @Voula_Mikr The type in indices, that allowed to group different kind of documents in the same index, is deprecated since Elasticsearch 6.x and removed with 7.x (see here for details, but in short the idea is to keep only the data that look similar in same indices, and to use custom fields to distinguish them if needed).
So you should remove types occurence in fluend configuration (as it's now useless), but I don't think it could cause performance issues, unless you try to index/query huge amount of data.
However you can check:
if there's any pending tasks / relocating shards on your cluster with _cluster/health endpoint
if there is rejected messages (on Elasticsearch logs) that can make your fluentd to buffer unsent data and fill its memory (and more generally if there's any resource saturation on both side)
@John_Xanthopoulos I think that your issue is different, as the data is rejected by Elasticsearch and no mention of type appears on your logs.
You may consider looking:
if your cluster is healthy and no proxy prevent you from writing in it
if there's some mismatch on data types you try to index vs your mapping (for instance, string vs date or so)
if your index is read-only due to a previous saturation, via the <index>/_settings endpoint, there should be no field like index.blocks.read_only_allow_delete (if there is, you can PUT a null value to it to remove the constraint)
Thanks abrx, with regards the log I posted the issue was under the category --> * if there's some mismatch on data types you try to index vs your mapping (for instance, string vs date or so)
With regards this log --> warning: 299 Elasticsearch-7.8.0-8e340a0952a52f4feb290854ae1051e9a83ca9de "[types removal] Specifying types in bulk requests is deprecated. We managed to suppress it by configuring fluentd with "suppress_type_name true" and log was vanished.
Performance issue is still there though, output of cluster health is
So it looks that there is no issue there right? Additionally I dont see any rejected messages neither in fluentd nor in elasticsearch.