Type usage in libbeat/outputs/elasticsearch/client causes issue on OpenSearch 2.0+ engine

[Suraj_Singh]

With _type removal in OpenSearch 2.0+ (previously deprecated in Elasticsearch 7.x); beat still uses _type as DocType while building BulkRequest meta data; resulting in illegal_argument_exception from OpenSearch engine.

Exception to end user

400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

BulkMeta in beat

type BulkMeta struct {
	Index    string `json:"_index" struct:"_index"`
	DocType  string `json:"_type,omitempty" struct:"_type,omitempty"`
	Pipeline string `json:"pipeline,omitempty" struct:"pipeline,omitempty"`
	ID       string `json:"_id,omitempty" struct:"_id,omitempty"`
}

Code Link: beats/client.go at 7.12 · elastic/beats · GitHub

Issue on OpenSearch: [BUG] 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]" · Issue #3484 · opensearch-project/OpenSearch · GitHub

[system]

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[warkolm (Mark Walkom)]

Unless this is an issue in Elasticsearch, we won't be making changes to Beats to resolve this sorry to say.