Elasticsearch json logging @timestamp missing

[logger (Malte Leek)]

Hello there,

I have a problem with elasticsearch logging and the filebeat module.

I collect the logs from elasticsearch with the filebeat module.
For example the deprecation logs:

- module: elasticsearch
  deprecation:
    enabled: true
    var.paths:
      - /var/log/elasticsearch/*_deprecation.json

Logging path on the system: "/var/log/elasticsearch/elastic_deprecation.json"

I have loaded all pipelines from filebeat:

• filebeat-7.17.5-elasticsearch-deprecation-pipeline
• filebeat-7.17.5-elasticsearch-deprecation-pipeline-json
• filebeat-7.17.5-elasticsearch-deprecation-pipeline-plaintext

Then I send it to our central logstashes and they will put it into elasticsearch:

logstash pipeline:

elasticsearch {
      hosts => ["https://elasticsearch-server:9220"] 
      index => "%{[@metadata][beat]}-int" 
      user => logstash_writer
      password => "<password>"
      pipeline => "%{[@metadata][pipeline]}"
      manage_template => false
      ssl => true
      manage_template => false
      ilm_enabled => false
      action => create
  }

It is a datastream and it works so far. But the elasticsearch logs do not log the timestamp correctly. The create a timestamp and put it into the field "timestamp" but the pipeline wants a "@timestamp" field to parse on.

I know I can just change the pipeline to this correctly. But in the next update process this would happen again and I dont want to do it manually everytime.

logstash logs this Warning:

{
  "level": "WARN",
  "loggerName": "logstash.outputs.elasticsearch",
  "timeMillis": 1659100307439,
  "thread": "[filebeat-int]>worker0",
  "logEvent": {
    "message": "Could not index event to Elasticsearch.",
    "status": 400,
    "action": [
      "create",
      {
        "_index": "filebeat-int",
        "pipeline": "filebeat-7.17.5-elasticsearch-deprecation-pipeline"
      },
      {
        "type": "deprecation.elasticsearch",
        "log": {
          "offset": 13609643,
          "file": {
            "path": "/var/log/elasticsearch/elastic_deprecation.json"
          }
        },
        "input": {
          "type": "log"
        },
        "service": {
          "environment": "int",
          "type": "elasticsearch"
        },
        "ecs": {
          "version": "1.12.0"
        },
        "host": {
          "name": "apm-int1"
        },
        "level": "DEPRECATION",
        "fileset": {
          "name": "deprecation"
        },
        "node.name": "apm-int1-2",
        "timestamp": "2021-11-11T07:55:22,480+01:00",
        "agent": {
          "type": "filebeat",
          "id": "21a8b519-2d08-41ac-81f5-e5fed36b42ff",
          "ephemeral_id": "297a785c-aefd-40d4-84dd-057352e9a1e7",
          "version": "7.17.5",
          "name": "apm-int1",
          "hostname": "host1"
        },
        "component": "o.e.d.c.s.Settings",
        "message": "[xpack.security.transport.ssl.keystore.password] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.",
        "cluster.name": "elastic",
        "@timestamp": "2022-07-29T13:11:45.052Z",
        "@version": "1",
        "event": {
          "module": "elasticsearch",
          "dataset": "elasticsearch.deprecation"
        }
      }
    ],
    "response": {
      "create": {
        "_index": ".ds-filebeat-int-2022.07.26-000017",
        "_type": "_doc",
        "_id": "F9sVSoIB2y2_gvQ4n4_c",
        "status": 400,
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "failed to parse",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "data stream timestamp field [@timestamp] is missing"
          }
        }
      }
    }
  }
}

The log line is:

{
  "type": "deprecation.elasticsearch",
  "timestamp": "2022-07-29T12:27:50,414+02:00",
  "level": "WARN",
  "component": "o.e.d.c.s.Settings",
  "cluster.name": "elastic-int",
  "node.name": "apm-int1-1",
  "message": "[xpack.security.transport.ssl.keystore.password] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.",
  "key": "xpack.security.transport.ssl.keystore.password",
  "category": "settings",
  "elasticsearch.elastic_product_origin": "beats",
  "cluster.uuid": "d_CHXTaaGdIpVYHbbrw",
  "node.id": "8gHrgv04Ccgc4cHynV5g"
}

As you can see the is only a "timestamp" field and no "@timestamp"

Can anybody give me a hint? Or does anyone know the solution?

Greetings
Malte