Metricbeat sends timestamp as keyword type instead of date type to Elasticsearch, old template endpoints work instead of new one

Current problem: Metricbeat sends timestamp as keyword type instead of date type. Have to use a separate command with the old index template api to update mapping.

Background:
Our ELK stack was upgraded from 6.8.23 to 7.10.2. Metricbeat also to 7.10.2. Soon after this, Elasticsearch cluster went to red state because it ran out of heap etc., so we uninstalled metricbeat in our application kubernetes clusters so that metric data would not be sent to elasticsearch. Now I am trying to fix this with ELK stack 7.10.2 and Metricbeat 7.10.2

I found there were more than 3000 fields that were being sent to elasticsearch by metricbeat (though this was the same before the elk upgrade) though most of them did not have any data and so I was at least not able to search with text of the field names like "zookeeper", "ceph", "couchdb" etc which we do not use. I thought this was causing "field explosion" and so I created a customised fields.yml file and added the following to my metricbeat.yml file

    setup.template.fields: ${path.config}/trimmed_fields.yml 

which contained only those fields that we were interested in. Although the number of fields now reduced as per the trimmed_fields.yml, However in the indexes that were getting created the timestamp field in each document appeared as type "keyword" rather than as type "date" ( )

GET /monitoring-2023.02.24/_mapping 
      "date_detection" : false,
      "properties" : {
        "@timestamp" : {
          "type" : "keyword",
          "ignore_above" : 1024
        },

I tried to update the mapping using the REST api of new template endpoint but it did not change the data type of timestamp in the mapping. I tried using following after using the add_field so but it didnt work either (timestamp field still appeared as type string in kibana).

     setup.template.append_fields:
     - name: aq_timestamp
       type: date
    processors:
      - add_cloud_metadata: ~
       - copy_fields:
           fields:
             - from: "@timestamp"
               to: aq_timestamp
           fail_on_error: false
           ignore_missing: true

I then tried the old template endpoint REST API and this was successful. i.e. It changed the datatype of timestamp to date in the mapping.

PUT _template/monitoring
{
  "index_patterns": [
    "monitoring-*"
  ],
  "template": {
    "mappings": {
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "date_optional_time"
        }
      },
      "date_detection" : true
    }
  },
  "version": 125
}

My question is :

  1. Why are the old template endpoint APIs effective with ELK 7.10.2 and not the new template api endpoint. It is so with all index template related APIs like GET, DELETE, PUT verbs.
  2. Why the timestamp field is set by metricbeat as type keyword because as per metricbeat ECS fields documentation it should be of type date.
  3. If fields are not set any values (most of the default fields sent by metricbeat e.g. zookeeper, ceph which we dont use etc) , then do they consume resources in the elasticsearch cluster which could degrade its performace???
  4. Any ideas on what might have caused ES cluster to go to red state. (we have multiple test environments and production environments all connecting to the same ES cluster)

MOST IMPORTANT QUESTION:
How to make timestamp field be set as type date by default which is the specification in ECS fields | Metricbeat Reference [8.6] | Elastic

All this is very puzzling for me. I am still relatively new with elastic components.

My metricbeat.yml file is:

   name: ${NODE_NAME}

    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    metricbeat.max_start_delay: 15s
    
    timeseries.enabled: true
    timestamp.precision: milliseconds

    setup.ilm.enabled: false

    setup.template.pattern: "monitoring-*"
    setup.template.fields: ${path.config}/trimmed_fields.yml
    setup.template.overwrite: false

{{ if .Values.config.kibana_host_username }}
      username: {{ .Values.config.kibana_host_username }}
{{ end }}
{{ if .Values.config.kibana_host_password }}
      password: {{ .Values.config.kibana_host_password }}
{{ end }}

    output.elasticsearch:
      {{- $host_port := .Values.config.host_port }}
      hosts: [ {{- range  .Values.config.host }}{{ . }}:{{ $host_port }},{{- end }} ]
      username: {{ .Values.config.host_username }}
      password: {{ .Values.config.host_password }}
      protocol: {{ .Values.config.host_scheme }}
      index: "monitoring-%{+yyyy.MM.dd}"

Look in the elasticsearch log for when the monitoring-2023.02.24 (or a newer generation) index was created, it should show what template was used. Just to verify that the template you think should be used was really used.

1 Like

I got the solution for the timestamp issue: I added the timestamp into the fields.yml (file containing the customised type) with type date (even though the documentation stated that ES core will send the timestamp by default with this type)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.