Current problem: Metricbeat sends timestamp as keyword type instead of date type. Have to use a separate command with the old index template api to update mapping.
Background:
Our ELK stack was upgraded from 6.8.23 to 7.10.2. Metricbeat also to 7.10.2. Soon after this, Elasticsearch cluster went to red state because it ran out of heap etc., so we uninstalled metricbeat in our application kubernetes clusters so that metric data would not be sent to elasticsearch. Now I am trying to fix this with ELK stack 7.10.2 and Metricbeat 7.10.2
I found there were more than 3000 fields that were being sent to elasticsearch by metricbeat (though this was the same before the elk upgrade) though most of them did not have any data and so I was at least not able to search with text of the field names like "zookeeper", "ceph", "couchdb" etc which we do not use. I thought this was causing "field explosion" and so I created a customised fields.yml file and added the following to my metricbeat.yml file
setup.template.fields: ${path.config}/trimmed_fields.yml
which contained only those fields that we were interested in. Although the number of fields now reduced as per the trimmed_fields.yml, However in the indexes that were getting created the timestamp field in each document appeared as type "keyword" rather than as type "date" ( )
GET /monitoring-2023.02.24/_mapping
"date_detection" : false,
"properties" : {
"@timestamp" : {
"type" : "keyword",
"ignore_above" : 1024
},
I tried to update the mapping using the REST api of new template endpoint but it did not change the data type of timestamp in the mapping. I tried using following after using the add_field so but it didnt work either (timestamp field still appeared as type string in kibana).
setup.template.append_fields:
- name: aq_timestamp
type: date
processors:
- add_cloud_metadata: ~
- copy_fields:
fields:
- from: "@timestamp"
to: aq_timestamp
fail_on_error: false
ignore_missing: true
I then tried the old template endpoint REST API and this was successful. i.e. It changed the datatype of timestamp to date in the mapping.
PUT _template/monitoring
{
"index_patterns": [
"monitoring-*"
],
"template": {
"mappings": {
"properties": {
"timestamp": {
"type": "date",
"format": "date_optional_time"
}
},
"date_detection" : true
}
},
"version": 125
}
My question is :
- Why are the old template endpoint APIs effective with ELK 7.10.2 and not the new template api endpoint. It is so with all index template related APIs like GET, DELETE, PUT verbs.
- Why the timestamp field is set by metricbeat as type keyword because as per metricbeat ECS fields documentation it should be of type date.
- If fields are not set any values (most of the default fields sent by metricbeat e.g. zookeeper, ceph which we dont use etc) , then do they consume resources in the elasticsearch cluster which could degrade its performace???
- Any ideas on what might have caused ES cluster to go to red state. (we have multiple test environments and production environments all connecting to the same ES cluster)
MOST IMPORTANT QUESTION:
How to make timestamp field be set as type date by default which is the specification in ECS fields | Metricbeat Reference [8.6] | Elastic
All this is very puzzling for me. I am still relatively new with elastic components.
My metricbeat.yml file is:
name: ${NODE_NAME}
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
metricbeat.max_start_delay: 15s
timeseries.enabled: true
timestamp.precision: milliseconds
setup.ilm.enabled: false
setup.template.pattern: "monitoring-*"
setup.template.fields: ${path.config}/trimmed_fields.yml
setup.template.overwrite: false
{{ if .Values.config.kibana_host_username }}
username: {{ .Values.config.kibana_host_username }}
{{ end }}
{{ if .Values.config.kibana_host_password }}
password: {{ .Values.config.kibana_host_password }}
{{ end }}
output.elasticsearch:
{{- $host_port := .Values.config.host_port }}
hosts: [ {{- range .Values.config.host }}{{ . }}:{{ $host_port }},{{- end }} ]
username: {{ .Values.config.host_username }}
password: {{ .Values.config.host_password }}
protocol: {{ .Values.config.host_scheme }}
index: "monitoring-%{+yyyy.MM.dd}"