Optional Fields Create Mapper Parsing Error

londonx · January 9, 2018, 3:20pm

I use optional fields in Heartbeat and since the 6.x release I have been getting mapper_parsing errors that fail to index the event on Elastic Search because of any optional field, once they are removed everything works.

I get an error even when pointing to a fresh new index and after reloading the default template ${path.config}/fields.yml, plus using the default template name of "heartbeat.version" etc

The fields are pretty basic e.g.

fields:
env: qa

e.g.
WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to find type parsed [keyword] for [env]"}
2018-01-08T13:46:38Z INFO Non-zero metrics in the last 30s: beat.memstats.gc_n

Note using Elastic Search v2.4.2, I did notice on a beta/6.0 release of heartbeat there was a slightly different formatting of the Template section in that you had to chose a specific mapping template when using v2.x Elastic but in the latest Heartbeat it seems the template section has been simplified.

steffens · January 10, 2018, 11:46am

Can you share your complete heartbeat config? The old and the new one please. Please format using the </>-Button.

Seems to complain about type being keyword. Not sure, but I think the types keyword and text have been introduced with Elasticseach 5.0. That is, the template installed is incompatible with your Elasticsearch version.

Template loading has been rewritten in 6.0. We have a file called fields.yml, defining all fields. This file is used at runtime to create the Elasticsearch mapping compatible with the Elasticsearch version being used. Running heartbeat export template --es.version 2.4.2 will print an template mapping for use with Elasticsearch 2.x to console.

Just noticed there might be a bug when exporting for 2.4. The dynamic_mapping section still uses keyword.

Can you try to modify the template by hand and confirm fixing the template works? Our docs do contain some instructions for manually loading templates on different OSes. With 6.0 we started versioning index names and templates. Make sure you use the correct version numbers, otherwise you might end up with 2 template for the same index (with the chance of the wrong/old template being applied).

Export template via heartbeat export template --es.version 2.4.2 > heartbeat.template.json
Search for dynamic_templates at the top of the document and replace all occurences of "mapping": { "type": "keyword" } with "mapping": { "ignore_above": 1024, "index": "not_analyzed", "type": "string"}.
Overwrite the template via curl -XPUT -H 'Content-Type: application/json' http://<es_url>:9200/_template/heartbeat-6.0.0 -d@heartbeat.template.json
Drop the index (templates are only applied when an index is generated): curl -XDELETE http://<es_url>:9200/<index_name>

londonx · January 10, 2018, 11:59am

Hi thanks, just noticed I'm running v6.0.0 of Heartbeat (i`ll try and update it later),

Also note I dont have access to the Elastic Search cluster myself so will be difficult for me to dig deeper there.

v6.0.0 config here:

################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.full.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Configure monitors
heartbeat.monitors:

- type: tcp 
  
  # Monitor name used for job name and document type
  name: Tibco_APAC

  # Enable/Disable monitor
  enabled: true

  # Configure task schedule
  schedule: '@every 30s'

  # configure hosts to ping.
  hosts: ["xxxxx:7222"]
  
- type: tcp 
  
  # Monitor name used for job name and document type
  name: Tibco AMERS

  # Enable/Disable monitor
  enabled: true

  # Configure task schedule
  schedule: '@every 30s'

  # configure hosts to ping.
  hosts: ["xxxxx:7222"]  
  


#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: heartbeat

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
fields:
  env: qa
  region: emea

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["xxxxxxxxxxxxxxxxxxxxxxx"]

 
  # Optional index name. The default is "heartbeat" plus date
  # and generates [heartbeat-]YYYY.MM.DD keys.
  index: "xxxxxxxxxxxxx-heartbeat-m-%{+YYYY.MM}"

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#============================== Template =====================================

# A template is used to set the mapping in Elasticsearch
# By default template loading is enabled and the template is loaded.
# These settings can be adjusted to load your own template or overwrite existing ones.

# Set to false to disable template loading.
setup.template.enabled: true

# Template name. By default the template name is "heartbeat-%{[beat.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.name: "heartbeat-%{[beat.version]}"

# Template pattern. By default the template pattern is "-%{[beat.version]}-*" to apply to the default index settings.
# The first part is the version of the beat and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.pattern: "xxxxxxxxxxxxx-heartbeat-%{[beat.version]}-m-*"

# Path to fields.yml file to generate the template
setup.template.fields: "${path.config}/fields.yml"

# Overwrite existing template
setup.template.overwrite: false

# Elasticsearch template settings
#setup.template.settings:

  # A dictionary of settings to place into the settings.index dictionary
  # of the Elasticsearch template. For more details, please check
  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
  #index:
    #number_of_shards: 1
    #codec: best_compression
    #number_of_routing_shards: 30

  # A dictionary of settings for the _source field. For more details, please check
  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html
  #_source:
    #enabled: false
  

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: info

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

# Logging to rotating files files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true
logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: D:\Elastic\logs

  # The name of the files where the logs are written to.
  name: heartbeat

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated
  rotateeverybytes: 10485760 # = 10MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

londonx · January 10, 2018, 12:10pm

Heartbeat Version: 5.5.2 config (pointed to a different Elastic Cluster):

################### Heartbeat Configuration Example #########################

# This file is an example configuration file highlighting only some common options.
# The heartbeat.full.yml file in the same directory contains all the supported options
# with detailed comments. You can use it for reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/heartbeat/index.html

############################# Heartbeat ######################################

# Configure monitors
heartbeat.monitors:

# xxxxxxxx Monitor
- type: tcp
  name: xxxx-tcp
  enabled: true
  schedule: '@every 5m' # every x minutes from start of beat
  timeout: 16s
  hosts:
    -  "xxxxxx:xxxx"
    -  "xxxxxx:xxxx"	

# xxxxx Monitor	
- type: tcp
  name: xxxxx-tcp
  enabled: false
  schedule: '@every 5m'
  timeout: 16s  
  hosts:
    -  "xxxxxx:xxxx"
    -  "xxxxxx:xxxx"

# xxxxxx Monitor		
- type: tcp
  name: xxxxx-tcp
  enabled: true
  schedule: '@every 5m'
  timeout: 16s
  hosts:
    -  "xxxxxx:xxxx"
    -  "xxxxxx:xxxx"

# xxxxxxx Monitor	
- type: tcp
  name: xxxxxxxxx-tcp
  enabled: true
  schedule: '@every 5m'
  timeout: 16s
  hosts: 
    -  "xxxxxx:xxxx"
  
# xxxxxxx	
- type: tcp
  name: xxxxxxxx-tcp
  enabled: true
  schedule: '@every 5m'
  timeout: 16s
  hosts: 
    -  "xxxxxx:xxxx"
  
# xxxxxxxxURL Monitor	  
- type: http
  name: "xxxxxxxx-http"
  enabled: true
  schedule: '@every 5m'
  timeout: 16s
  urls: 
    -  "https://xxxxxxxxxxxx"
    -  "https://xxxxxxxxxxxx"
    -  "https://xxxxxxxxxxxx"
    -  "https://xxxxxxxxxxxx"
  
  
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
fields:
  env: London-Desktop

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"]

 
  # Optional index name. The default is "heartbeat" plus date
  # and generates [heartbeat-]YYYY.MM.DD keys.
  index: "xxxxxxxxxxxxxx-%{+YYYY.MM}"

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"


#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: info

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

# Logging to rotating files files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true
logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: C:\Logstash\logs

  # The name of the files where the logs are written to.
  name: heartbeat

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated
  rotateeverybytes: 10485760 # = 10MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

londonx · January 10, 2018, 12:32pm

Can you try to modify the template by hand and confirm fixing the template works?

Just a thought, does my heartbeat index now need to include the version number as well if the template.pattern includes it?

index: "xxxxxxxxxxxxx-heartbeat-m-%{+YYYY.MM}"

setup.template.pattern: "xxxxxxxxxxxxx-heartbeat-%{[beat.version]}-m-*"

steffens · January 10, 2018, 12:41pm

Yes, you will also need the version number.

You will also need to remove all old templates, so they don't match by accident (pre 6.x templates are unversioned and might match by accident).

system · February 7, 2018, 12:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.