Heartbeat 8.5.0 Parsing Errors

[christophercutajar]

Hi team,

After upgrading our heartbeat deployment from 8.4.3 to 8.5.0, heartbeat data ingestion stopped.

From the logs, we can see that the reason why the event cannot be ingested is due to parsing issues.

\"reason\":\"failed to parse date field [4.0614878e+07] with format [strict_date_optional_time||epoch_millis]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"date_time_parse_exception: Failed to parse with all enclosed parsers\"}}}

The only way to solve the issue was to revert back the beat to 8.4.3.

Could we have your feedback on this, please?

Thanks

[warkolm (Mark Walkom)]

Welcome to our community!

Can you share your config?

[christophercutajar]

Hi @warkolm ,

It's a very basic config with multiple monitors. Here is a sample of the config with only 1 monitor:

############################# Heartbeat ######################################
    # Define a directory to load monitor definitions from. Definitions take the form
    # of individual yaml files.
    heartbeat.config.monitors:
      # Directory + glob pattern to search for configuration files
      path: ${path.config}/monitors.d/*.yml
      # If enabled, heartbeat will periodically check the config.monitors path for changes
      reload.enabled: false
      # How often to check for changes
      reload.period: 5s
    # Configure monitors inline
    heartbeat.monitors:
    # Monitor 1
    - type: http
      name: "monitor-1"
      # List or urls to query
      urls: ["https://<monitor-1-url>"]
      # Configure task schedule
      schedule: '@every 1m'

    processors:
      - add_kubernetes_metadata:
          in_cluster: true

    #==================== Elasticsearch template setting ==========================
    setup.template.enabled: false

    # setup.template.settings:
    #   index.number_of_shards: 1
    #   index.codec: best_compression
    #   #_source.enabled: false
    #================================ Outputs =====================================
    # Configure what output to use when sending the data collected by the beat.
    #-------------------------- Elasticsearch output ------------------------------
    output.elasticsearch:
      hosts: 'https://${ELASTICSEARCH_ADDRESS}:443'
      username: '${BEATS_USERNAME}'
      password: '${BEATS_PASSWORD}'

[Andrew_Cholakian1 (Andrew Cholakian)]

This is a bug in Heartbeat 8.5.0 that we're actively working to fix in an upcoming patch. There is a simple workaround described here: Heartbeat 8.5.0 mapper error workaround if you're currently experiencing it.

[stephenb (Stephen Brown)]

@christophercutajar 8.5.1 Is out if you want to give it a try!

[christophercutajar]

@stephenb yeah we already upgraded our heartbeat deployment to 8.5.1. We're not seeing any errors and the issue is resolved. Thanks everyone.