# Failed to encode event: unsupported float value

**URL:** <https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [September 12, 2019, 2:18pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270 "2019-09-12T14:18:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 12, 2019, 2:18pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/1 "2019-09-12T14:18:03Z")

</div>

I'm doing a POC to use a metricbeat pod to scrape all metrics from a prometheus pod in the same namespace of a kubernetes cluster. The metricbeat pod is able to connect to the prometheus pod's /federate end point and gather results. Something in the result set is causing a JSON processing error. I found another post with a similar error and someone recommended turning the DEBUG logging level on. I did that and harvested a the errors and the associated JSON.  
First, here is the configuration of the metricbeat pod:

#MetricBeat Deployment  
apiVersion: extensions/v1beta1  
kind: Deployment  
metadata:  
name: metricbeat-prometheus  
namespace: monitoring  
labels:  
app: metricbeat-prometheus  
spec:  
template:  
metadata:  
labels:  
app: metricbeat-prometheus  
spec:  
serviceAccountName: metricbeat-prometheus  
terminationGracePeriodSeconds: 30  
hostNetwork: true  
dnsPolicy: ClusterFirstWithHostNet  
containers:  
- name: metricbeat-prometheus  
image: [docker.elastic.co/beats/metricbeat:7.3.1](http://docker.elastic.co/beats/metricbeat:7.3.1)  
args: [  
"-c", "/etc/metricbeat.yml",  
"-e",  
"-system.hostfs=/hostfs",  
]  
env:  
- name: ELASTICSEARCH\_HOST  
value: elasticsearch  
- name: ELASTICSEARCH\_PORT  
value: "9200"  
- name: ELASTICSEARCH\_USERNAME  
value: elastic  
- name: ELASTICSEARCH\_PASSWORD  
value: changeme  
- name: setup.dashboards.enabled  
value: "true"  
- name: setup.kibana.host  
value: "kibana.logging:5601"  
- name: NODE\_NAME  
valueFrom:  
fieldRef:  
fieldPath: spec.nodeName  
securityContext:  
runAsUser: 0  
resources:  
limits:  
memory: 200Mi  
requests:  
cpu: 100m  
memory: 100Mi  
volumeMounts:  
- name: config  
mountPath: /etc/metricbeat.yml  
readOnly: true  
subPath: metricbeat.yml  
- name: modules  
mountPath: /usr/share/metricbeat/modules.d  
readOnly: true  
- name: dockersock  
mountPath: /var/run/docker.sock  
- name: proc  
mountPath: /hostfs/proc  
readOnly: true  
- name: cgroup  
mountPath: /hostfs/sys/fs/cgroup  
readOnly: true  
- name: data  
mountPath: "/data/metricbeat"  
volumes:  
- name: proc  
hostPath:  
path: /proc  
- name: cgroup  
hostPath:  
path: /sys/fs/cgroup  
- name: dockersock  
hostPath:  
path: /var/run/docker.sock  
- name: config  
configMap:  
defaultMode: 0600  
name: metricbeat-daemonset-config  
- name: modules  
configMap:  
defaultMode: 0600  
name: metricbeat-daemonset-modules  
- name: data  
persistentVolumeClaim:  
claimName: metricbeat-prometheus-pvc

---

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 12, 2019, 2:22pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/3 "2019-09-12T14:22:53Z")

</div>

Here are the configmaps the yaml refers to:  
#MetricBeat Modules ConfigMap  
apiVersion: v1  
kind: ConfigMap  
metadata:  
name: metricbeat-daemonset-modules  
namespace: monitoring  
labels:  
app: metricbeat-prometheus  
data:  
prometheus.yml: |-  
- module: prometheus  
period: 10s  
hosts: ["prometheus-k8s.monitoring:9090"]  
metrics\_path: "/federate"  
query:  
match: '{_name_)!=""}'  
#username: "user"  
#password: "secret"

```
  # This can be used for service account based authorization:
  # bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  #ssl.certificate_authorities:
  # - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
```

---

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 12, 2019, 2:28pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/4 "2019-09-12T14:28:17Z")

</div>

Here is the other configmap:

#MetricBeat Base ConfigMap  
apiVersion: v1  
kind: ConfigMap  
metadata:  
name: metricbeat-daemonset-config  
namespace: monitoring  
labels:  
app: metricbeat-prometheus  
data:  
metricbeat.yml: |-  
metricbeat.config.modules:  
# Mounted `metricbeat-daemonset-modules` configmap:  
path: {path.config}/modules.d/\*.yml # Reload module configs as they change: reload.enabled: false # To enable hints based autodiscover uncomment this: metricbeat.autodiscover: providers: - type: kubernetes host: {NODE\_NAME}  
hints.enabled: true  
#logging.level: debug  
output.elasticsearch:  
#hosts: ['{ELASTICSEARCH\_HOST:elasticsearch}:{ELASTICSEARCH\_PORT:9200}']  
hosts: '[elasticsearch-host-ip]:9200'  
username: {ELASTICSEARCH\_USERNAME} password: {ELASTICSEARCH\_PASSWORD}

---

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 12, 2019, 2:30pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/5 "2019-09-12T14:30:22Z")

</div>

There is another very similar post where it was recommended to enable the DEBUG logging level. You can see that post [here](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value-nan/168666). I have done that and I have some data captured from a transaction but it's very large. Assuming someone is willing to help, could I send you the file directly instead of posting it here? I didn't see a way to attach a file.

---

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 12, 2019, 2:44pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/6 "2019-09-12T14:44:29Z")

</div>

@kaiyan-sheng do you have any thoughts on the correct way to handle this?

---

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 20, 2019, 11:03am UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/7 "2019-09-20T11:03:45Z")

</div>

Does anyone have any advice to offer? I'm stuck and I really need some help.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [September 24, 2019, 6:21pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/8 "2019-09-24T18:21:24Z")

</div>

Hi @drew.flint Sorry for the late response! I see we just opened [https://github.com/elastic/beats/issues/13750](https://github.com/elastic/beats/issues/13750) to track more NaN problem. I didn't see any error message in your previous comments but I assume this is the issue you are having?

---

<div class="post-metadata">

**Author:** ![drew.flint](https://avatars.discourse-cdn.com/v4/letter/d/f9ae1b/32.png) [@drew.flint](https://discuss.elastic.co/u/drew.flint)\
**Post date:** [September 26, 2019, 6:32pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/9 "2019-09-26T18:32:39Z")

</div>

I was able to identify the NaN labels in Prometheus and built a query to exclude those. However, now I'm only receiving "Unable to decode response from prometheus endpoint". I'm failing in a whole new and exciting way! Any help you may have to offer would be greatly appreciated. Thanks!  
I believe issues 13749 and 13750 are both associated with the Elastic Support case I opened shortly after creating this community post. If the case number would be helpful, I'm glad to get that for you. If it's any help, I'm able to run a curl command and get results from a /bin/bash shell on the metricbeat container.

---

<div class="post-metadata">

**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\
**Post date:** [September 27, 2019, 1:32pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/10 "2019-09-27T13:32:13Z")

</div>

Hmm sorry to hear that 😂 Could you give us more info on the new error you are getting please?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 25, 2019, 1:46pm UTC](https://discuss.elastic.co/t/failed-to-encode-event-unsupported-float-value/199270/11 "2019-10-25T13:46:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.