Filebeat 6.8 changing floating point zero to integer in json

[Zulu (Zee Nastalski)]

Hi,
I am using filebeat to process telemetry data in json format from my application. The application generates files with json objects. When I need to log the floating point number zero, I log it as 0.0 with the intention of elasticsearch auto detecting the type as float type. Unfortunately, filebeat when parsing the json document converts 0.0 to 0 which makes elasticsearch interpret it as long.

Heres' how the original document looks like in my json log file:
{"@t":"2019-12-15T20:32:15.3321532+11:00","DemoJsonTelemetry":{"demoFloatZero":0.0,"demoDoubleZero":0.0},"type":"DemoJsonTelemetry"}

However, in filebeat DEBUG log I see the 0.0 changed to 0

2019-12-15T20:32:22.027+1100	DEBUG	[publish]	pipeline/processor.go:309	Publish event: {
  "@timestamp": "2019-12-15T09:32:22.026Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.8.1",
    "pipeline": "zl-events"
  },
  "DemoJsonTelemetry": {
    "demoFloatZero": 0,
    "demoDoubleZero": 0
  },
  "type": "DemoJsonTelemetry",
  "log": {
    "file": {}
  },
  "beat": {
    "timezone": "+11:00",
    "hostname": "DESKTOP-26QI0CF",
    "version": "6.8.1"
  },
  "@t": "2019-12-15T20:32:15.3321532+11:00"
}

My filebeat config looks like this:

# structured json events
- type: log
  enabled: true
  fields_under_root: true
  pipeline: "zl-events"
  paths:
    - c:\Logging\*.json
  json.keys_under_root: true
  json.add_error_key: true  

I'd really like to stick to type auto detection and would prefer not sending floats as strings. Is there a workaround or some trick to overcome this problem?

Any help is very much appreciated. Thank You
Zee Nastalski

[jsoriano (Jaime Soriano)]

Hey @Zulu and welcome

There is little control on how Filebeat formats the different types of values in JSON documents, but there are some ways to explicitly define mappings for the fields in the indexes where you store this data.

If you know the fields that you want to map to specific types before-hands, you can use the setup.template.append_fields option, that allows to define additional custom mappings.

Something like this could work for your example:

setup.template.overwrite: true
setup.template.append_fields:
- name: DemoJsonTelemetry.demoFloatZero
  type: float
- name: DemoJsonTelemetry.demoDoubleZero
  type: float

You can also replace the default fields mappings included in Filebeat using the setup.template.fields, where you can include your own definitions.

[Zulu (Zee Nastalski)]

Thank you for your help @jsoriano. The approach you propose unfortunately will not work for me. Basically, the names of the keys in the json documents are defined by developers building the application. There will quite a few of them and I have no way of knowing the keys upfront. That's why I am relying on elasticsearch type auto detection system. This solution works great, except for zeroes in the float type.

My current thinking is, instead of 0.0, use smallest possible float64 value of 1.401298E-45. It is not ideal but might work for my use case.

Ideally, if I could define something in the ingest pipeline to detect any field of value 1.401298E-45 and snap it back to 0.0. However, it looks like all ingest pipeline processors are applied to the specific field names - which again, I don't know upfront. Unless I am wrong which I hope I am

I hope I am explaining the problem clearly. Any help is much appreciated!
z

[jsoriano (Jaime Soriano)]

Yeah, I see, maybe we should think on adding a dynamic mapping to make all numbers float by default, I think we could do it only in beats index mappings and it could work. But this would need to be further discussed and tested.

Something else you could try is to enforce a different behavior for the dynamic mapping if you have a common structure for all your metrics. For example if they are all like namespace.object.metric, you could try something like this:

setup.template.overwrite: true
setup.template.append_fields:
- name: namespace.*.*
  type: object
  object_type: float

If this works, something else you can try is to convert all your numbers to float, but I have never tested anything like that:

setup.template.overwrite: true
setup.template.append_fields:
- name: namespace.*
  type: object
  object_type: float
  object_type_mapping_type: long

[Zulu (Zee Nastalski)]

Thanks for your help @jsoriano. I'll try that and let you know I went after Christmas break.
Warm Regards
Zee