Logic Error after ingesting number

Elastic Stack Logstash
1

I have an xlsx file that I convert in CSV file using ssconvert tool. In the new generated file, there's one field in floating point type. But after ingestion, the values taken from the file is no more in float.

For exemple: This value 99,8923 become "99,8923" in the generated file but other values in the same field are correctly parsed, like 54 is 54.

I suspect that the probleme may come from " added around the number. After ingesting this number with Logstash, it becomes 998923 even if I specify in logstash that it is a float and I can see clearly that the field is typed with float in Kibana.

Here is the version I'm currently using:
Elasticsearch : 8.3
Kibana : 8.3
Operating Sys : Red Hat Enterprise Linux 8.5 (Ootpa)

Thanks for your help.

2

Hello There!

How about changing "," to "." as a decimal separator in ssconvert?

3

Hello @cheshirecat ,

I didn't find a way for changing this with ssconvert. Do you know how to do it with it? or do you know a way to change "," to "."?

4

Perhaps look at mutate convert here

Also make sure you create a mapping ahead of time for that type

  • float:

    • integers are converted to floats
    • strings are parsed; comma-separators and dot-decimals are supported (e.g., "1,000.5" produces a float with value of one thousand and one half)
    • boolean true and boolean false are converted to 1.0 and 0.0 respectively

  • float_eu:

    • same as float, except string values support dot-separators and comma-decimals (e.g., "1.000,5" produces a float with value of one thousand and one half)
5

Hello @stephenb ,

I already used convert, and there is others that I convert perfectly in floating point type with logstash.

The field with the issue is in type float, but values are not coherent...

I'm actually working on a way to treat this before sending to logstash...

6

OK if you showed us a full source line... pretty sure we could fix it in logstash... but at the source is even better.

7

I just ran this pipeline

input { stdin {  ecs_compatibility => "v8"} }

filter {
   mutate {
        convert => {
          "message" => "float_eu"
        }
    }
}

output { stdout {} }

stdin

> 99,8923
{
    "@timestamp" => 2022-09-14T21:12:13.177705Z,
          "host" => {
        "hostname" => "hyperion"
    },
       "message" => 99.8923,
      "@version" => "1",
         "event" => {
        "original" => "99,8923"
    }
}
> 1.123,45     
{
    "@timestamp" => 2022-09-14T21:14:21.722249Z,
          "host" => {
        "hostname" => "hyperion"
    },
       "message" => 1123.45,
      "@version" => "1",
         "event" => {
        "original" => "1.123,45"
    }
}

Worked Fine Note it is a normal float on the output not text/keword

8

Hello @stephenb ,

I really appreciate your help and it's working. Sometimes, I feel like "Why I didn't see that" :roll_eyes:... That's why the community is around :grinning:, so your help was a light...

Thanks so much...

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.