Libbeat common schema mapstriface

[dcroonen (Davy Croonen)]

Hi all

Am I missing something or is there a reason why there is no Float function (anymore) in mapstriface like in mapstrstr when converting events?

Kind regards
Davy

[exekias (Carlos Pérez Aradros)]

Hi @dcroonen,

I think it never had it, probably because no one needed it? I'm just guessing here, not really sure

Could you explain a bit about the use you plan with it?

[dcroonen (Davy Croonen)]

Hi @exekias,

Thanks for your reply and willingness to help. I'm pretty new when it comes to developing (metric)beats. Last week I started with a simple example where I pulled some data from a gearman daemon. For the mapping I used the mapstrstr library. In the example in the manual I saw the following mapping example in the schema definition:

            "test_float":  c.Float("testFloat"),

Now I moved on to a more complex scenario where I pull individual queue data from a rabbitmq daemon. When I do a manual pull with curl I get the following json string (notice the float type in 'rate'):

{"memory":21840,"messages":0,"messages_details":{"rate":0.0},"messages_ready":0}

After unmarshalling it in Go I get the following 'nested' map[string]interface{}:

map[memory:21840 messages:0 messages_details:map[rate:0] messages_ready:0]

(I still have figure out why the type of 'rate' swithed from float to int but that's not relavant for the time being, I think) To write a schema for this I use the mapstriface library like this:

var (
schema = s.Schema{
    "memory":                       c.Int("memory"),
    "messages":                     c.Int("messages"),
    "messages_details": c.Dict("messages_details", s.Schema{
        "rate": c.Int("rate"),
    }), 
    "messages_ready":               c.Int("messages_ready"),
})

Instead of using c.Int("rate"), I want to use c.Float("rate") to preserve the floating point character of the "rate" property. This is the reason for my initial question in this forum topic, any suggestions?

I tried to be as compact and clear as possible, if not, just ask. As I said I'm pretty new to this so maybe I'm missing some clues or doing things totally wrong....

Regards
Davy

[exekias (Carlos Pérez Aradros)]

You should be able to use Float, by usinng mapstrstr instead of mapstriface: https://github.com/elastic/beats/blob/master/libbeat/common/schema/mapstrstr/mapstrstr.go#L88

For instance, it is used here: https://github.com/elastic/beats/blob/master/metricbeat/module/redis/info/data.go#L22

Br

[dcroonen (Davy Croonen)]

Ok, I understand, the schema definition would be then as follows:

    var (
        schema = s.Schema{
            "memory":                        c.Int("memory"),
            "messages":                      c.Int("messages"),
            "messages_details": s.Object{
                "rate": c.Float(**???**),
            }, 
           "messages_ready":                 c.Int("messages_ready"),
    })

Only, how do I access the "messages_details" "rate" property in the schema definition? In other words how do I acces a property in a nested map object in a schema definition?

@dcroonen

map[memory:21840 messages:0 messages_details:map[rate:0] messages_ready:0]

Regards
Davy

[steffens (Steffen Siering)]

The purpose of s.Schema is to introduce some schema translation with predefined namespacing. As you don't do any processing, but copy all contents you can just do data = common.MapStr(parsedBody). Using Get/Set of MapStr you can modify the field to be float inplace.

mapstrstr is about mapping a flat key/value schema into a structured schema. This one expects the input value to be a string. As your source is json and the value is parsed as json.Number, the conversion via mapstrstr with Float converter might fail.
mapstriface is about re-mapping a nested/structured schema originating from json documents. Unfortunately s.Float is missing in mapstriface. But adding Float should be quite simple by copy-paste-modify the Int converter to generate a float value. Float missing from mapstriface is basically an oversight + no one ever needed it so far it seems. Btw. one can define custom converters the very same way Int is implemented.

Given the way schema did evolve over time (I haven't had a look at it since it was first introduced), I wonder if we can merge mapstrstr and mapstriface into one common package.

[dcroonen (Davy Croonen)]

Hi @steffens

Thanks a lot for your clarification. I already started suspecting something like this. I succeeded into to get a working example with mapStrStr but this involved to flatten my entire JSON document and to typecast every value to a String, not something very efficient in my opinion.

Because I don't need any remapping I will continue with your first suggestion.

Regards
Davy