Using logstash to denomalize data?

Elastic Stack Logstash
1

I have data coming in with field names like this:
"field_1" => "10"
"field_2" => "20"

and I'm trying to figure out how to convert it to something more like this:
"field" => [ {"num" => "1", "val" => "10"}, {"num" => "2", "val" => "20"}]

Issue is the source is sending in dozens of these fields, with an index that could theoretically go to 10k on each, so I'd end up with an index with potentially hundreds of thousands of fields.

I got tasked with this by my boss, but I really know next to nothing about logstash. For this, I'm not even sure what I'm looking for - just a function name would be useful so I could google it. :S

2

Assuming that they are all top-level fields and the names really do start with "field_" you could try something like

ruby {
    code => '
        fields = []
        event.to_hash.each { |k, v|
            if k =~ /^field_\d+/
                newK = k.sub(/^field_/, "")
                fields << { "num" => newK, "val" => v }
                event.remove(k)
            end
        if fields != []
            event.set("fields", fields)
        end
    '
}
3

Thanks! I'll give this a shot.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.