New collectd types do not show up in logstash/ES

Elastic Stack Logstash
1

I've created a collectd exec plugin with new collectd data types to send to logstash+ES, I get the ES documents but amputated without the new data type fields and wonder why. Hints appreciated, TIA.

new collectd type:

$ grep haproxy /usr/share/collectd/types.db
haproxy_backend quecur:COUNTER:0:4294967295,sescur:COUNTER:0:4294967295,sestot:COUNTER:0:4294967295

I expect collectd picks up it's new type correct as I've logged it's output to csv as well and find new multi value type like these:

epoch,quecur,sescur,sestot
1456412134,0,0,17
1456412144,0,0,17

created from exec plugin by:

PUTVAL /haproxy/haproxy_backend-be-web 1456409997:0:0:107

but I see logstash only receiving this from collectd:

{
"host" => "hapA",
"@timestamp" => "2016-02-25T14:39:32.000Z",
"plugin" => "haproxy",
"collectd_type" => "haproxy_backend",
"type_instance" => "be-web",
"@metadata" => {
"input-collectd" => "true",
"output-esmx" => "true",
"output-stdout" => "false",
"output-debug" => "true",
"esindex" => "collectd"
}
}

Do I need to patch collectd codec somehow to accept new types?

input {

udp {
port => 25826 # 25826 matches port specified in collectd.conf
buffer_size => 8192 # 8192 as we support jumboframes upto 9000
codec => collectd { } # specific Collectd codec to invoke
add_field => {'[@metadata][input-collectd]' => 'true' '[@metadata][output-es
mx]' => 'true' '[@metadata][output-stdout]' => 'false' }
}

2

seems it might be fine to sync logstash's type db with end point collectors new data type into here:

/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-collectd-2.0.2/vendor/types.db

3

You are correct! You would need to add or update custom types in the types.db file. I'd recommend copying that one out to a known location, and edit that version. You can use the typesdb directive in the codec settings to specify the location.

4

Thanks, do I need all existing types in an element path of typesdb or just add my few custom types to a known path and just add this path to typesdb, ie. will defining typesdb loose default types or will they still be read/known?

5

Seems I need supply all default types as well

6

That is why I suggested copying the source, and editing that :slightly_smiling:

7

Yeap or add an extra element more to typesdb array pointing to default typesdb path... only it' in a versioned path, so an update of this plugin might remove such a path. But yes I copying all default type patterns to my own path and added custom types.

Thanks for point out!

8

Guess a RFE for collectd input filter plugin could be to always read in std default typesdb first regardless of an optional typesdb array given thus one only needs to specific custom types and add this path only... just a $.02

9

This is problematic in the case that you have overridden one of those values for some reason. This is why it's not done. What might work is to add an array element of default or something like that first, and have the codec interpret that to use the included one first. You could always submit an issue or pull request with that feature at GitHub - logstash-plugins/logstash-codec-collectd