# \`collectd\` codec intermittently dropping some telemetry information **URL:** \ **Category:** Logstash\ **Created:** [May 6, 2020, 7:47am UTC](https://discuss.elastic.co/t/collectd-codec-intermittently-dropping-some-telemetry-information/231286 "2020-05-06T07:47:39Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![sjlongland](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjlongland/32/11468_2.png) [@sjlongland](https://discuss.elastic.co/u/sjlongland)\ **Post date:** [May 6, 2020, 7:47am UTC](https://discuss.elastic.co/t/collectd-codec-intermittently-dropping-some-telemetry-information/231286/1 "2020-05-06T07:47:39Z")
Hi all… We've been using ELK stack via a Docker container ([`sebp/elk`](https://hub.docker.com/r/sebp/elk)) for some time along-side ElastAlert to monitor a number of virtual hosts in our care via `collectd`, and until very recently it's been working well. We were running ElasticSearch 2.3.4/Logstash 2.3.4/Kibana 4.5.3… and a week two back, I did an update of that container through a number of revisions, migrating everything to ElasticSearch 7.6.1. The (virtual) host is running Ubuntu 16.04, we just have the one VM running ElasticSearch. `collectd` was not changed, and things seemed to be working at the time. The latest release for that Ubuntu version (`5.5.1-1build2`) is installed. We have a port exposed on the loopback interface from the ELK stack docker container for `collectd` to talk to. We have also mounted the directory where `collectd` keeps its `types.db` so it can be accessed by `logstash`. ```auto # docker-compose.yml services: # ELK Stack production configuration elk: restart: always volumes: # Mount logstash configuration from the host. - /etc/logstash:/etc/logstash - /etc/elasticsearch:/etc/elasticsearch - /usr/share/collectd:/usr/share/collectd - /var/backup/elk:/var/backup environment: # Heap size should be under 50% of available memory, but should be greater than 1GB # https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html # https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html - ES_HEAP_SIZE=1g ports: - "127.0.0.1:514:5514/udp" # Syslog loopback socket - "127.0.0.1:25827:25826/udp" # collectd loopback socket # etc … other port mappings here ``` ```auto # collectd.conf # client setup: Server "127.0.0.1" "25826" # ELK stack container # server setup: SecurityLevel Encrypt AuthFile "/etc/collectd/passwd" Interface "eth1" SecurityLevel Encrypt AuthFile "/etc/collectd/passwd" Interface "tun0" # # proxy setup (client and server as above): Forward true ``` ```auto # logstash/conf.d/04-collectd.conf input { udp { host => "0.0.0.0" port => 25826 buffer_size => 1452 type => "collectd" codec => collectd { typesdb => "/usr/share/collectd/types.db" } } } ``` We're noting that whilst _most_ of our instances' `collectd` stats are making it through to ELK stack, the host _actually_ running the `collectd` proxy and ELK stack itself is not showing up. When using `tcpdump` to capture the traffic on `lo`, _all_ traffic is seen heading into the ELK-stack instance, so presumably is being received by `logstash`. A frame from one of the nodes which isn't appearing looks like this when analysed by Wireshark: ```auto No. Time Source Destination Protocol Length Info 27 28.899751 127.0.0.1 127.0.0.1 collectd 1373 Host=elkstack.example.com, 27 values for 12 plugins, 0 messages Frame 27: 1373 bytes on wire (10984 bits), 1373 bytes captured (10984 bits) …snip… collectd network data collectd HOST segment: "elkstack.example.com" Type: HOST (0x0000) Length: 32 Host name: elkstack.example.com collectd TIME_HR segment: May 6, 2020 10:41:03.590266326 EST Type: TIME_HR (0x0008) Length: 12 Timestamp: May 6, 2020 10:41:03.590266326 EST collectd INTERVAL_HR segment: 1 minute Type: INTERVAL_HR (0x0009) Length: 12 Interval: 60.000000000 seconds collectd PLUGIN segment: "df" Type: PLUGIN (0x0002) Length: 7 Plugin: df collectd PLUGIN_INSTANCE segment: "root" Type: PLUGIN_INSTANCE (0x0003) Length: 9 Plugin instance: root collectd TYPE segment: "df_complex" Type: TYPE (0x0004) Length: 15 Type: df_complex collectd TYPE_INSTANCE segment: "used" Type: TYPE_INSTANCE (0x0005) Length: 9 Type instance: used collectd VALUES segment: 1 value Type: VALUES (0x0006) Length: 15 Value count: 1 1 value Gauge: 2.94918e+11 Value type: GAUGE (0x01) Gauge value: 294917779456 [Assembled metric] Host name: elkstack.example.com Plugin: df Plugin instance: root Type: df_complex Type instance: used Timestamp: May 6, 2020 10:41:03.590266326 EST Interval: 60.000000000 seconds collectd TIME_HR segment: May 6, 2020 10:41:03.589932766 EST Type: TIME_HR (0x0008) Length: 12 Timestamp: May 6, 2020 10:41:03.589932766 EST collectd PLUGIN segment: "memory" Type: PLUGIN (0x0002) Length: 11 Plugin: memory collectd PLUGIN_INSTANCE segment: "" Type: PLUGIN_INSTANCE (0x0003) Length: 5 Plugin instance: collectd TYPE segment: "percent" Type: TYPE (0x0004) Length: 12 Type: percent collectd VALUES segment: 1 value Type: VALUES (0x0006) Length: 15 Value count: 1 1 value Gauge: 38.871 Value type: GAUGE (0x01) Gauge value: 38.8709877262965 [Assembled metric] Host name: elkstack.example.com Plugin: memory Plugin instance: Type: percent Type instance: used Timestamp: May 6, 2020 10:41:03.589932766 EST Interval: 60.000000000 seconds collectd TIME_HR segment: May 6, 2020 10:41:03.590267718 EST Type: TIME_HR (0x0008) Length: 12 Timestamp: May 6, 2020 10:41:03.590267718 EST collectd PLUGIN segment: "df" Type: PLUGIN (0x0002) Length: 7 Plugin: df collectd PLUGIN_INSTANCE segment: "root" Type: PLUGIN_INSTANCE (0x0003) Length: 9 Plugin instance: root collectd TYPE segment: "percent_bytes" Type: TYPE (0x0004) Length: 18 Type: percent_bytes collectd TYPE_INSTANCE segment: "reserved" Type: TYPE_INSTANCE (0x0005) Length: 13 Type instance: reserved collectd VALUES segment: 1 value Type: VALUES (0x0006) Length: 15 Value count: 1 1 value Gauge: 5.08306 Value type: GAUGE (0x01) Gauge value: 5.08306264877319 [Assembled metric] Host name: elkstack.example.com Plugin: df Plugin instance: root Type: percent_bytes Type instance: reserved Timestamp: May 6, 2020 10:41:03.590267718 EST Interval: 60.000000000 seconds collectd TIME_HR segment: May 6, 2020 10:41:03.590267107 EST Type: TIME_HR (0x0008) Length: 12 Timestamp: May 6, 2020 10:41:03.590267107 EST collectd TYPE_INSTANCE segment: "free" Type: TYPE_INSTANCE (0x0005) Length: 9 Type instance: free collectd VALUES segment: 1 value Type: VALUES (0x0006) Length: 15 Value count: 1 1 value Gauge: 7.73987 Value type: GAUGE (0x01) Gauge value: 7.73987340927124 [Assembled metric] Host name: elkstack.example.com Plugin: df Plugin instance: root Type: percent_bytes Type instance: free Timestamp: May 6, 2020 10:41:03.590267107 EST Interval: 60.000000000 seconds collectd TIME_HR segment: May 6, 2020 10:41:03.590268296 EST Type: TIME_HR (0x0008) Length: 12 Timestamp: May 6, 2020 10:41:03.590268296 EST collectd TYPE_INSTANCE segment: "used" Type: TYPE_INSTANCE (0x0005) Length: 9 Type instance: used collectd VALUES segment: 1 value Type: VALUES (0x0006) Length: 15 Value count: 1 1 value Gauge: 87.1771 Value type: GAUGE (0x01) Gauge value: 87.1770553588867 [Assembled metric] Host name: elkstack.example.com Plugin: df Plugin instance: root Type: percent_bytes Type instance: used Timestamp: May 6, 2020 10:41:03.590268296 EST Interval: 60.000000000 seconds collectd TIME_HR segment: May 6, 2020 10:41:03.589932766 EST Type: TIME_HR (0x0008) Length: 12 Timestamp: May 6, 2020 10:41:03.589932766 EST collectd PLUGIN segment: "memory" Type: PLUGIN (0x0002) Length: 11 Plugin: memory collectd PLUGIN_INSTANCE segment: "" Type: PLUGIN_INSTANCE (0x0003) Length: 5 Plugin instance: collectd TYPE segment: "percent" Type: TYPE (0x0004) Length: 12 Type: percent collectd TYPE_INSTANCE segment: "slab_unrecl" Type: TYPE_INSTANCE (0x0005) Length: 16 Type instance: slab_unrecl collectd VALUES segment: 1 value Type: VALUES (0x0006) Length: 15 Value count: 1 1 value Gauge: 0.379304 Value type: GAUGE (0x01) Gauge value: 0.379303957034613 [Assembled metric] Host name: elkstack.example.com Plugin: memory Plugin instance: Type: percent Type instance: slab_unrecl Timestamp: May 6, 2020 10:41:03.589932766 EST Interval: 60.000000000 seconds collectd HOST segment: "customer1.example.com" Type: HOST (0x0000) Length: 25 Host name: customer1.example.com collectd TIME_HR segment: May 6, 2020 10:40:05.514224234 EST … snip more data because forum software couldn't take it … ``` The data for `customer1.example.com` _does_ show up, but not `elkstack.example.com`. I have a suspicion this is [issue #13](https://github.com/logstash-plugins/logstash-codec-collectd/issues/13) rearing its ugly head, but I'm not sure. Is there way I can get `logstash` to report what it sees from `collectd`? ---
**Author:** ![sjlongland](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjlongland/32/11468_2.png) [@sjlongland](https://discuss.elastic.co/u/sjlongland)\ **Post date:** [June 2, 2020, 6:30am UTC](https://discuss.elastic.co/t/collectd-codec-intermittently-dropping-some-telemetry-information/231286/2 "2020-06-02T06:30:59Z")
No clues? If I've omitted something, please tell me. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 30, 2020, 6:31am UTC](https://discuss.elastic.co/t/collectd-codec-intermittently-dropping-some-telemetry-information/231286/3 "2020-06-30T06:31:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.