Ganglia Packets as Input in Logstash

Elastic Stack Logstash
1

Hello guys,

i would like to take ganglia packets in Logstash as input. From Logstash documention (http://www.logstash.net/docs/1.4.2/inputs/ganglia) i make that i have to configure my Logstash Server to listen on a port (8649) and an address to listen on (my Logstash Server IP?). Right?

But when it comes to Ganglia's side, how i can configure Ganglia to send the packets to my Logstash Server?

I know that this question is not mainly about Logstash itself, but if someone has done sth similar or have an idea about this please share.! :wink:

Thanx!

2

Probably the easiest solution would be to adjust the gmond.conf file on each node to be monitored and set up a new udp_send_channel configuration directive that points at your Logstash server:

Something like the following in your gmond.conf should work:

 udp_send_channel {
   host = my.logstash.server
   port = 8649
   ttl = 1
 }
3

Thanx for the quick reply!

I will check this and leave feedback soon.

4

Hi again,

we have configured gmond.conf without trouble as mentionde above, but in my LogstashServer side nothing seems to work. When i start logstash service everything is fine but it seems that logstash doesn't listen on port 8649.
My configuration file is:

input {

tcp {
type => "apache"
port => 3333
}

ganglia {
port => 8649
type => "ganglia"
}

lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

And netstat gives:
ubuntu@logstashserver:~$ netstat -nat | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 :::3333 :::* LISTEN
tcp6 0 0 :::5000 :::* LISTEN

->Port 8649 doesn't appear...

logstash.log doesn't give any error or warning.
Any ideas???

5

Hello,

i am refresing the topic hoping for an answer. So if anyone has done something similar (connecting ganglia and logstash) and has the experience plz share because i am stack.

6

Hey @ChrisMark,

Ganglia uses UDP by default and the LS plugin follows along. So to check it's listening, change the -t in your netstat command to -u, i.e., try netstat -nau to see Logstash listening on UDP port 8649.

Hope this helps!

7

Hello,

i followed the suggestion and i see:

ubuntu@logstashganglia:~$ netstat -nau
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 0.0.0.0:62582 0.0.0.0:*
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp6 0 0 :::8649 :::*
udp6 0 0 :::54328 :::*
udp6 0 0 :::10124 :::*

So i make that connection is not ESTABLISED ???
Thnx!

8

You probably won't generally see a constant ESTABLISHED connection in netstat for Ganglia. UDP is connectionless, so clients will just be sending a data stream to the Logstash server when needed. They won't leave the connection open like you might see with some programs using TCP.

9

Ok got it, but then how i will figure out if packets are coming?
The fact that i have not configured a filter for gagnlia's type logs may affect the whole thing?

10

Even without filters in place, Logstash is still processing any Ganglia events that are sent. What outputs do you have configured? Try adding a simple stdout output like the following and then watch the /var/log/logstash/logstash.stdout file for any Ganglia events:

output {
  if [type] == "ganglia" {
    stdout {
      codec => "rubydebug"
    }
  }
}
11

Τhat's a good idea, but i think i have located the problem. As i posted before my Logstash listens ubuntu@logstashganglia:~$ netstat -nau
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp6 0 0 :::8649 :::*

This means that 8649 udp is on my ipV6, rigth? So i have to configure Ganglia's side to send the packets on my ipv6 not to ipv4? Right?

12

If you aren't using IPv6, it will probably be best to configure Logstash to listen on IPv4 only. You can do this by editing either /etc/default/logstash (on Debian/Ubuntu) or /etc/sysconfig/logstash (on RedHat/CentOS) and adding -Djava.net.preferIPv4Stack=true to the LS_JAVA_OPTS setting. After making this change, restart Logstash to have it listen just on an IPv4 address.

13

Hi Chris:

I am also trying to configure ganglia as input to logstash. As of now, i am running logstash and ganglia on the same machine.

Ganglia command line:
bin/logstash -e 'input { ganglia { host=>"192.168.1.7" port=> 8686 } } output { stdout {} }'

As you can see, i am using a different port for logstash and it starts fine BUT i don't see any ganglia messages ..

All i see is messages like this:
2015-08-23T14:11:41.571Z 192.168.1.7 %{message}
2015-08-23T14:11:41.572Z 192.168.1.7 %{message}
2015-08-23T14:11:41.572Z 192.168.1.7 %{message}
2015-08-23T14:11:41.573Z 192.168.1.7 %{message}
2015-08-23T14:12:01.446Z 192.168.1.7 %{message}

On the ganglia conf file, i am using 2 udp send channels.
udp_send_channel {
#bind_hostname = yes # Highly recommended, soon to be default.
# This option tells gmond to use a source address
# that resolves to the machine's hostname. Without
# this, the metrics may appear to come from any
# interface and the DNS names associated with
# those IPs will be used to create the RRDs.
mcast_join = 239.2.11.71
port = 8649
ttl = 1
}

udp_send_channel {
host = 192.168.1.7
port = 8686
}

I am trying to debug further but any help at this point will really help in expedite the problem resolution.

Thanks
Sunil

14

@saggarsunil – Since your question is unrelated to the original question (you're receiving messages just fine) please start a new topic.

This is clearly a case of poor documentation and I've filed github.com/logstash-plugins/logstash-input-ganglia issue #8 on your behalf.

15

Hi magnus

Following up form the earlier message of saggarsunil
I also am in same situation, I see %{message} on the stdout

However when i use the below option..'input {ganglia {port=>8649 type=>"ganglia" }} output { stdout { codec => rubydebug } file { path => "/tmp/gmond-log.txt"}}'

i see continuous steram of following data (with some value changiing)

@version" => "1",
"@timestamp" => "2015-08-28T11:00:55.541Z",
"log_host" => "E8a7--13",
"dmax" => 0,
"tmax" => 180,
"slope" => "both",
"type" => "float",
"units" => "KB",
"host" => "10.40.94.157"

It seems not all data is comming from gmond but only static information
Howevedr when i run telnet localhost 8649, I see all of the data (below is the snippet for it)

METRIC NAME="swap_free" VAL="4194296" TYPE="float" UNITS="KB" TN="26" TMAX="180" DMAX="0" SLOPE="both
Why logstash is not outputting the value of the all the caputured metrics ( as seen in telnet command)

How do i get all the metrics data printed through logstash so that it can be caputerd by elasticsearch

I am stuck and dont know the way forward !!

P.S I was facing this issue where logstash was not outputing anything from ganglia
It was only after i ran iptables --flush, i was able to get above shown output through logstash

16

The issue below seems relevant to your problem.

1 Like
18

Hi @magnusbaeck,

I was able to send some metrics of ganglia to logstash. Here is my input configuration
input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
udp {
port => 8649
codec => json_lines
}
}
This configuration is working and I able to visaulized my ganglia metric the awesome kibana dashbroad. But, logstash is unable to understand the message of the metric. That is the message look like
"message" => "\u0000\u0000\u0000\x86\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\fload_fifteen\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004%.2f=L\xCC\xCD\u0000\u0000\u0000\x84\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\theartbeat\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002%u\u0000\u0000V\xAC\xDFF\u0000\u0000\u0000\x84\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\theartbeat\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002%u\u0000\u0000V\xAC\xDFF\u0000\u0000\u0000\x86\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\bmem_free\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004%.0fH\xEBK\u0000\u0000\u0000\u0000\x86\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000"
I have attach a screenshot.
Now my question is how can I configure logstash to understand the messages sent by ganglia.
Thanks for the concern.

Capture.PNG1122×534 39.5 KB

19

@jstar—please start a new thread for your question.