Ganglia Packets as Input in Logstash


(Christos Markou) #1

Hello guys,

i would like to take ganglia packets in Logstash as input. From Logstash documention (http://www.logstash.net/docs/1.4.2/inputs/ganglia) i make that i have to configure my Logstash Server to listen on a port (8649) and an address to listen on (my Logstash Server IP?). Right?

But when it comes to Ganglia's side, how i can configure Ganglia to send the packets to my Logstash Server?

I know that this question is not mainly about Logstash itself, but if someone has done sth similar or have an idea about this please share.! :wink:

Thanx!


Ganglia into logstash
(Joshua Rich) #2

Probably the easiest solution would be to adjust the gmond.conf file on each node to be monitored and set up a new udp_send_channel configuration directive that points at your Logstash server:

Something like the following in your gmond.conf should work:

 udp_send_channel {
   host = my.logstash.server
   port = 8649
   ttl = 1
 }

(Christos Markou) #3

Thanx for the quick reply!

I will check this and leave feedback soon.


(Christos Markou) #4

Hi again,

we have configured gmond.conf without trouble as mentionde above, but in my LogstashServer side nothing seems to work. When i start logstash service everything is fine but it seems that logstash doesn't listen on port 8649.
My configuration file is:

input {

tcp {
type => "apache"
port => 3333
}

ganglia {
port => 8649
type => "ganglia"
}

lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

And netstat gives:
ubuntu@logstashserver:~$ netstat -nat | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:5601 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 :::3333 :::* LISTEN
tcp6 0 0 :::5000 :::* LISTEN

->Port 8649 doesn't appear...

logstash.log doesn't give any error or warning.
Any ideas???


(Christos Markou) #5

Hello,

i am refresing the topic hoping for an answer. So if anyone has done something similar (connecting ganglia and logstash) and has the experience plz share because i am stack.


(Joshua Rich) #6

Hey @ChrisMark,

Ganglia uses UDP by default and the LS plugin follows along. So to check it's listening, change the -t in your netstat command to -u, i.e., try netstat -nau to see Logstash listening on UDP port 8649.

Hope this helps!


(Christos Markou) #7

Hello,

i followed the suggestion and i see:

ubuntu@logstashganglia:~$ netstat -nau
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 0.0.0.0:62582 0.0.0.0:*
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp6 0 0 :::8649 :::*
udp6 0 0 :::54328 :::*
udp6 0 0 :::10124 :::*

So i make that connection is not ESTABLISED ???
Thnx!


(Joshua Rich) #8

You probably won't generally see a constant ESTABLISHED connection in netstat for Ganglia. UDP is connectionless, so clients will just be sending a data stream to the Logstash server when needed. They won't leave the connection open like you might see with some programs using TCP.


(Christos Markou) #9

Ok got it, but then how i will figure out if packets are coming?
The fact that i have not configured a filter for gagnlia's type logs may affect the whole thing?


(Joshua Rich) #10

Even without filters in place, Logstash is still processing any Ganglia events that are sent. What outputs do you have configured? Try adding a simple stdout output like the following and then watch the /var/log/logstash/logstash.stdout file for any Ganglia events:

output {
  if [type] == "ganglia" {
    stdout {
      codec => "rubydebug"
    }
  }
}

(Christos Markou) #11

Τhat's a good idea, but i think i have located the problem. As i posted before my Logstash listens ubuntu@logstashganglia:~$ netstat -nau
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp6 0 0 :::8649 :::*

This means that 8649 udp is on my ipV6, rigth? So i have to configure Ganglia's side to send the packets on my ipv6 not to ipv4? Right?


(Joshua Rich) #12

If you aren't using IPv6, it will probably be best to configure Logstash to listen on IPv4 only. You can do this by editing either /etc/default/logstash (on Debian/Ubuntu) or /etc/sysconfig/logstash (on RedHat/CentOS) and adding -Djava.net.preferIPv4Stack=true to the LS_JAVA_OPTS setting. After making this change, restart Logstash to have it listen just on an IPv4 address.


(Saggarsunil) #13

Hi Chris:

I am also trying to configure ganglia as input to logstash. As of now, i am running logstash and ganglia on the same machine.

Ganglia command line:
bin/logstash -e 'input { ganglia { host=>"192.168.1.7" port=> 8686 } } output { stdout {} }'

As you can see, i am using a different port for logstash and it starts fine BUT i don't see any ganglia messages ..

All i see is messages like this:
2015-08-23T14:11:41.571Z 192.168.1.7 %{message}
2015-08-23T14:11:41.572Z 192.168.1.7 %{message}
2015-08-23T14:11:41.572Z 192.168.1.7 %{message}
2015-08-23T14:11:41.573Z 192.168.1.7 %{message}
2015-08-23T14:12:01.446Z 192.168.1.7 %{message}

On the ganglia conf file, i am using 2 udp send channels.
udp_send_channel {
#bind_hostname = yes # Highly recommended, soon to be default.
# This option tells gmond to use a source address
# that resolves to the machine's hostname. Without
# this, the metrics may appear to come from any
# interface and the DNS names associated with
# those IPs will be used to create the RRDs.
mcast_join = 239.2.11.71
port = 8649
ttl = 1
}

udp_send_channel {
host = 192.168.1.7
port = 8686
}

I am trying to debug further but any help at this point will really help in expedite the problem resolution.

Thanks
Sunil


(Magnus Bäck) #14

@saggarsunil – Since your question is unrelated to the original question (you're receiving messages just fine) please start a new topic.

This is clearly a case of poor documentation and I've filed github.com/logstash-plugins/logstash-input-ganglia issue #8 on your behalf.


#15

Hi magnus

Following up form the earlier message of saggarsunil
I also am in same situation, I see %{message} on the stdout

However when i use the below option..'input {ganglia {port=>8649 type=>"ganglia" }} output { stdout { codec => rubydebug } file { path => "/tmp/gmond-log.txt"}}'

i see continuous steram of following data (with some value changiing)

@version" => "1",
"@timestamp" => "2015-08-28T11:00:55.541Z",
"log_host" => "E8a7--13",
"dmax" => 0,
"tmax" => 180,
"slope" => "both",
"type" => "float",
"units" => "KB",
"host" => "10.40.94.157"

It seems not all data is comming from gmond but only static information
Howevedr when i run telnet localhost 8649, I see all of the data (below is the snippet for it)

METRIC NAME="swap_free" VAL="4194296" TYPE="float" UNITS="KB" TN="26" TMAX="180" DMAX="0" SLOPE="both
Why logstash is not outputting the value of the all the caputured metrics ( as seen in telnet command)

How do i get all the metrics data printed through logstash so that it can be caputerd by elasticsearch

I am stuck and dont know the way forward !!

P.S I was facing this issue where logstash was not outputing anything from ganglia
It was only after i ran iptables --flush, i was able to get above shown output through logstash


(Magnus Bäck) #16

The issue below seems relevant to your problem.


(Josh Star Long) #18

Hi @magnusbaeck,

I was able to send some metrics of ganglia to logstash. Here is my input configuration
input {
lumberjack {
port => 5043
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
udp {
port => 8649
codec => json_lines
}
}
This configuration is working and I able to visaulized my ganglia metric the awesome kibana dashbroad. But, logstash is unable to understand the message of the metric. That is the message look like
"message" => "\u0000\u0000\u0000\x86\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\fload_fifteen\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004%.2f=L\xCC\xCD\u0000\u0000\u0000\x84\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\theartbeat\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002%u\u0000\u0000V\xAC\xDFF\u0000\u0000\u0000\x84\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\theartbeat\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0002%u\u0000\u0000V\xAC\xDFF\u0000\u0000\u0000\x86\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000\bmem_free\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004%.0fH\xEBK\u0000\u0000\u0000\u0000\x86\u0000\u0000\u0000\u0010ip-172-31-37-235\u0000\u0000\u0000"
I have attach a screenshot.
Now my question is how can I configure logstash to understand the messages sent by ganglia.
Thanks for the concern.


(Magnus Bäck) #19

@jstar—please start a new thread for your question.


(system) #20