Simple setup question

Elastic Stack Elasticsearch
1

I have ELK setup and from I think working fine, trying to just simply
import syslog data into it - can anyone help me with this? I am running
Ubuntu 14.04 with the default syslog service built in and have also tried
syslog-ng. I have verified data is getting to syslog by checking the
/var/log/syslog. I have also verified elasticsearch/kibana is working by
testing it with shakspeare JSON
(http://www.elasticsearch.org/guide/en/kibana/current/using-kibana-for-the-first-time.html)

here is snippet of logstash log

itadmin@st-log:/opt/logstash$ tail -50 /var/log/logstash/logstash.log
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.InvalidClassException: failed to read class descriptor
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1603)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167)
... 23 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.ElasticsearchException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at
org.jruby.util.JRubyClassLoader.findClass(JRubyClassLoader.java:128)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
... 34 more
log4j, [2014-07-09T15:23:00.233] WARN: org.elasticsearch.transport.netty:
[Firelord] Message not fully read (response) for [78] handler
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$4@25c8d6ed,
error [true], resetting

here is logstash config

input {
syslog {
}
}
filter {
}
output {
elasticsearch {
host => "172.16.40.28"
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5428ec7b-84a5-4f63-8e7d-ec957ec2ba9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Which versions of logstash and elasticsearch are you using? There was some
class renaming back in 1.0 and it caused a few ClassNotFoundException:
org.elasticsearch.ElasticsearchException for several plugins.

Other than that, I am not sure.

Cheers,

Ivan

On Wed, Jul 9, 2014 at 12:24 PM, Kevin M mcgkev29@gmail.com wrote:

I have ELK setup and from I think working fine, trying to just simply
import syslog data into it - can anyone help me with this? I am running
Ubuntu 14.04 with the default syslog service built in and have also tried
syslog-ng. I have verified data is getting to syslog by checking the
/var/log/syslog. I have also verified elasticsearch/kibana is working by
testing it with shakspeare JSON (
Elasticsearch Platform — Find real-time answers at scale | Elastic
)

here is snippet of logstash log

itadmin@st-log:/opt/logstash$ tail -50 /var/log/logstash/logstash.log
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.InvalidClassException: failed to read class descriptor
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1603)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at
org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167)
... 23 more
Caused by: java.lang.ClassNotFoundException:
org.elasticsearch.ElasticsearchException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at
org.jruby.util.JRubyClassLoader.findClass(JRubyClassLoader.java:128)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93)
at
org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
... 34 more
log4j, [2014-07-09T15:23:00.233] WARN: org.elasticsearch.transport.netty:
[Firelord] Message not fully read (response) for [78] handler
org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$4@25c8d6ed,
error [true], resetting

here is logstash config

input {
syslog {
}
}
filter {
}
output {
elasticsearch {
host => "172.16.40.28"
}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5428ec7b-84a5-4f63-8e7d-ec957ec2ba9d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5428ec7b-84a5-4f63-8e7d-ec957ec2ba9d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAjVhZjqdhVwMpkG%2BBg51SY_MhjKjCtCxJaCGC9jfxeVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

I don't know about this particular problem, but you could try the UDP input
instead:

input {
udp {
port => your_portnumber_here (514 is the syslog default)
type => your_typename_here
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f29db21c-3c49-4f1f-b8de-f78ff79b9507%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.