Hi,
this is my first post and I am new user of ELK. I have succesfully configured ELK on CentOS and working fine and now I am sending data so that i can create dashboards. However I am stuck at one point and though i have read many post and the online logstash documentation, i dont seem to get past.
So basically i have a log file and I am shipping that logfile to logstash which is working fine via filebeat for windows.
Now I have the following logstash configuration, which will extract fields from the unstructured data which looks like this
{user@domain.com} Active 11.11.11.11 12.12.12.12 Eap Tls 47975 9/12/2018 5:10:15 AM 8046963 10435405 Ikev2 216
I have created a conf file named filebeat1.conf and placed in /etc/logstash/conf.d and it looks like following:
input {
beats {
port => 5044
type => "log"
}
}
filter {
if [type] == "log" {
grok {
match => { "message" => "{%{GREEDYDATA:username}} %{SPACE} %{DATA:status} %{SPACE} %{IPV4:clientip} %{SPACE} %{IPV4:ispip} %{SPACE} %{DATA:protocol} %{SPACE}
%{INT:duration} %{SPACE} %{DATESTAMP:connectiontime} %{DATA} %{SPACE} %{INT:bytesin} %{SPACE} %{INT:bytesout} %{SPACE} %{HOSTNAME:security} %{SPACE} %{INT:ba
ndwidth} %{SPACE}" }
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
}
However in Kibana i a getting _grokparsefailure. I have tested the pattern in https://grokdebug.herokuapp.com/ and it looks fine there but when i copy and paste the pattern in conf file then i get grok parse error and the logstash log file shows
[2018-09-12T02:15:10,415][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 192.168.1.30:49278] Handling exception: Connection reset by p
eer
[2018-09-12T02:15:10,417][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It
usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_102]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_102]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_102]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_102]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_102]
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1108) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:345) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:126) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]