Exception on configuring Logstash

[Anca_Turc (Anca Ioana)]

Hi ,
I am trying to filter the traffic for search queries and I have followed this tutorial where is specified the logstash config file sniff_search.conf.

Input { 
    beats { 
        port => 5044
      }
}
filter {
    if "search" in [request]{
        grok {
            match => { "request" => ".*\n\{(?<query_body>.*)"} 
            }
        grok {
            match => { "path" => "\/(?<index>.*)\/_search"}     
            }
     If [index] {
      } else {
            mutate {
              add_field  => { "index" => "All" }
        }
      }
      mutate {
          update  => { "query_body" => "{%{query_body}" }
      }
   }
}
output {
  if "search" in [request] and "ignore_unmapped" not in [query_body]{
        elasticsearch {
        hosts => "http://localhost:9200"
        }
   }
}

But when I run ./bin/logstash -f sniff_search.conf , I have this error
[2018-05-28T12:40:31,065][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
at java.lang.Thread.run(Thread.java:844) [?:?]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.0.15.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
... 10 more

If you have any idea what can cause this I would really apreaciate. I just start working with elasticsearch and I cannot see the reason why is happening.

[magnusbaeck (Magnus Bäck)]

Have you configured Filebeat (or whatever you're using) to use SSL?

[Anca_Turc (Anca Ioana)]

I am a bit confused . I am using Beats input plugin , which from what I understood enables Logstash to receive events from the Elastic Beats framework, right ?

Do I have to configure this one to use SSL ?

Thank you very much for your reply

[magnusbaeck (Magnus Bäck)]

I am a bit confused . I am using Beats input plugin , which from what I understood enables Logstash to receive events from the Elastic Beats framework, right ?

Yes.

Do I have to configure this one to use SSL ?

No, but if you do the Logstash input plugin for beats also needs to have SSL enabled.

Please post your Beats configuration file (filebeat.yml or whatever you call it). Make sure you format it as preformatted text using Markdown notation or using the </> toolbar button.

[Anca_Turc (Anca Ioana)]

Ok , so firstly I successfully installed elasticsearch and kibana. Then I have installed logstash and beats input plugin. Then in logstash home I have created sniff_search.conf file and after installed Packetbeat and and configured packetbeat.yml file

      #-------------------------- Elasticsearch output ------------------------------
    #output.elasticsearch:
    # Array of hosts to connect to.
    # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

But for Beats input plugin I am not sure where I can find yml file .
Firstly I am trying to start logstash
/bin/logstash -f sniff_search.conf

and after the Packetbeat for sniffing.

Also I have read that for Beats input plugin ssl can be enabled like this
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/ca.crt"]
ssl_certificate => "/etc/server.crt"
ssl_key => "/etc/server.key"
ssl_verify_mode => "force_peer"
}
}

but I don't know how to generate ssl for localhost .
Actually this is the tutorial which I have followed

https://www.elastic.co/blog/monitoring-the-search-queries
only adapting for localhost, instead of 10.255.4.166.
Thank you very much .