Getting start (logstash 6.3.2-1 / filebeat 6.3.2) on debian 9.4 /

Hello,

New user, i'd like try the power of ELK on a debian 9.4 / openjdk version "1.8.0_151"

2 days I try to run ELK with no success so I decided to start again from the beginning and I encountered a issue fairly quickly.... Internet and Google is not my friend this time :frowning:

I have installed package DEB filebeat 6.3.2 and logstash 6.3.2-1 (lastet versions on the site).

I used the "getting start" guide https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html

The step 2 "Stashing your first event" in the guide) works fine

logstash -e 'input { stdin { } } output { stdout {} }'

It return as expeted but on this format :

bonjour
{
"@timestamp" => 2018-08-07T13:51:33.491Z,
  "@version" => "1",
   "message" => "bonjour",
      "host" => "debian"
}

The step 3 "Parsing Logs with Logstash" doesn't work as expected and i think i respect all the guide. So, i put all my config files, command and output if necessary.

My filebeat config file is

    root@debian:/etc/filebeat# cat filebeat.yml | egrep -v '^\ *#|^$'
filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/messages
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.logstash:
  hosts: ["localhost:5044"]

My /var/log/message contains lines like
Aug 7 15:14:34 debian liblogging-stdlog: action 'action 0' resumed (module 'builtin:omfwd') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
I also run some "logger" commands which are presents in /var/log/messages but nothing in logstash.

My logstash config file is

root@debian:/etc/filebeat# cat /etc/logstash/test.conf
input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
    stdout { codec => rubydebug }
}

I start filebeat like the user guide (foreground and debug mode)

filebeat -e -c filebeat.yml -d "publish"
...
2018-08-07T16:49:16.004+0200    INFO    crawler/crawler.go:48   Loading Inputs: 1
2018-08-07T16:49:16.006+0200    INFO    crawler/crawler.go:82   Loading and starting Inputs completed. Enabled inputs: 0
2018-08-07T16:49:16.004+0200    INFO    [monitoring]    log/log.go:97   Starting metrics logging every 30s
2018-08-07T16:49:16.008+0200    INFO    cfgfile/reload.go:122   Config reloader started
2018-08-07T16:49:16.009+0200    INFO    cfgfile/reload.go:214   Loading of config files completed.

And logstash...
First, i try the config.test_and_exit and the return is ok.
When I start the process now

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --config.reload.automatic

I have the output below but nothing looks like a "normal" event as thrid part of the user guide.

    root@debian:/etc/logstash# /usr/share/logstash/bin/logstash -f /etc/logstash/test.conf --config.reload.automatic
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continu ing using the defaults
    Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
    [WARN ] 2018-08-07 16:33:54.428 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [INFO ] 2018-08-07 16:33:57.166 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.2"}
    [INFO ] 2018-08-07 16:34:09.457 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline .batch.size"=>125, "pipeline.batch.delay"=>50}
    [INFO ] 2018-08-07 16:34:11.222 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
    [INFO ] 2018-08-07 16:34:11.270 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0 x24fdd77a run>"}
    [INFO ] 2018-08-07 16:34:11.674 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelin es=>[:main], :non_running_pipelines=>[]}
    [INFO ] 2018-08-07 16:34:11.714 [[main]<beats] Server - Starting server on port: 5044
    [INFO ] 2018-08-07 16:34:12.895 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
    [INFO ] 2018-08-07 16:34:34.313 [defaultEventExecutorGroup-5-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:60474] Handling exception: org.logs tash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 52
    [WARN ] 2018-08-07 16:34:34.325 [nioEventLoopGroup-3-2] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pip eline. It usually means the last handler in the pipeline did not handle the exception.
    io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 52
            at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
            at 
            ... 8 more
    [INFO ] 2018-08-07 16:34:34.537 [defaultEventExecutorGroup-5-1] BeatsHandler - [local: 127.0.0.1:5044, remote: 127.0.0.1:60474] Handling exception: org.logs tash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 62
    [WARN ] 2018-08-07 16:34:34.547 [nioEventLoopGroup-3-2] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pip eline. It usually means the last handler in the pipeline did not handle the exception.
    io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 62
            at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]

            ... 10 more

I loop on the same articles on the net without success. Hope someone will have mercy on me or simply give some investigation leads i will gracefully...

enabled: true

that's all

Simple and efficient, Thanks you VERY much :grinning:.

logstash output is very happy now (and me too)

{
         "input" => {
        "type" => "log"
    },
    "prospector" => {
        "type" => "log"
    },
       "message" => "Aug  7 17:28:47 debian root: test 1235",
          "host" => {
        "name" => "debian"
    },
      "@version" => "1",
        "source" => "/var/log/messages",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "beat" => {
         "version" => "6.3.2",
            "name" => "debian",
        "hostname" => "debian"
    },
    "@timestamp" => 2018-08-07T15:28:53.213Z,
        "offset" => 3903836
}

For information, enabled: true must be configured in filebeat.yml:

root@debian:/etc# cat /etc/filebeat/filebeat.yml | grep "  enabled:"
  enabled: true

Come on for the next step !!!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.