Parsing sample data, but no output printed to the console

Elastic Stack Logstash
1

I downloaded LogStash 6.6.2 today and I am currently going through the tutorial here.

On my Windows 10 machine, I followed the instructions here to install/configure/run Filebeat.

I followed every single step in the tutorial, up to and including executing this instruction:

logstash -f first-pipeline.conf --config.reload.automatic

According to the tutorial, I am supposed to see something like the following printed to the console:

{
    "@timestamp" => 2017-11-09T01:44:20.071Z,
        "offset" => 325,
      "@version" => "1",
          "beat" => {
            "name" => "My-MacBook-Pro.local",
        "hostname" => "My-MacBook-Pro.local",
         "version" => "6.0.0"
    },
          "host" => "My-MacBook-Pro.local",
    "prospector" => {
        "type" => "log"
    },
        "source" => "/path/to/file/logstash-tutorial.log",
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}

However, this is all I see:

Sending Logstash logs to C:/Users/M/Downloads/logstash-6.6.2/logs which is now configured via log4j2.properties
[2019-03-25T11:01:35,062][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-25T11:01:35,082][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-25T11:01:45,311][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-25T11:01:45,900][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-03-25T11:01:45,941][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xfa9705 run>"}
[2019-03-25T11:01:46,036][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-25T11:01:46,057][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-03-25T11:01:46,459][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

There is no other output whatsoever printed to the console.

What should I do to ensure that the output of parsing my sample data is displayed on the console?

2

In your logstash configuration add stdout filter in output part

3

Like this?

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }	
}
output {
    stdout { codec => rubydebug }
	stdout filter
}
4

yes and remove this line

5

Hello Miao,

I've found with previous installs of Filebeat, when I output to Logstash, this below needs to be completed as well. Your Stack Overflow thread does not mention it, but it is listed in the Elastic "Getting Started with Filebeat" steps (Step#4)

Best of Luck & Regards,
John

A connection to Elasticsearch is required to load the index template. If the output is not Elasticsearch, you must load the template manually.

win:

Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator ).

From the PowerShell prompt, change to the directory where you installed Filebeat, and run:

PS > .\filebeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

6

Thanks for your reply, John.

I ran the following command in PowerShell as administrator, as you recommended:

PS C:\Program Files\Filebeat> .\filebeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.
hosts=["localhost:9200"]'

In response, I saw the following message:

Loaded index template

I then ran the following command:

C:\Users\Miao\Downloads\logstash-6.6.2\bin>logstash -f first-pipeline.conf --config.reload.automatic

I saw the following message, and nothing else:

Sending Logstash logs to C:/Users/Miao/Downloads/logstash-6.6.2/logs which is now configured via log4j2.properties
[2019-03-26T08:27:15,065][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-26T08:27:15,081][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-26T08:27:24,432][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-26T08:27:24,823][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"C:/Users/Miao/Downloads/logstash-6.6.2/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}
[2019-03-26T08:27:24,979][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-03-26T08:27:25,026][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x15cf1c2 sleep>"}
[2019-03-26T08:27:25,120][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-26T08:27:25,151][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-03-26T08:27:25,511][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

My first-pipeline.conf file looks like this:

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }	
}
output {
    stdout { codec => rubydebug }
}

My filebeat.yml file looks like this:

filebeat.prospectors:
- type: log
  paths:
    - C:/Users/Miao/Downloads/logstash-tutorial.log/logstash-tutorial-dataset.log
output.logstash:
  hosts: ["localhost:5044"]

According to the tutorial, I should be seeing output like the following:

{
    "@timestamp" => 2017-11-09T01:44:20.071Z,
        "offset" => 325,
      "@version" => "1",
          "beat" => {
            "name" => "My-MacBook-Pro.local",
        "hostname" => "My-MacBook-Pro.local",
         "version" => "6.0.0"
    },
          "host" => "My-MacBook-Pro.local",
    "prospector" => {
        "type" => "log"
    },
        "source" => "/path/to/file/logstash-tutorial.log",
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}

However, I failed to see any such output printed to the console.

Any ideas on what else I might be missing?

7

Hi Miao,

This below may be your problem. It appears, since PS-commands, you are WindowsOS. The Elastic Tutorial you used shows a Linux Setup as the example (below):

Try adjusting your path like so ( \ ):
C:\Users\Miao\Downloads\logstash-tutorial.log\logstash-tutorial-dataset.log

Edit: However, the slash may be interchangeable for setting a location with Filebeat. Windows does allow for it, but I have never tested it.

Stack Overflow: "Modern Windows can generally use both \ and / interchangeably for filepaths"

8

@John_Hampton Thank you for your reply. I still have no success seeing any output on the console. :frowning:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.