Filebeat Elastic working, but filebeat logstash not working

Elastic Stack Logstash
1

Hi, I am searching for a solution to this for past 2 days, but have not found it.

Filebeat can send data to elasticsearch, but it is not working if I am sending it by logstash.

Here is the filebeat.yml file,

filebeat.prospectors:
- input_type: log
  paths:  
   - /var/log/syslog
     document_type: syslog

- input_type: log
   paths:
     - /var/log/nginx/kibana_access.log
     document_type: nginx

output.logstash:
hosts: ["localhost:5044"]

This is the input file in logstash,

input {
 beats {
   port => 5044
 }
}

This is the output file

output {
     elasticsearch {
   hosts => ["localhost:9200"]
   manage_template => false
   index => "%{type}-index"
   document_type => "[type]"
 }
}

If I create an index by directly outputing to elasticsearch from filebeat, it works perfectly.

The logstash never opens its port 5044, why so ?

2

The logstash never opens its port 5044, why so ?

Is Logstash starting at all? Is there anything in Logstash's own log? How did you verify that the port isn't opened?

It looks like your document_type lines are indented one step too far.

3

Logs are being written there. Here are a few lines from the log, just pasting 1 from each step,

[2017-04-21T11:26:29,938][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"beats", :type=>"input", :class=>LogStash::Inputs::Beats}

[2017-04-21T11:26:29,958][DEBUG][logstash.codecs.plain ] config LogStash::Codecs::Plain/@id = "plain_59be7464-bfed-450a-baa2-35b2101696e3"

[2017-04-21T11:26:29,960][DEBUG][logstash.inputs.beats ] config LogStash::Inputs::Beats/@port = 5044

[2017-04-21T11:26:29,998][DEBUG][logstash.filters.grok ] config LogStash::Filters::Grok/@match = {"message"=>"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}"}

[2017-04-21T11:26:30,329][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@hosts = [//localhost:9200]

BUt the Filebeat logs are,

2017-04-21T02:36:01+05:30 ERR Connecting error publishing events (retrying): dial tcp 127.0.0.1:5044: getsockopt: connection refused

2017-04-21T02:36:31+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.publisher.published_events=2046

4

How can we specify the index name when writing from filebeat to logstash and then to elasticsearch ?

5

Logs are being written there. Here are a few lines from the log, just pasting 1 from each step,

Okay, but those are just early debug-level logs. Are there any errors or warnings? Is Logstash starting up properly? Do you have any firewall or similar that might be blocking the access? Have you checked with e.g. netstat whether Logstash is in fact listening on the port?

How can we specify the index name when writing from filebeat to logstash and then to elasticsearch ?

It's Logstash that chooses the index name based on the index option of its elasticsearch output.

6

Have you started a logstash pipeline properly?
Try to start a pipeline using the following command.
And include your configuration options in the first-pipeline.conf file
Go to your bin folder of logstash using the command-line and try to execute the following command and check whether the pipeline has started properly.
C:\elk\logstash-5.1.2\bin>logstash -f first-pipeline.conf --config.reload.automatic

After the pipeline has successfully started following message will be displayed.
11:43:35.045 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
11:43:35.120 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.