Logstash not receiving any Logs

(Vamshi Krishna) #1


I have installed and configured Filebeat in Wind River Linux And I have installed and configured ELK in Ubuntu 16.04. I am trying to access syslog from Wind River. Everything seems fine and all four configurations are running without any error and fault. But the problem is I can't see any log file receiving at Logstash.

below I am adding Filebeat config file.
#=========================== Filebeat prospectors =============================


Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.


    • /var/log/auth.log
    • /var/log/syslog
    • /var/log/*.log
      #- c:\programdata\elasticsearch\logs*

Thanks in Advance!

(Pier-Hugues Pellerin) #2

Hello @vamshi_krishna

First, when posting configuration to discuss can you use code blocks defined in the following example, this helps a lot with readability.

My code

In your provided configured I don't see a logstash output defined?


(Vamshi Krishna) #3

Hello @pierhugues
Sorry for the inconvenience.

Below I am adding Logstash Output from Filebeat Config

# The Logstash hosts
hosts: [""]
bulk_max_size: 1024

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"


(Pier-Hugues Pellerin) #4

Can you also answer the following questions:

  • Can you Add your Logstash config?
  • Are there any errors in the Logstash log or the Filebeat log?

(Vamshi Krishna) #5

Hi, @pierhugues
below I am adding logstash- filter.conf file.

indent preformatted text by 4 spaces
filter {
if [type] == "syslog" {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %
{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

I have also had two other config files. I have followed [https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04#test-filebeat-installation] link to install ELk server.

I have tried many times but still I am not getting logs.
when I try this command,

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

my output is giving all 0 values.

  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]

In Filebeat log,m the last line is showing

2018-02-21T12:08:03.374+0100    ERROR   pipeline/output.go:74   Failed to connect: dial tcp i/o timeout

In Logstash I am getting Connection refused error.


(Pier-Hugues Pellerin) #6

Can we get the input/output part of your Logstash configuration?

(Vamshi Krishna) #7

Logstash Input Configuration

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

Logstash output configuration

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %
 {SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
 {GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]


(Pier-Hugues Pellerin) #8

At first the configuration look fine.

Lets take a few steps back to debug this issue, can you try the following:

  • Remove SSL from both Logstash and Filebeat
  • Remove the All filters from logstash

Do you see any events going through?

(Vamshi Krishna) #9

I have tried as you mentioned in above comment. But it gives me error

curl:(7) Failed to connect to localhost port 9200: Connection refused
zsh: exit 7   curl -XGET 'http://localhost:9200//filebeat-*/_search?pretty'

(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.