Logstash not receiving any Logs

Elastic Stack Beats
1

Hello,

I have installed and configured Filebeat in Wind River Linux 5.0.1.4. And I have installed and configured ELK in Ubuntu 16.04. I am trying to access syslog from Wind River. Everything seems fine and all four configurations are running without any error and fault. But the problem is I can't see any log file receiving at Logstash.

below I am adding Filebeat config file.
#=========================== Filebeat prospectors =============================

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /var/log/auth.log
    • /var/log/syslog
    • /var/log/*.log
      #- c:\programdata\elasticsearch\logs*

Thanks in Advance!

2

Hello @vamshi_krishna

First, when posting configuration to discuss can you use code blocks defined in the following example, this helps a lot with readability.

```
My code
```

In your provided configured I don't see a logstash output defined?

Thanks

3

Hello @pierhugues
Sorry for the inconvenience.

Below I am adding Logstash Output from Filebeat Config

output.logstash:
# The Logstash hosts
hosts: ["192.168.100.239:5044"]
bulk_max_size: 1024

# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"

# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

Thanks!

4

Can you also answer the following questions:

  • Can you Add your Logstash config?
  • Are there any errors in the Logstash log or the Filebeat log?
5

Hi, @pierhugues
below I am adding logstash- filter.conf file.

indent preformatted text by 4 spaces
filter {
if [type] == "syslog" {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %
{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

I have also had two other config files. I have followed [https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-16-04#test-filebeat-installation] link to install ELk server.

I have tried many times but still I am not getting logs.
when I try this command,

curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

my output is giving all 0 values.

pretty'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

In Filebeat log,m the last line is showing

2018-02-21T12:08:03.374+0100    ERROR   pipeline/output.go:74   Failed to connect: dial tcp 192.168.100.239:5044: i/o timeout

In Logstash I am getting Connection refused error.

Thanks.

6

Can we get the input/output part of your Logstash configuration?

7

Logstash Input Configuration

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Logstash output configuration

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %
 {SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
 {GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

thanx!

8

At first the configuration look fine.

Lets take a few steps back to debug this issue, can you try the following:

  • Remove SSL from both Logstash and Filebeat
  • Remove the All filters from logstash

Do you see any events going through?

9

I have tried as you mentioned in above comment. But it gives me error

curl:(7) Failed to connect to localhost port 9200: Connection refused
zsh: exit 7   curl -XGET 'http://localhost:9200//filebeat-*/_search?pretty'
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.