Configuring filebeat to send specified logs to E.L.K server's logstash HELP!

Elastic Stack Beats
1

Hi there,

About two days ago I started looking into ELK and decided to try it out on a few machines.

My set-up is the following: 1 E.L.K server, 2 Client servers and 1 Laptop to access Kibana.

I have followed the official guide of installation on Ubuntu 14.04 and reached the part where I start configuring my client servers to send out the logs to the logstash.

I have created the following filebeat.yml stored in /etc/filebeat/

filebeat:
  prospectors:
    -
      paths:
        - /var/log/auth.log
        - /var/log/syslog
      #  - /var/log/*.log

      input_type: log
      
      document_type: syslog

  registry_file: /var/lib/filebeat/registry

output:
  logstash:
    hosts: ["192.168.71.104:5044"]
    bulk_max_size: 1024

    tls:
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

shipper:

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

I think it's fine and should work out, but when I try to restart the service by executing sudo service filebeat restart it gives me the following message:

* Restarting Sends log files to Logstash or directly to Elasticsearch. filebeat 2016/03/18 10:28:35.803779 transport.go:125: ERR SSL client failed to connect with: dial tcp 192.168.71.104:5044: getsockopt: connection refused

First I thought something was wrong with the certificate that I created on the E.L.K server and transferred over, but I triple checked it and even created new ones and it still gave me that error.

What is going on? My logstash should be properly configured over at the E.L.K server, if necessary I can show the config files. There's 3.

I also tried to troubleshoot a little further and noticed that the logstash service on the E.L.K server was not running. I tried to restart it and it successfully started, but after a while I checked the status again it told me that filebeat was not running. Shouldn't it be up at all times?

Hopefully someone can help me out, thanks.

2

Please do. Surround it with <pre> </pre>tags so we don't lose the formatting.

3

Is logstash running? Can you ping target machine? Can you telnet into logstash?

What's your logstash config?

4

Logstash config:

/etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

/etc/logstash/conf.d/10-syslog-filter.conf

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

/etc/logstash/conf.d/30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

@steffens I can ping the target machine just fine, haven't tried to telnet into logstash. How do I do that? Logstash is not running, when I try to start it manually it will go back to not running status shortly after.

5

Can you look for logstash log output why it's not running?

for telnet use telnet <ip> <port>.

6

Hi,

Thank you and sorry for the late reply.

I can not telnet into logstash or into the server at all. ELK Server > Client Server and the other way around.
It tells me that the connection is refused.

7

cause logstash is not running. Check for logs from logstash why it's not running.

8

Hi, thank you for your fast reply.

I'm new to this and and it's my first experience with Ubuntu as well, but am very interested and appreciate your help.

How would I be able to see my logstash logs? Are they saved in a specific directory? Sorry!

9

If you want to see possible error messages or log information on the logstash server you can do:
tail -100 /var/log/logstash/logstash.log

tail -100 /var/log/logstash/logstash.err

From the error message you get, it could also be a firewall rule that block the communication between the two servers

10

Damn, the log told me a lot. I can elaborate on this if necessary.

In short:
I had to wrong cert location set up + a few typos on the input configuration file.

I fixed those and everything is working! Thanks a lot, I really appreciate it.

1 Like
11

i want to ask how about my error after i do the command in my server:
tail -100 /var/log/logstash/logstash.log
{:timestamp=>"2016-04-06T09:02:54.238000+0700", :message=>"The error reported is: \n pattern %{HOST:service} not defined"}

please help me.
Thanks

12

This means that although you have defined a pattern "HOST" in your logstash config, the pattern is not defined in the pattern directory.

13

your config (I guess for elasticsearch output) is faulty?

14

Hi @brayndasilva, can you help me how to fix the "host" ? I got same error. -(

15

@bv4wolf this topic is rather old. please start a new discussion including logs + config files if possible (use </> button to format file content).

16

Thanks @steffens. I've know how to fix it.