Migration of LSF to filebeat

Elastic Stack Beats
1

I am trying to migrate logstash-forwarder to filebeat. getting certificate error in filebeat though same is being used in logstash-forwarder. I have changed the template and the index is good with %type.
logstash version is 2.2.0 and filebeat is 1.2. Can anyone help?
My running logstash-forwarder configuration is as below:

{
  "network": {
  "servers": [ "ELK_IP:6551" ],
  "ssl ca": "/opt/logstash-forwarder/cert/server.crt",
  "timeout": 30
  },
 "files": [
   {
    "paths": [ "/opt/logstash-forwarder/status.log" ],
    "fields": {
          "type": "status" }
   }
  ]
}

and output as:

input {
  lumberjack {
    port => 6551
    ssl_certificate => "/installdir/ELK/logstash-1.5.4/cert/vm-fead-server.crt"
    ssl_key => "/installdir/ELK/logstash-1.5.4/cert/vm-fead-server.key"
  }
}
filter {
if [type] == "status" {
  grok {
    match => [ "message", "%{DATESTAMP:datestamp} %{GREEDYDATA:name} %{WORD:status}" ]
  }
}
}
output {
    elasticsearch {
        action => "index"
        hosts => "localhost:9200"
        index => "%{type}"
        workers => 1
        manage_template => false
        template_overwrite => true
        template => "/installdir/ELK/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json"
    }
     stdout {
         codec => json
     }
}

I am trying with below filebeat conf

filebeat:
  prospectors:
    -
      paths:
        - /etc/filebeat/logs/*.log
      input_type: log
      document_type: status
  registry_file: /etc/filebeat/registry
output:
  logstash:
    hosts: ["ELK_IP:5044"]
    timeout: 15
    tls:
      certificate: "/etc/filebeat/cert/server.crt"
      certificate_key: "/etc/filebeat/cert/server.key"
  file:
    path: "/etc/filebeat/log/filebeat"
    number_of_files: 3
shipper:
  ignore_outgoing: true
logging:
  files:
    path: /etc/filebeat/log/mybeat
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 3

output as:

input {
  beats {
    port => 5044
    ssl_certificate => "/installdir/ELK/beat/cert/server.crt"
    ssl_key => "/installdir/ELK/beat/cert/server.key"
    ssl => true
  }
}

filter {
if [type] == "status" {
  grok {

    match => [ "message", "%{DATESTAMP:datestamp} %{NUMBER:utilization:int} %{GREEDYDATA:partition}" ]
  }
}
}
output {
  stdout { codec => rubydebug }
  elasticsearch {
    action => "index"
    hosts => "localhost:9200"
    manage_template => false
    template_overwrite => true
    template => "/installdir/ELK/logstash-2.2.0/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.4.1-java/lib/logstash/outputs/elasticsearch/elasticsearch-template.json"
    index => "%{[@metadata][type]}"
    document_type => "%{[@metadata][type]}"
  }
}

The error that i am getting is Starting filebeat: 2016/04/06 07:38:39.473521 transport.go:125: ERR SSL client failed to connect with: x509: certificate signed by unknown authority

2

In logstash you have to configure server certificate + private key and in filebeat the certificate_authorities (certificate containing public key only). You sample configs configure certificate + private key in logstash and filebeat without any certificate authority.

In filebeat:

output:
  logstash:
    hosts: ["ELK_IP:5044"]
    timeout: 15
    tls:
      certificate_authorities: ["/etc/filebeat/cert/server.crt"]

Should do the trick. Please remove private key from filebeat machine.

3

Thank you! it worked this time. I tried with the same parameter, may be I was having some typo.

See you soon on next thread :slight_smile: