Logstash service stop working

Elastic Stack Logstash
1

Here is my configuration that works for us at the moment ( Filebeat -> Logstash -> Elasticsearch)

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

The problem is I also want my log to be stored into files on the server so I added the file plugin.
output { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } file { path => "/var/data/syslog/%{+YYYY}/%{+MM}/%{+dd}/%{hostname}/%{source}.log" } }

logstash configtest
Configuration OK

I restart logstash and it runs fine for few minutes then as soon as Filebeat sends a file to logstash the service stop working.

I have nothing in my logstash log even with ' stdout ' in my output config.

Anything wrong with my config ? My codec ? Is it possible to send filebeat input to file output ?

2

Can you try running with debug and verbose modes?

3

./bin/logstash -f /etc/logstash/conf.d/*.conf -v --debug --verbose
> Settings: Default pipeline workers: 1
> Logstash startup completed

Meanwhile my filebeat is running and sent file:

2016/06/29 12:54:39.159609 output.go:87: DBG  output worker: publish 4 events
2016/06/29 12:54:39.168321 publish.go:104: INFO Events sent: 4
2016/06/29 12:54:39.168648 registrar.go:162: INFO Registry file updated. 2 states written.

Still nothing on the Logstash side and nothing in the /var/log/logstash/logstash.log

If we look at the Filebeat report above we see the date 2016/06/29 12:54:39 while my system time is : Wed Jun 29 08:54:39 EDT 2016.
As far as I know it should not be an issue since im parsing the logs in Logstash to adjust timestamp.
Thanks

4

[solved]
I did not have the right permission on the folder !!!