Filebeat and Logstash with log rotation


(Uliana Andreeva) #1

Hi there! I'm totally new in ELK framework. I use Filebeat on CentOs 7 to ship logs to Logstash, here is my filebeat.yml:

filebeat.prospectors:
    - type: log
      paths:
        - /opt/tomcat/logs/calendar/log.log
      exclude_lines: ['^org.springframework.']

    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    setup.template.settings:
      index.number_of_shards: 3

    setup.kibana:
      host: "localhost:5601"

    output.logstash:
      hosts: ["localhost:5044"]

Logstash .conf file:

input {
    beats {
        port => "5044"
    }
}
filter {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{SYSLOG5424SD:thread} %{SYSLOGHOST:logger} \[%{JAVACLASS:class}\:%{NUMBER:javaline}\] %{GREEDYDATA:message}" }
      overwrite => [ "message" ]
    }
}
filter {
    date {
      match => [ "timestamp", ISO8601 ]
    }
}
output {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "calendar7-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}

Yesterday everything was fine, I saw all logs in Kibana web-interface, logs were updated normally. But when I checked Kibana today, I saw nothing. Then I launched logstash with -f key and it's config file, new index was created and after that all log lines appeared in Kibana.
log.log file is rotated every 24hours: old log.log is renamed to log.[date].log, and new log.log file is created.

filebeat version 6.1.1
logstash version 6.1.1

Could you please help me to understand where is a problem?


(Kurt Schraeyen) #2

Was Logstash still running before you (re-)launched it?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.