Logstash Not able to listen events from Filebeats


(SandhyaRani) #1

Hi there!

I'm trying to send logs through filebeats to logstash,
I have:
Logstash--on local machine
filebeat-on local machine
Elastic search---on host machine(server)

In filebeat.yml I enabled logstash output.
My filebeat.yml
================================
filebeat.prospectors:
- input_type: log

  paths:
    - /home/sandhyarani/Downloads/logstash-5.4.1/bin/*.log
    
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
   #hosts: ["xx.xx.xxx.xx:9200"]  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"#================================ 


Logstash Conf file:
    =========================================================
                input {
                    stdin {
                      
                    }
                }filter {
                    
                    mutate{ add_field => { "Source" => "SKY"}
                      }
                    
                    environment{
                        add_field => ["my_environment", "Hello World, from %{host}"]
                        }
                    if "exception" not in [tags] {        # example output:
                        grok {
                            match => {
                                message => "%{DATESTAMP:timestamp} %{LOGLEVEL:level}( +)\[%{DATA:thread}\] \[%{DATA:mdc}\] %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}"
                    #message => "%{DATESTAMP:timestamp} %{LOGLEVEL:level}( +)\[%{DATA:thread}\] \[%{DATA:mdc}\] %{JAVACLASS:class} - %{GREEDYDATA:logmsg}"
                            }
                            add_tag => ["log"]
                      
                        }        if "log" in [tags] {            grok {
                                match => {
                                    mdc => "%{DATA:username}:%{DATA:deviceId:int}:%{DATA:sessionId}:%{DATA:userInteraction:int}:%{GREEDYDATA:workflowName}"
                                }
                            }
                            date {
                                timezone => GMT
                                match => [
                                               # "16-12-16 21:58:20,606"
                                    "timestamp", "yy-MM-dd HH:mm:ss,SSS"
                                ]
                            }        }    }
                    if [level] in ["ERROR", "error"] or [level] in ["FATAL", "fatal"]{
                        mutate {
                            add_tag => ["alert"]
                        }
                    }
                   
                   
                 if [level] in ["TRACE", "trace"] {
                        mutate {
                        replace => {
                        "level" => "%{level}, 0"
                        }
                      }
                    }
                     
                    else if [level] in ["DEBUG", "debug"]{
                        mutate {
                            replace => {
                            "level" => "%{level}, 1"
                        }
                        }
                    }
                   else if [level] in ["INFO", "info"]{
                        mutate {
                            replace => {
                            "level" => "%{level}, 2"
                        }
                        }
                    }
                    else if [level] in ["WARN", "warn"]{
                        mutate {
                            replace => {
                            "level" => "%{level}, 3"
                        }
                        }
                    }
                    else if [level] in ["ERROR", "error"]{
                        mutate {
                            replace => {
                            "level" => "%{level}, 4"
                        }
                        }
                    }
                        else if [level] in ["FATAL", "fatal"]{
                        mutate {
                            replace => {
                            "level" => "%{level}, 5"
                        }
                        }
                    }}
                output {
                    if "_grokparsefailure" in [tags] {
                        stdout { codec => rubydebug {metadata => true }}
                    }
                if "log" in [tags]{
                if "ERROR" in [level]{
                    elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
                }
                else if "WARN" in [level]{
                    elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
                }
                else if "INFO" in [level]{
                    elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
                }
                else if "FATAL" in [level]{
                    elasticsearch { hosts => ["xxx.xxx.xxx.com"] }
                }
                }
                }

I enabled logstash output in filebeat.yml:
when i run filebeat its showing this

Could any one help with this?

Thanks!


(Noémi Ványi) #2

It seems to me that Filebeat has already read the log files and saved the states in the registry file. But if you delete the registry file (its path is in the second line of your Filebeat output), all the log files will be read again and sent to Logstash.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.