FileBeat not logging from a server but it is from other 6

Elastic Stack Beats
1

I have Beats installed in 7 different servers, all with same config but just a little path differences.
It results that from one of those 7 they first day that i just start the service it send the logs perfectly but after that the process keeps running but no data at kibana from that host.

I don't have any special log into registry. That can mean the result of this error.

filebeat.prospectors:

- input_type: log
  paths:
      - /path/to/log/file.log
  document_type: "blabla"

- input_type: log
  paths:
      - /path/to/log/file.log
  document_type: "blabla1"

- input_type: log
  paths:
      - /path/to/log/file.log
  document_type: "blabla2"

  filebeat.idle_timeout: 5s
  fields_under_root: true

#-------------------------Logstash---------------------------

output.logstash:
        hosts: ["hostnumber:5849"]
        loadbalance: true
        worker: 3

Also i realize that i'm the only one working with this stuff in a production servers (don't know why for real). So i know i have a lot of configuration mistakes that i need to solve with more experience.

Want to add here that i have only one elastic node consuming more than 20gb of data daily

Adding logstash configuration if needed:

input {
	beats {
		port => 5844
		codec => json
	}
	beats {
		port => 5845
		codec => json
	}
	beats {
		port => 5846
		codec => json
	}
...

	file {
		path => ["/path/to/log/file.log"]
		codec => json
		type => "blabla"
	}
	file {
		path => ["/path/to/log/file.log"]
		codec => json
		type => "blabla1"
	}
}

filter {
	json {
		source => "message"
	}
}

output {
	if [type]=="blabla" {
		elasticsearch {
                        user => user
                        password => password
			hosts => ["numberhost:9200"]
			index => "blabla-%{+YYYY.MM.dd}"
                        ssl => false
                        ssl_certificate_verification => false
                        truststore => "/path/to/truststore.jks"
                        truststore_password => password
		}
	} else if [type]=="blabla" {
		elasticsearch {
                        user => user
                        password => password
			hosts => ["numberhost:9200"]
			index => "blabla1-%{+YYYY.MM.dd}"
                        ssl => false
                        ssl_certificate_verification => false
                        truststore => "/path/to/truststore.jks"
                        truststore_password => password
		}
	}

	stdout{ codec => rubydebug }
}

Debugging I found that it keeps harvesting a log file for ever. And other 3 just almost ignore, where them all are getting data every second almost. So it makes no sense that filebeat it's ignoring this files. In the other servers it is working perfectly

2

Just check the logstash logs and anything about this specific host. No entry errors or something.

3

Solved by reading this post Infinite harvest - No reading last line

1 Like
4

This topic was automatically closed after 21 days. New replies are no longer allowed.