Filebeat not publishing log to ES


(Tat Dat Pham) #1

Hi,
Im using ES 5.5.0, LS 5.5.0. Kibana 5.5.0 and Filebeat 5.5.0.
Beat -> LS -> ES

Im instaliing filbeat in windows server 2012 for collecting log.
This is my filebeat config

filebeat.prospectors:
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\Office365\Office365\bin\Debug\log\Audit.SharePoint\*.json
  document_type: sharepoint

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\Office365\Office365\bin\Debug\log\Audit.Exchange\*.json
  document_type: exchange

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\Office365\Office365\bin\Debug\log\Audit.AzureActiveDirectory\*.json
  document_type: azure

- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\Office365\Office365\bin\Debug\log\Audit.General\*.json
  document_type: azure



#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["10.1.6.10:5044","10.1.6.20:5044"]
  worker: 2
  compression_level: 3
  loadbalance: true
  index: 'fb-office'

I get log from three folder and have many new file created. ( 100file per 5min; file size from 5-300kb).
When filebeat started, dont have any log (in ES) or connection from server log to LS.
I run filebeat with debug option, dont see any error. Only this log here https://pastebin.com/ABAtE9kx

Thanks!


(Joseph Johney) #2

@tatdat, Please check your logstash pipeline and configurations.


(Tat Dat Pham) #3

@josephjohney , my config LS is running successfull. It is receiving many log from many another serrver :slight_smile:


(Tat Dat Pham) #4

Someone help me ?


(Joseph Johney) #5

Sorry @tatdat.

Your filebeat config is pushing logs to the logstash instances.

If your logstash configurations are correct, the pipeline will be up, with the data pushed to ES.

For verifying, could you please share the logstash config aswell.

Also, check how your Logstash and ES server host configs are (elasticsearch.yml)


(Tat Dat Pham) #6

Thanks @josephjohney
Currently, my LS is working and sending many log to ES. But can't get log from filebeat in server log Office 365 only

Here is my config LS

input {
beats {
port => 5044
}
}
filter {
if [type] == "wineventlog" and [event_id] == 5156 {
drop { }
}
mutate {
add_field => {"beatname" => "%{[@metadata][beat]}"}
add_field => {"beattype" => "%{[@metadata][type]}"}
}
}

output {
rabbitmq {
exchange => "logstash"
exchange_type => "direct"
key => "logstash-key"
host => "10.1.6.23"
vhost => "elastic-stack"
durable => true
persistent => true
port => 5677
user => "logstash"
password => "my-password"
}
}

input {
rabbitmq {
host => "10.1.6.23"
queue => "logstash-queue"
key => "logstash-key"
exchange => "logstash"
threads => 5
exclusive => false
prefetch_count => 256
vhost => "elastic-stack"
port => 5677
user => "logstash"
password => "fim.fpt@123456"
}

}

output {
elasticsearch {
hosts => ["https://10.1.6.16:9200", "https://10.1.6.17:9200", "https://10.1.6.18:9200"]
user => "elastic"
cacert => "/etc/logstash/ca.pem"
ssl => true
password => "mypassword"
manage_template => false
flush_size => 5000
index => "%{beatname}-%{+xxxx.ww}"
document_type => "%{beattype}"
}
}


(Joseph Johney) #7

Could you please try without the RabbitMQ configurations. That is, just the input, filter, and the ES output.

Just trying it step by step

Also try to log the logstash processing.


(Tat Dat Pham) #8

Thanks, I found my problem.
I try to add some file log nginx and get theme (in server collect log O365) and i saw log is sent to ES.
I think format log is problem. Can't get them. Here is log, that i cant get..

https://pastebin.com/eQWJcqT6

How can collect them? All my log file is same format.


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.