Why I can't get all log from filebeat to elasticsearch?

Elastic Stack Beats
1

Here is my logstash config:

input {
   beats {
      port => 5044
      ssl => true
      ssl_certificate_authorities => ["/etc/pki/tls/certs/filebeat.crt"]
      ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
      ssl_key => "/etc/pki/tls/private/logstash.key"
      ssl_verify_mode => "force_peer"
      codec => json
   }
}

filter {
   if [fields][logIndex] == "nginx" {
      urldecode {
         charset => "UTF-8"
         field => "url"
      }
      if [upstreamtime] == "" or [upstreamtime] == "null" {
         mutate {
            update => { "upstreamtime" => "0" }
         }
      }
      date {
         match => ["logtime", "dd/MMM/yyyy:HH:mm:ss Z"]
         target => "@timestamp"
      }
      mutate {
         convert => {
           "params.addr" => "string"
           "params.price" => "float"
         }

         remove_field => ["port","logtime","message","offset","source","beat","tags"]
      }

   }
}

output {
   if [fields][docType] == "nginx-access" {
   elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "nginx-access-log-%{+YYYY.MM.dd}"
      document_type => "nginx-access"
      codec => "json"
   }
   }

   if [fields][docType] == "nginx-pro" {
   elasticsearch {
      hosts => "localhost:9200"
      manage_template => false
      index => "nginx-pro-log-%{+YYYY.MM.dd}"
      document_type => "nginx-pro"
      codec => "json"
   }
   }

}

here is my filebeat config:
filebeat.prospectors:

- type: log
filebeat.prospectors:


- type: log


  # Paths that should be crawled and fetched. Glob based paths.
  paths:

    - /usr/local/nginx/logs/accessLog.log


  fields:
    logIndex: nginx
    docType: nginx-access
    project: app-nginx

- type: log
  paths:

    - /usr/local/nginx/logpath/*.log

  fields:
    logIndex: nginx
    docType: nginx-pro
    project: app-nginx

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["xx.xxx.xx.xxx:5044"]
  ssl.certificate_authorities:  ["/etc/pki/tls/certs/logstash.crt"]
  ssl.certificate:  "/etc/pki/tls/certs/filebeat.crt"
  ssl.key: "/etc/pki/tls/private/filebeat.key"

But now I can only get log doctype "nginx-access" at elasticsearch and get nothing doctype "nginx-pro"
Here is one of my pro-log file ,I wrote it by my php coding:

[{"product":"minebit_otc","userId":"","brand":null,"devid":null,"dpi":null,"mac":null,"model":null,"net":null,"plat":null,"clicktime":"2018-03-06 10:04:52am","token":"","is_build":1,"eventid":"login","errInfo":{"err_msg":"\u7b7e\u540d\u65e0\u6548\u3002"},"params":[]},{"product":"minebit_otc","userId":"","brand":"","devid":"","dpi":"","mac":"","model":"","net":"","plat":"","clicktime":"2018-03-06 10:05:39am","token":"","is_build":1,"eventid":"login","errInfo":{"error_status":null,"err_msg":"\u767b\u5f55\u5931\u8d25\u3002","is_wrong_login":"1"},"params":{"login_name":"wangyi","passwd":"123456","type":"mobile"}}]

strangely，I can get log when I "vi" my log file and just "wq" without doing anything.But I need to get doctype "nginx-pro" automatic.
JSON format is ok,can anyone give me some advise?Thanks a lot!

2

Could you try first to either use the filebeat file output or use the most simple logstash config to test if all events come through? If yes you have an issue on the LS side, if no we need to look into Filebeat.

3

Thank for reply.I used file output and I still got nothing.I think the problem is on filebeat

4

If you didn't have something in the output it seems somehow the log files are not found. Could you enable debug logging and share the log output?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.