# Why I can't get all log from filebeat to elasticsearch?
**URL:** https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107
**Category:** Beats
**Created:** [March 8, 2018, 3:26pm UTC](https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107 "2018-03-08T15:26:35Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@vincnetwang0736](https://discuss.elastic.co/u/vincnetwang0736)
#### Post date: [March 8, 2018, 3:26pm UTC](https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107/1 "2018-03-08T15:26:35Z")
Here is my logstash config:
```
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/pki/tls/certs/filebeat.crt"]
ssl_certificate => "/etc/pki/tls/certs/logstash.crt"
ssl_key => "/etc/pki/tls/private/logstash.key"
ssl_verify_mode => "force_peer"
codec => json
}
}
filter {
if [fields][logIndex] == "nginx" {
urldecode {
charset => "UTF-8"
field => "url"
}
if [upstreamtime] == "" or [upstreamtime] == "null" {
mutate {
update => { "upstreamtime" => "0" }
}
}
date {
match => ["logtime", "dd/MMM/yyyy:HH:mm:ss Z"]
target => "@timestamp"
}
mutate {
convert => {
"params.addr" => "string"
"params.price" => "float"
}
remove_field => ["port","logtime","message","offset","source","beat","tags"]
}
}
}
output {
if [fields][docType] == "nginx-access" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "nginx-access-log-%{+YYYY.MM.dd}"
document_type => "nginx-access"
codec => "json"
}
}
if [fields][docType] == "nginx-pro" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "nginx-pro-log-%{+YYYY.MM.dd}"
document_type => "nginx-pro"
codec => "json"
}
}
}
```
here is my filebeat config:
`filebeat.prospectors:`
```
- type: log
filebeat.prospectors:
- type: log
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /usr/local/nginx/logs/accessLog.log
fields:
logIndex: nginx
docType: nginx-access
project: app-nginx
- type: log
paths:
- /usr/local/nginx/logpath/*.log
fields:
logIndex: nginx
docType: nginx-pro
project: app-nginx
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["xx.xxx.xx.xxx:5044"]
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash.crt"]
ssl.certificate: "/etc/pki/tls/certs/filebeat.crt"
ssl.key: "/etc/pki/tls/private/filebeat.key"
```
But now I can only get log doctype "nginx-access" at elasticsearch and get nothing doctype "nginx-pro"
Here is one of my pro-log file ,I wrote it by my php coding:
[{"product":"minebit\_otc","userId":"","brand":null,"devid":null,"dpi":null,"mac":null,"model":null,"net":null,"plat":null,"clicktime":"2018-03-06 10:04:52am","token":"","is\_build":1,"eventid":"login","errInfo":{"err\_msg":"\u7b7e\u540d\u65e0\u6548\u3002"},"params":[]},{"product":"minebit\_otc","userId":"","brand":"","devid":"","dpi":"","mac":"","model":"","net":"","plat":"","clicktime":"2018-03-06 10:05:39am","token":"","is\_build":1,"eventid":"login","errInfo":{"error\_status":null,"err\_msg":"\u767b\u5f55\u5931\u8d25\u3002","is\_wrong\_login":"1"},"params":{"login\_name":"wangyi","passwd":"123456","type":"mobile"}}]
strangely，I can get log when I "vi" my log file and just "wq" without doing anything.But I need to get doctype "nginx-pro" automatic.
JSON format is ok,can anyone give me some advise?Thanks a lot!
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 12, 2018, 9:23am UTC](https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107/2 "2018-03-12T09:23:11Z")
Could you try first to either use the filebeat file output or use the most simple logstash config to test if all events come through? If yes you have an issue on the LS side, if no we need to look into Filebeat.
---
### Author:  [@vincnetwang0736](https://discuss.elastic.co/u/vincnetwang0736)
#### Post date: [March 13, 2018, 1:11pm UTC](https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107/3 "2018-03-13T13:11:19Z")
Thank for reply.I used file output and I still got nothing.I think the problem is on filebeat
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 14, 2018, 1:26pm UTC](https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107/4 "2018-03-14T13:26:54Z")
If you didn't have something in the output it seems somehow the log files are not found. Could you enable debug logging and share the log output?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 11, 2018, 3:27pm UTC](https://discuss.elastic.co/t/why-i-cant-get-all-log-from-filebeat-to-elasticsearch/123107/5 "2018-04-11T15:27:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.