# Filebeat logs to logstash to kibana

**URL:** <https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 17, 2018, 7:47am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842 "2018-01-17T07:47:18Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 17, 2018, 7:47am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/1 "2018-01-17T07:47:18Z")

</div>

I am using ELK and filebeat of version 6 and above. I have configured everything but struggling with seeing filebeat logs in kibana. If i put output as elasticsearch, i can see logs. But when I put as logstash in filebeat.yml, I am unable to see syslogs in kibana. I am trying for first time. I have ensured all ports are open. Please advise.

when i do

curl [http://localhost:9200/filebeat-\*/\_count?pretty](http://localhost:9200/filebeat-*/_count?pretty)  
{  
"count" : 267888,  
"\_shards" : {  
"total" : 5,  
"successful" : 5,  
"skipped" : 0,  
"failed" : 0  
}  
}

and

root@ip-xxxxxx:/usr/share/filebeat/scripts# ./import\_dashboards -dir /etc/kibana/filebeat  
Initialize the Elasticsearch 6.1.1 loader  
Elasticsearch URL [http://127.0.0.1:9200](http://127.0.0.1:9200)  
For Elasticsearch version \>= 6.0.0, the Kibana dashboards need to be imported via the Kibana API.

Please advise

my filebeat.yml

filebeat:  
prospectors:  
- input\_type: log  
paths:  
- /var/log/syslog  
document\_type: syslog  
registry\_file: /var/lib/filebeat/registry  
setup.dashboards.enabled: true  
output:  
logstash:  
hosts: ["0.0.0.0:5044"]  
bulk\_max\_size: 1024

beats.conf  
input{  
beats{  
port =\> "5044"  
}  
}

filter{  
if [type] == "syslog" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}"}  
}  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
}  
}  
output{  
elasticsearch{  
hosts =\> ["localhost:9200"]  
}  
}

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [January 17, 2018, 10:26am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/2 "2018-01-17T10:26:03Z")

</div>

Hi @firdaus,

Could you please share the logs you get from Filebeat? Also, logstash logs could help here

Best regards

---

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 17, 2018, 10:46am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/3 "2018-01-17T10:46:24Z")

</div>

hi,

i somehow found in forums that we need to push dashboards so that we can read logs in kibana dashboard. and I followed same but I get below error when i run

root@ip-xxxxxx:/usr/share/filebeat/bin# ./filebeat setup -E setup.elasticsearch.output=enabled "setup.dashboards.directory=/usr/share/filebeat/kibana"  
Exiting: Template loading requested but the Elasticsearch output is not configured/enabled

Advise please

---

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 17, 2018, 10:47am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/4 "2018-01-17T10:47:10Z")

</div>

these are the logs

filebeat

root@ip-xxxxxx:/usr/share# tail /var/log/filebeat/filebeat  
2018-01-17T09:48:12Z INFO Stopping filebeat  
2018-01-17T09:48:12Z INFO Stopping Crawler  
2018-01-17T09:48:12Z INFO Stopping 0 prospectors  
2018-01-17T09:48:12Z INFO Dynamic config reloader stopped  
2018-01-17T09:48:12Z INFO Crawler stopped  
2018-01-17T09:48:12Z INFO Stopping Registrar  
2018-01-17T09:48:12Z INFO Ending Registrar  
2018-01-17T09:48:12Z INFO Total non-zero values: beat.info.uptime.ms=1193939 beat.memstats.gc\_next=4194304 beat.memstats.memory\_alloc=1333528 beat.memstats.memory\_total=9273104 filebeat.harvester.open\_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.config.reloads=1 libbeat.output.type=elasticsearch libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=1 registrar.writes=1  
2018-01-17T09:48:12Z INFO Uptime: 19m53.939785564s  
2018-01-17T09:48:12Z INFO filebeat stopped.

---

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 17, 2018, 10:47am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/5 "2018-01-17T10:47:42Z")

</div>

logstash logs:

root@ip-172-31-0-150:/usr/share# tail /var/log/logstash/logstash.log  
{:timestamp=\>"2018-01-16T11:54:58.065000+0000", :message=\>#\<LogStash::PipelineReporter::Snapshot:0x5276e99 @data={:events\_filtered=\>1914, :events\_consumed=\>1914, :worker\_count=\>2, :inflight\_count=\>21, :worker\_states=\>[{:status=\>"sleep", :alive=\>true, :index=\>0, :inflight\_count=\>12}, {:status=\>"sleep", :alive=\>true, :index=\>1, :inflight\_count=\>9}], :output\_info=\>[{:type=\>"elasticsearch", :config=\>{"hosts"=\>"localhost", "index"=\>"influxCSVData", "document\_type"=\>"influxCSV\_data\_document\_type"}, :is\_multi\_worker=\>true, :events\_received=\>1914, :workers=\>\<Java::JavaUtilConcurrent::CopyOnWriteArrayList:419037665 [\<LogStash::Outputs::ElasticSearch hosts=\>["localhost"], index=\>"influxCSVData", document\_type=\>"influxCSV\_data\_document\_type", codec=\>\<LogStash::Codecs::Plain charset=\>"UTF-8"\>, workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, flush\_size=\>500, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, action=\>"index", path=\>"/", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>, \<LogStash::Outputs::ElasticSearch hosts=\>["localhost"], index=\>"influxCSVData", document\_type=\>"influxCSV\_data\_document\_type", codec=\>\<LogStash::Codecs::Plain charset=\>"UTF-8"\>, workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, flush\_size=\>500, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, action=\>"index", path=\>"/", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>]\>, :busy\_workers=\>2}, {:type=\>"stdout", :config=\>{}, :is\_multi\_worker=\>false, :events\_received=\>1893, :workers=\>\<Java::JavaUtilConcurrent::CopyOnWriteArrayList:-535213015 [\<LogStash::Outputs::Stdout codec=\>\<LogStash::Codecs::Line charset=\>"UTF-8", delimiter=\>"\n"\>, workers=\>1\>]\>, :busy\_workers=\>0}, {:type=\>"elasticsearch", :config=\>{"hosts"=\>["13.59.198.44:9200"], "index"=\>"%{[@metadeta][beat]}-%{+YYYY.MM.dd}", "document\_type"=\>"%{[@metadata][type]}"}, :is\_multi\_worker=\>true, :events\_received=\>1893, :workers=\>\<Java::JavaUtilConcurrent::CopyOnWriteArrayList:1703971105 [\<LogStash::Outputs::ElasticSearch hosts=\>["13.59.198.44:9200"], index=\>"%{[@metadeta][beat]}-%{+YYYY.MM.dd}", document\_type=\>"%{[@metadata][type]}", codec=\>\<LogStash::Codecs::Plain charset=\>"UTF-8"\>, workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, flush\_size=\>500, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, action=\>"index", path=\>"/", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>, \<LogStash::Outputs::ElasticSearch hosts=\>["13.59.198.44:9200"], index=\>"%{[@metadeta][beat]}-%{+YYYY.MM.dd}", document\_type=\>"%{[@metadata][type]}", codec=\>\<LogStash::Codecs::Plain charset=\>"UTF-8"\>, workers=\>1, manage\_template=\>true, template\_name=\>"logstash", template\_overwrite=\>false, flush\_size=\>500, idle\_flush\_time=\>1, doc\_as\_upsert=\>false, max\_retries=\>3, script\_type=\>"inline", script\_var\_name=\>"event", scripted\_upsert=\>false, retry\_max\_interval=\>2, retry\_max\_items=\>500, action=\>"index", path=\>"/", ssl\_certificate\_verification=\>true, sniffing=\>false, sniffing\_delay=\>5\>]\>, :busy\_workers

---

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 17, 2018, 10:49am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/6 "2018-01-17T10:49:01Z")

</div>

And when i ran below command i am getting

root@ip-xxxxxxx:/usr/share/filebeat/bin# ./filebeat setup -c /etc/filebeat/filebeat.yml -E "setup.dashboards.directory=/usr/share/filebeat/kibana"  
filebeat2018/01/17 10:21:42.928649 beat.go:635: CRIT Exiting: error unpacking config data: more then one namespace configured accessing 'output' (source:'/etc/filebeat/filebeat.yml')  
Exiting: error unpacking config data: more then one namespace configured accessing 'output' (source:'/etc/filebeat/filebeat.yml')

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [January 17, 2018, 11:31am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/7 "2018-01-17T11:31:12Z")

</div>

It looks like you have several outputs configured. Could you please paste the output of: `filebeat export config`. Please paste it as preformatted text, so it doesn't loss indenting

---

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 18, 2018, 6:05am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/8 "2018-01-18T06:05:33Z")

</div>

Thank you so much for your help!! for some reason my system got crashed and i have to start again. Now when i am starting again, what versions do you suggest ? ELK and filebeat??  
Awaiting for your reply..

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [January 18, 2018, 11:12am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/9 "2018-01-18T11:12:44Z")

</div>

Hi,

I would encourage you to use the latest release of each software.  
Looking forward to your configurations!

---

<div class="post-metadata">

**Author:** ![Firdaus](https://avatars.discourse-cdn.com/v4/letter/f/3be4f8/32.png) [@Firdaus](https://discuss.elastic.co/u/Firdaus)\
**Post date:** [January 19, 2018, 10:18am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/10 "2018-01-19T10:18:18Z")

</div>

Hi,

I have used ELK with 2 version and was successful in fetching logs. I have one more query,  
Can we monitor influxdb data in ELK?? if yes, then please advise any link for it. I have searched everywhere and couldnt find influxdb as input plugin. I would be very thankful to you. I had raised a query as well, but nobody responded..

> [@Influxdb and ELK stack](https://discuss.elastic.co/t/influxdb-and-elk-stack/114803):
>
> Please advise how to monitor influx db using ELK stack

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 16, 2018, 10:18am UTC](https://discuss.elastic.co/t/filebeat-logs-to-logstash-to-kibana/115842/11 "2018-02-16T10:18:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.