# Parse the logs with FileBeat-ElasticSearch and Display on Kibana

**URL:** <https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 15, 2017, 6:28pm UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802 "2017-11-15T18:28:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shreyas\_Gade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreyas_gade/32/24342_2.png) [@Shreyas\_Gade](https://discuss.elastic.co/u/Shreyas_Gade)\
**Post date:** [November 15, 2017, 6:28pm UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/1 "2017-11-15T18:28:10Z")

</div>

I'm very new to the forums of Elastic. I have gone through these tools and few videos some hours ago and setup Kibana, Logstash and Elasticsearch on my local windows machine (on a private network).

I managed to parse the lines from command line.  
Setup used was ==\> logstash --\> elastic search --\> Kibana.

I need some help to Parse the logs with FileBeat and elastic search then display on Kibana with different filters and statistics. Currently, I don't want to parse using logstash. Simply, want to display the log file in Kibana after processing through FIleBeats --\> ElasticSearch --\> Kibana.

Can you please help me with config files for the same?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [November 15, 2017, 8:09pm UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/2 "2017-11-15T20:09:38Z")

</div>

You need to configure Elasticsearch output for Filebeat:  
[https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html](https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html)

Do you have any specific question regarding the configuration?

---

<div class="post-metadata">

**Author:** ![Shreyas\_Gade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreyas_gade/32/24342_2.png) [@Shreyas\_Gade](https://discuss.elastic.co/u/Shreyas_Gade)\
**Post date:** [November 16, 2017, 7:54am UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/3 "2017-11-16T07:54:57Z")

</div>

**filebeat.yml**

filebeat.prospectors:

`- input_type: log`

```
  paths:
    - C:\Users\gadeshr\elastic\logs\*.log

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

```

Running filebeat on windows as --\> filebeat -e -c filebeat.yml -d "publish"  
Elastic search is running on the default port 9200.  
I can't see the files being processed and displayed in Kibana.

The path C:\Users\gadeshr\elastic\logs\*.log contains log files with extension .log

Sample Log content:

0.216.200.86 - [abc.pqr@xyz.com](mailto:abc.pqr@xyz.com) [10/Nov/2017:10:00:36 +0100] "GET /CONTENT/images/recycle.gif HTTP/1.1" 200 617  
10.216.200.86 - [abc.pqr@xyz.com](mailto:abc.pqr@xyz.com) [10/Nov/2017:10:00:36 +0100] "GET /CONTENT/images/info.gif HTTP/1.1" 200 953  
10.216.200.86 - [abc.pqr@xyz.com](mailto:abc.pqr@xyz.com) [10/Nov/2017:10:00:36 +0100] "GET /CONTENT/de/img/tabButton50.gif HTTP/1.1" 200 6863  
10.217.32.112 - - [10/Nov/2017:10:00:49 +0100] "POST /CONTENT/HTTP/1.1" 200 40554  
10.217.32.114 - - [10/Nov/2017:10:00:50 +0100] "POST /CONTENT/HTTP/1.1" 200 68016  
10.216.200.88 - - [10/Nov/2017:10:00:49 +0100] "POST /CONTENT/HTTP/1.1" 200 316647  
10.216.39.164 - - [10/Nov/2017:10:00:52 +0100] "POST /CONTENT/HTTP/1.1" 200 88765

Also, please help me clean up Kibana. I tried many things from command line earlier and messed up Kibana with lot of text lines.

---

<div class="post-metadata">

**Author:** ![Shreyas\_Gade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreyas_gade/32/24342_2.png) [@Shreyas\_Gade](https://discuss.elastic.co/u/Shreyas_Gade)\
**Post date:** [November 16, 2017, 8:20am UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/4 "2017-11-16T08:20:59Z")

</div>

**FileBeat Logs:**

2017/11/16 08:07:29.812643 spooler.go:63: INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
2017/11/16 08:07:29.840643 log.go:91: INFO Harvester started for file: C:\Users\gadeshr\elastic\logs\Sample.log  
2017/11/16 08:07:29.812643 sync.go:41: INFO Start sending events to output  
2017/11/16 08:07:34.905643 client.go:667: INFO Connected to Elasticsearch version 5.6.4  
2017/11/16 08:07:34.906643 output.go:317: INFO Trying to load template for client: [http://localhost:9200](http://localhost:9200)  
2017/11/16 08:07:34.918643 output.go:341: INFO Template already exists and will not be overwritten.  
2017/11/16 08:07:59.746643 metrics.go:39: INFO Non-zero metrics in the last 30s: filebeat.harvester.open\_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.es.call\_count.PublishEvents=1 libbeat.es.publish.read\_byte  
nd\_acked\_events=1 libbeat.publisher.published\_events=1 publish.events=5 registrar.states.current=3 registrar.states.update=5 registrar.writes=1  
2017/11/16 08:08:29.746643 metrics.go:34: INFO No non-zero metrics in the last 30s

**Elastic Search Logs:**

[2017-11-16T13:33:11,524][INFO][o.e.n.Node] [0oD1ygx] starting ...  
[2017-11-16T13:33:12,528][INFO][o.e.t.TransportService] [0oD1ygx] publish\_address {127.0.0.1:9300}, bound\_addresses {127.0.0.1:9300}, {[::1]:9300}  
[2017-11-16T13:33:16,058][INFO][o.e.c.s.ClusterService] [0oD1ygx] new\_master {0oD1ygx}{0oD1ygxfRS-n\_6IuXYbPZg}{EyGm7v2ARlKrr1KiTUhgZg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)  
[2017-11-16T13:33:16,492][INFO][o.e.g.GatewayService] [0oD1ygx] recovered [4] indices into cluster\_state  
[2017-11-16T13:33:17,444][INFO][o.e.h.n.Netty4HttpServerTransport] [0oD1ygx] publish\_address {127.0.0.1:9200}, bound\_addresses {127.0.0.1:9200}, {[::1]:9200}  
[2017-11-16T13:33:17,444][INFO][o.e.n.Node] [0oD1ygx] started  
[2017-11-16T13:33:17,500][INFO][o.e.c.r.a.AllocationService] [0oD1ygx] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).  
[2017-11-16T13:37:34,944][INFO][o.e.c.m.MetaDataCreateIndexService] [0oD1ygx] [filebeat-2017.11.16] creating index, cause [auto(bulk api)], templates [filebeat], shards [5]/[1], mappings [_default_]  
[2017-11-16T13:37:35,295][INFO][o.e.c.m.MetaDataMappingService] [0oD1ygx] [filebeat-2017.11.16/lDi2N3LNQESM5Wk9AshzeQ] create\_mapping [doc]

How can I check the content in Kibana?  
I cannot see any log content displayed in Kibana.

---

<div class="post-metadata">

**Author:** ![cleverrocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cleverrocks/32/24311_2.png) [@cleverrocks](https://discuss.elastic.co/u/cleverrocks)\
**Post date:** [November 16, 2017, 9:50am UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/5 "2017-11-16T09:50:47Z")

</div>

> [@Shreyas\_Gade](#):
>
> INFO Non-zero metrics in the last 30s:

have you checked the Elasticsearch indexes?  
yourhost:9200/\_cat/indices?v

Please check this and show the output.

---

<div class="post-metadata">

**Author:** ![Shreyas\_Gade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreyas_gade/32/24342_2.png) [@Shreyas\_Gade](https://discuss.elastic.co/u/Shreyas_Gade)\
**Post date:** [November 16, 2017, 11:34am UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/6 "2017-11-16T11:34:54Z")

</div>

**OUTPUT**

```
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2017.11.15 06UitZN_TK6GrIuXIkMQxw 5 1 5 0 37.5kb 37.5kb
yellow open .kibana 0ykkKHGdRgWI6rQPOzZJog 1 1 3 0 25.4kb 25.4kb
yellow open logstash-2017.11.14 IqE7GqoYS3uzJ0b3yVIqaw 5 1 18 0 86.3kb 86.3kb
yellow open filebeat-2017.11.16 lDi2N3LNQESM5Wk9AshzeQ 5 1 1 0 7.4kb 7.4kb
yellow open filebeat-2017.11.15 bzLMP3KfR1KCVupO7bbTBQ 5 1 107 0 90.5kb 90.5kb

```

How do I see it in Kibana?

---

<div class="post-metadata">

**Author:** ![Shreyas\_Gade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shreyas_gade/32/24342_2.png) [@Shreyas\_Gade](https://discuss.elastic.co/u/Shreyas_Gade)\
**Post date:** [November 16, 2017, 12:04pm UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/7 "2017-11-16T12:04:34Z")

</div>

Now, I have added the index 'filebeat-\*' in Kibana and able to see the data loaded there.

Thanks for the help !

---

<div class="post-metadata">

**Author:** ![cleverrocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cleverrocks/32/24311_2.png) [@cleverrocks](https://discuss.elastic.co/u/cleverrocks)\
**Post date:** [November 16, 2017, 1:45pm UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/8 "2017-11-16T13:45:25Z")

</div>

@Shreyas_Gade You are welcome. Glad you able to get the data, 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 14, 2017, 1:45pm UTC](https://discuss.elastic.co/t/parse-the-logs-with-filebeat-elasticsearch-and-display-on-kibana/107802/9 "2017-12-14T13:45:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.