# Basic filebeats =\> logstash =\> elastic setup

**URL:** <https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323>\
**Category:** Logstash\
**Created:** [September 20, 2018, 3:48pm UTC](https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323 "2018-09-20T15:48:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nuketro0p3r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuketro0p3r/32/32584_2.png) [@nuketro0p3r](https://discuss.elastic.co/u/nuketro0p3r)\
**Post date:** [September 20, 2018, 3:48pm UTC](https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323/1 "2018-09-20T15:48:40Z")

</div>

I'm trying to debug / check if I got everything right with my logstash setup

History:  
The logs fed from Filebeats to ES work perfectly  
Only downside is that the geoip information is not set

I tried to add logstash in the middle. The setup works, but now I get very limited logs instead of the original format and the dashboards don't work.

I figured if I start with an empty filter, this should work and then I can add additional fields incrementally, but apparently something doesn't work...

My config file for logstash is:

input {  
beats {  
port =\> 5400  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "filebeat-%{+YYYY.MM.dd}"  
document\_type =\> "doc"  
}  
stdout { codec =\> rubydebug }  
}

End goal: Have the exact same output format in ES as if logstash was not in the middle  
End goal2: Add a filter to enrich output

Problem: The data fed by logstash is very limited (see below):  
{  
"\_index": "filebeat-2018.09.20",  
"\_type": "doc",  
"\_id": "PaWo92UBhe-uS7seR1Tz",  
"\_version": 1,  
"\_score": null,  
"\_source": {  
"input": {  
"type": "log"  
},  
"source": "C:\nginx-1.15.2\logs\test\_PROJ\_DEV\_access.log",  
"message": "127.0.0.1 - - [20/Sep/2018:16:57:46 +0200] "POST /api/test\_setup/FindAll HTTP/1.1" 200 6888 "[http://localhost:6600/test\_project/search-item](http://localhost:6600/test_project/search-item)" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36"",  
"offset": 70812,  
"beat": {  
"name": "ME\_PC",  
"hostname": "ME\_PC",  
"version": "6.4.1"  
},  
"host": {  
"name": "ME\_PC"  
},  
"@timestamp": "2018-09-20T15:44:59.479Z",  
"tags": [  
"beats\_input\_codec\_plain\_applied"  
],  
"prospector": {  
"type": "log"  
},  
"@version": "1"  
},  
"fields": {  
"@timestamp": [  
"2018-09-20T15:44:59.479Z"  
]  
},  
"sort": [  
1537458299479  
]  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 20, 2018, 5:15pm UTC](https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323/2 "2018-09-20T17:15:49Z")

</div>

Logstash has no built-in support for HTTP access logs. See [https://www.elastic.co/guide/en/logstash/current/config-examples.html](https://www.elastic.co/guide/en/logstash/current/config-examples.html) for an example of how to configure it by hand.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 20, 2018, 5:21pm UTC](https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323/3 "2018-09-20T17:21:53Z")

</div>

Filebeat modules do the parsing through an ingest pipeline in Elasticsearch. Now that you are no longer writing directly into Elasticsearch, this pipeline is not called. The Logstash Elasticsearch output plugin supports specifying a pipeline, so that is what probably need to change.

I would suspect this to be documented somewhere, so let me look around. How to replicate the Filebeat module processing in Logstash seems to be documented [here](https://www.elastic.co/guide/en/logstash/6.4/filebeat-modules.html).

---

<div class="post-metadata">

**Author:** ![nuketro0p3r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuketro0p3r/32/32584_2.png) [@nuketro0p3r](https://discuss.elastic.co/u/nuketro0p3r)\
**Post date:** [September 21, 2018, 8:30am UTC](https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323/4 "2018-09-21T08:30:46Z")

</div>

Thanks Christian. I would've never found that in a million years 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2018, 8:30am UTC](https://discuss.elastic.co/t/basic-filebeats-logstash-elastic-setup/149323/5 "2018-10-19T08:30:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.