# Full Stack Best Practices

**URL:** <https://discuss.elastic.co/t/full-stack-best-practices/90665>\
**Category:** Elasticsearch\
**Created:** [June 23, 2017, 3:52pm UTC](https://discuss.elastic.co/t/full-stack-best-practices/90665 "2017-06-23T15:52:29Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![DangMA](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@DangMA](https://discuss.elastic.co/u/DangMA)\
**Post date:** [June 23, 2017, 3:52pm UTC](https://discuss.elastic.co/t/full-stack-best-practices/90665/1 "2017-06-23T15:52:29Z")

</div>

This week I spent some time learning the EK stack with a small project.

After battling with EK specific mac OS docker issues (I guess I should blog about this) I started with a small project: tracking firewall activity on a consumer router. The solution ramped me up on EK though I'm wondering what "best practices" I should have considered. Any feedback?

**Hardware/logging:**  
router -\> rpi -\> mac OS/docker EK stack

The router logs are in syslog format. The system logs are stored on the rpi using generic log forwarding. The router.log file is manually pulled from rpi onto the mac for processing. (In a live system, rpi would feed data directly to EK for realtime analysis)

**EK Config**  
Filebeat processes the router.log file, sending it to elasticsearch pipelines which perform additional processing. Docker-compose describes a 2 es node, 1 Kibana node configuration. Visualization in Kibana (geo\_point and related fields).

**Filebeat Config:**  
Custom filebeat.yml and filebeat.template.json. Relevant fields are:  
_filebeat.yml:_  
`prospector: document_type:syslog close_inactive: 2m tags: ["merlin-syslog"] _output.elasticsearch:_ index: "merlin-%{+yyyy.MM.dd}" indices: - index: "merlin-dhcp-%{+yyyy.MM.dd}" when.contains: message: "dnsmasq-dhcp" pipeline: "merlin-syslog" pipelines: - pipeline: "merlin-firewall" when.contains: message: "DROP" - pipeline: "merlin-firewall" when.contains: message: "ACCEPT"`  
_filebeat.template.json_  
`properties: { "@timestamp": { "type": "date" }, "geoip": { "properties": { "city_name": { "ignore_above": 1024, "type": "keyword" }, "continent_name": { "ignore_above": 1024, "type": "keyword" }, "country_iso_code": { "ignore_above": 1024, "type": "keyword" }, "location": { "type": "geo_point" }, "region_name": { "ignore_above": 1024, "type": "keyword" } } } } "template": "merlin-*"`

**Elasticsearch Pipelines**  
Two pipelines: one for generic syslog processing and one for firewall log processing.  
_Generic Pipeline_  
`"processors": [{ "grok": { "field": "message", "patterns": ["%{SYSLOGBASE} %{GREEDYDATA:message}"] }, "date": { "field": "timestamp", "formats": ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"], "ignore_failure": true } } ]`  
_Firewall Pipeline_  
`"processors": [{ "grok": { "field": "message", "patterns": ["%{SYSLOGBASE} %{WORD:action} <4>%{WORD} IN=%{WORD:in_device} OUT=%{WORD:out_device}?(?:%{GREEDYDATA})<1>SRC=%{IP:src_ip} DST=%{IP:dest_ip} %{GREEDYDATA:message}"] }, "date": { "field": "timestamp", "formats": ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"], "ignore_failure": true }, "geoip": { "field": "src_ip", "ignore_failure": true }, "set": { "field": "program", "value": "firewall" } } ]`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 21, 2017, 3:52pm UTC](https://discuss.elastic.co/t/full-stack-best-practices/90665/2 "2017-07-21T15:52:42Z")

</div>

Closed on July 21, 2017, 3:52pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.