# Collecting from syslog data-lake

**URL:** <https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 15, 2020, 9:01pm UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363 "2020-07-15T21:01:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\
**Post date:** [July 15, 2020, 9:01pm UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/1 "2020-07-15T21:01:29Z")

</div>

hi  
we got a datalake from syslog datasets and the devices/endpoints are logging in format of  
`/data/<collect_port>/<ip_address>/<severity>.<facility>.log`  
eg  
`/data/514/172.128.4.2/auth.info.log`

And within the datasets, the data pattern is [rfc5424](https://tools.ietf.org/html/rfc5424) format

```auto
<38>1 2020-07-15T18:40:49+01:00 client.hostname sshd 30077 - - user root login class [preauth]

```

Obviously the grok expressions in default filebeat cannot parse this

How to make

- the host.ip from the name of the file
- the host.name from the client.hostname parameter
- Any ready-made pattern for rfc5424 template?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [July 16, 2020, 7:59am UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/2 "2020-07-16T07:59:01Z")

</div>

Hi!

It seems there is something in progress [https://github.com/elastic/beats/pull/15467](https://github.com/elastic/beats/pull/15467). Maybe you can try out this branch otherwise I would suggest you to just ship the logs to a Logstash server and process them with a GROK pattern.

---

<div class="post-metadata">

**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\
**Post date:** [July 16, 2020, 4:00pm UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/3 "2020-07-16T16:00:47Z")

</div>

@ChrsMark  
Thanks for the info.

When you say ship to logstash, by which means? i.e. How to get into logstash? Do you mean by using filebeat and consider everything as "raw message" into logstash?  
what's the best method to ship to logstash?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [July 17, 2020, 7:08am UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/4 "2020-07-17T07:08:40Z")

</div>

Yes shipping from Filebeat to Logstash: [https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html](https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html)

And processing the logs at Logstash's side with GROK patterns: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html)

---

<div class="post-metadata">

**Author:** ![kelk](https://avatars.discourse-cdn.com/v4/letter/k/13edae/32.png) [@kelk](https://discuss.elastic.co/u/kelk)\
**Post date:** [July 17, 2020, 7:18am UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/5 "2020-07-17T07:18:15Z")

</div>

ok. But that's still the chicken-egg problem whereby the filebeat would need basic parsing.  
Is there an option to send a raw field output (i.e uncooked data) directly to logstash?  
as though filebeat just refect exact/pure event from the data lake to logstash without parsing it at all?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [July 17, 2020, 7:27am UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/6 "2020-07-17T07:27:32Z")

</div>

Yes, Filebeat will send the messages to Logstash without any pre processing. In this Filebeat will be used as a basic collector and Logstash as the aggregator which will be performing the analysis of the logs.

C.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2020, 9:27am UTC](https://discuss.elastic.co/t/collecting-from-syslog-data-lake/241363/7 "2020-08-14T09:27:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.