# Passing Apache logs file to Kafka

**URL:** <https://discuss.elastic.co/t/passing-apache-logs-file-to-kafka/129342>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 24, 2018, 3:54pm UTC](https://discuss.elastic.co/t/passing-apache-logs-file-to-kafka/129342 "2018-04-24T15:54:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kaplun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaplun/32/30420_2.png) [@kaplun](https://discuss.elastic.co/u/kaplun)\
**Post date:** [April 24, 2018, 3:54pm UTC](https://discuss.elastic.co/t/passing-apache-logs-file-to-kafka/129342/1 "2018-04-24T15:54:04Z")

</div>

Hello,

I have this somehow simple yet tricky workflow I am trying to setup with just Filebeat and Kafka.

In particular, I am trying to pipe Apache logs to Kafka via Filebeat but I would like to partition data on Kafka using the origin IP address (this because log ordering is preserved in Kafka only within a partition, and I am trying to preserve the ordering of events at least for a specific IP address).

The Kafka output component in Filebeat supports configuring a `key` using a _format string_. Is there a way to actually set it to extract the IP address at the beginning of each Apache log entry? (e.g. with a regexp)?

Best regards,  
Samuele

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [April 26, 2018, 9:46am UTC](https://discuss.elastic.co/t/passing-apache-logs-file-to-kafka/129342/2 "2018-04-26T09:46:30Z")

</div>

Hi @kaplun,

I don't think that's possible as of today, as most of the processing is done later in the chain, by Ingest node or Logstash.

We are working on something that should help you on this task, but it will take some time to have it in: [https://github.com/elastic/beats/pull/6925](https://github.com/elastic/beats/pull/6925).

Something you can could do as of today is deploying logstash, and process logs there, then use the kafka output once the IP field is extracted.

Best regards

---

<div class="post-metadata">

**Author:** ![kaplun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaplun/32/30420_2.png) [@kaplun](https://discuss.elastic.co/u/kaplun)\
**Post date:** [April 26, 2018, 11:42am UTC](https://discuss.elastic.co/t/passing-apache-logs-file-to-kafka/129342/3 "2018-04-26T11:42:02Z")

</div>

Hi @exekias,

I see. Yeah we were trying to keep the chain super thin, and avoid deploying Logstash with the whole Java framework. Let's see if we can find a workaround.  
Thanks a lot for your answer!

BTW the PR you pointed me to got opened 3 days ago 😄 nice timings!  
Samuele

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 24, 2018, 11:42am UTC](https://discuss.elastic.co/t/passing-apache-logs-file-to-kafka/129342/4 "2018-05-24T11:42:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.