# When send logs with filebeat input with pipeline, only sent logs to one partiton of kafka

**URL:** <https://discuss.elastic.co/t/when-send-logs-with-filebeat-input-with-pipeline-only-sent-logs-to-one-partiton-of-kafka/199635>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 16, 2019, 10:44am UTC](https://discuss.elastic.co/t/when-send-logs-with-filebeat-input-with-pipeline-only-sent-logs-to-one-partiton-of-kafka/199635 "2019-09-16T10:44:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jinyoung\_Yeom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinyoung_yeom/32/45777_2.png) [@Jinyoung\_Yeom](https://discuss.elastic.co/u/Jinyoung_Yeom)\
**Post date:** [September 16, 2019, 10:44am UTC](https://discuss.elastic.co/t/when-send-logs-with-filebeat-input-with-pipeline-only-sent-logs-to-one-partiton-of-kafka/199635/1 "2019-09-16T10:44:39Z")

</div>

I set logging system with filebeat, kafka and elasticsearch

## Configs : filebeat

#================================ General =====================================  
fields:  
log\_kafka\_topic: es-log-stream

#=========================== Filebeat inputs =============================  
filebeat.inputs:

- type: log  
enabled: true  
paths: ["/var/app/current/logs/debug.log", "/var/app/current/logs/error.log"]  
pipeline: 'filebeat-7.2.0-app-logs'

#============================= Filebeat modules ===============================  
filebeat.config.modules:  
path: ${path.config}/modules.d/\*.yml  
reload.enabled: false  
#reload.period: 10s

## #================================ Outputs ===================================== #----------------------------- Kafka output ------------------------------------ output.kafka: enabled: true hosts: ["localhost:9092"] topic: '%{[fields.log\_kafka\_topic]}' codec.json: pretty: false metadata: full: true partition.round\_robin: reachable\_only: false required\_acks: 1 worker: 2 timeout: 60s version: "2.0.0"

modules.d/nginx.yml

- module: nginx  
access:  
enabled: true  
var.paths:

* * *

## Logs like below:

## 2019-09-16T19:17:10.294+0900 DEBUG [processors] processing/processors.go:183 Publish event: { "@timestamp": "2019-09-16T10:17:10.294Z", "@metadata": { "beat": "filebeat", "type": "\_doc", "version": "7.2.0", "pipeline": "filebeat-7.2.0-nginx-access-default", "topic": "es-log-stream", "partition": 0 }, "agent": { "id": "24fc8854-87fb-4d7c-a69d-c5434b1e90ce", "version": "7.2.0", "type": "filebeat", "ephemeral\_id": "3c4b3941-92ec-4b8f-94d7-627e1425f506", "hostname": "ip-xxxxxxxx" }, "service": { "type": "nginx" }, "input": { "type": "log" }, "fields": { "log\_kafka\_topic": "es-log-stream", }, "event": { "module": "nginx", "dataset": "nginx.access" }, "ecs": { "version": "1.0.0" }, "log": { "offset": 1484771, "file": { "path": "/var/logs/nginx/access.log" } }, "message": "10.xxx.xxx.xxx - - [16/Sep/2019:19:17:09 +0900] "GET / HTTP/1.1" 200 72 "-" "ELB-Health/2.0" 0.000 0.000 "127.0.0.1:8080" "-" -", "fileset": { "name": "access" }, "host": { "name": "ip-xxxxxxxx" } }

every event of logs are sent to partition number "0" all the time.

Without pipeline configuration, "partition.roundrobin" configs works fine with every partitions of kafka.

I think the Processor([https://github.com/elastic/beats/blob/master/libbeat/publisher/processing/default.go#L228](https://github.com/elastic/beats/blob/master/libbeat/publisher/processing/default.go#L228)) might add partition and topic with metadata when "pipeline" was configured.

Any body faced with issue, please give me some help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 14, 2019, 10:44am UTC](https://discuss.elastic.co/t/when-send-logs-with-filebeat-input-with-pipeline-only-sent-logs-to-one-partiton-of-kafka/199635/2 "2019-10-14T10:44:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.