When send logs with filebeat input with pipeline, only sent logs to one partiton of kafka

Elastic Stack Beats
1

I set logging system with filebeat, kafka and elasticsearch

Configs :
filebeat

#================================ General =====================================
fields:
log_kafka_topic: es-log-stream

#=========================== Filebeat inputs =============================
filebeat.inputs:

  • type: log
    enabled: true
    paths: ["/var/app/current/logs/debug.log", "/var/app/current/logs/error.log"]
    pipeline: 'filebeat-7.2.0-app-logs'

#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#reload.period: 10s

#================================ Outputs =====================================
#----------------------------- Kafka output ------------------------------------
output.kafka:
enabled: true
hosts: ["localhost:9092"]
topic: '%{[fields.log_kafka_topic]}'
codec.json:
pretty: false
metadata:
full: true
partition.round_robin:
reachable_only: false
required_acks: 1
worker: 2
timeout: 60s
version: "2.0.0"

modules.d/nginx.yml

  • module: nginx
    access:
    enabled: true
    var.paths:

    • /var/logs/nginx/access.log
      var.convert_timezone: false

    error:
    enabled: true
    var.paths:

    • /var/logs/nginx/error.log
      var.convert_timezone: false

Logs like below:

2019-09-16T19:17:10.294+0900 DEBUG [processors] processing/processors.go:183 Publish event: {
"@timestamp": "2019-09-16T10:17:10.294Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.2.0",
"pipeline": "filebeat-7.2.0-nginx-access-default",
"topic": "es-log-stream",
"partition": 0
},
"agent": {
"id": "24fc8854-87fb-4d7c-a69d-c5434b1e90ce",
"version": "7.2.0",
"type": "filebeat",
"ephemeral_id": "3c4b3941-92ec-4b8f-94d7-627e1425f506",
"hostname": "ip-xxxxxxxx"
},
"service": {
"type": "nginx"
},
"input": {
"type": "log"
},
"fields": {
"log_kafka_topic": "es-log-stream",
},
"event": {
"module": "nginx",
"dataset": "nginx.access"
},
"ecs": {
"version": "1.0.0"
},
"log": {
"offset": 1484771,
"file": {
"path": "/var/logs/nginx/access.log"
}
},
"message": "10.xxx.xxx.xxx - - [16/Sep/2019:19:17:09 +0900] "GET / HTTP/1.1" 200 72 "-" "ELB-Health/2.0" 0.000 0.000 "127.0.0.1:8080" "-" -",
"fileset": {
"name": "access"
},
"host": {
"name": "ip-xxxxxxxx"
}
}

every event of logs are sent to partition number "0" all the time.

Without pipeline configuration, "partition.roundrobin" configs works fine with every partitions of kafka.

I think the Processor(https://github.com/elastic/beats/blob/master/libbeat/publisher/processing/default.go#L228) might add partition and topic with metadata when "pipeline" was configured.

Any body faced with issue, please give me some help.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.