Filebeat version 6.4.0 unable to send logs to Kafka directly . Filebeat not starting

I want to send filebeat (logs) data to kafka directly. I have commented logstash/elasticsearch output and set the kafka output like below. But after that my filebeat instance is not starting . Getting error message
Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch
#--------------------------- KAFKA ----------------------------------------------
output.kafka:
enabled:true
hosts: ["127.0.0.1:9092"]
topic: log

Welcome to our community! :smiley:

Can you paste your Filebeat logs?
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Thanks for the reply . here is the /var/log/filebeat/filebeat log

    2020-08-09T13:06:46.572-0400    INFO    kafka/log.go:53 producer/broker/0 starting up

    2020-08-09T13:06:46.572-0400    INFO    kafka/log.go:53 producer/broker/0 state change to [open] on logs/0

    2020-08-09T13:06:46.572-0400    INFO    kafka/log.go:53 producer/leader/logs/0 selected broker 0

    2020-08-09T13:06:46.572-0400    INFO    kafka/log.go:53 producer/leader/logs/0 state change to [flushing-3]

    2020-08-09T13:06:46.572-0400    INFO    kafka/log.go:53 producer/leader/logs/0 state change to [normal]

    2020-08-09T13:06:46.832-0400    INFO    kafka/log.go:53 Connected to broker at 100.97.62.105:9092 (registered as #0)

    2020-08-09T13:06:47.093-0400    INFO    kafka/log.go:53 producer/broker/0 state change to [closing] because EOF

    2020-08-09T13:06:47.094-0400    INFO    kafka/log.go:53 Closed connection to broker 100.97.62.105:9092

    2020-08-09T13:06:47.094-0400    INFO    kafka/log.go:53 producer/leader/logs/0 state change to [retrying-3]

    2020-08-09T13:06:47.094-0400    INFO    kafka/log.go:53 producer/leader/logs/0 abandoning broker 0

    2020-08-09T13:06:47.094-0400    INFO    kafka/log.go:53 producer/broker/0 shut down

    2020-08-09T13:06:47.194-0400    INFO    kafka/log.go:53 client/metadata fetching metadata for [logs] from broker 100.97.62.105:9092

    2020-08-09T13:06:47.456-0400    INFO    kafka/log.go:53 producer/broker/0 starting up

    2020-08-09T13:06:47.456-0400    INFO    kafka/log.go:53 producer/broker/0 state change to [open] on logs/0

    2020-08-09T13:06:47.456-0400    INFO    kafka/log.go:53 producer/leader/logs/0 selected broker 0

    2020-08-09T13:06:47.456-0400    INFO    kafka/log.go:53 producer/leader/logs/0 state change to [flushing-3]

    2020-08-09T13:06:47.456-0400    INFO    kafka/log.go:53 producer/leader/logs/0 state change to [normal]

    2020-08-09T13:06:47.494-0400    INFO    beater/filebeat.go:437  Stopping filebeat
    2020-08-09T13:06:47.494-0400    INFO    crawler/crawler.go:139  Stopping Crawler
    2020-08-09T13:06:47.494-0400    INFO    crawler/crawler.go:149  Stopping 1 inputs
    2020-08-09T13:06:47.494-0400    INFO    input/input.go:149      input ticker stopped
    2020-08-09T13:06:47.494-0400    INFO    input/input.go:167      Stopping Input: 8669900041677695518
    2020-08-09T13:06:47.494-0400    INFO    log/harvester.go:272    Reader was closed: /var/log/dmesg. Closing.
    2020-08-09T13:06:47.494-0400    INFO    log/harvester.go:272    Reader was closed: /var/log/messages. Closing.
    2020-08-09T13:06:47.494-0400    INFO    crawler/crawler.go:165  Crawler stopped
    2020-08-09T13:06:47.495-0400    INFO    registrar/registrar.go:356      Stopping Registrar
    2020-08-09T13:06:47.495-0400    INFO    registrar/registrar.go:282      Ending Registrar
    2020-08-09T13:06:47.501-0400    INFO    [monitoring]    log/log.go:149  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":490,"time":{"ms":495}},"total":{"ticks":1510,"time":{"ms":1519},"value":1510},"user":{"ticks":1020,"time":{"ms":1024}}},"info":{"ephemeral_id":"cf0eecef-2c56-4978-9baf-0a3ff454e68b","uptime":{"ms":155366}},"memstats":{"gc_next":4993408,"memory_alloc":5979264,"memory_total":151842768,"rss":19914752}},"filebeat":{"events":{"active":25,"added":32,"done":7},"harvester":{"closed":4,"open_files":0,"running":0,"started":4},"input":{"log":{"files":{"truncated":2}}}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":21,"batches":112,"failed":773,"total":794},"type":"kafka"},"outputs":{"kafka":{"bytes_read":18800,"bytes_write":119657}},"pipeline":{"clients":0,"events":{"active":21,"filtered":11,"published":21,"retry":789,"total":32}}},"registrar":{"states":{"cleanup":1,"current":4,"update":7},"writes":{"success":8,"total":8}},"system":{"cpu":{"cores":2},"load":{"1":0.59,"15":0.46,"5":0.45,"norm":{"1":0.295,"15":0.23,"5":0.225}}}}}}
    2020-08-09T13:06:47.501-0400    INFO    [monitoring]    log/log.go:150  Uptime: 2m35.36951196s
    2020-08-09T13:06:47.501-0400    INFO    [monitoring]    log/log.go:127  Stopping metrics logging.
    2020-08-09T13:06:47.501-0400    INFO    instance/beat.go:373    filebeat stopped.

There's nothing there that suggests a problem. It's started, connected to kafka, then something has asked it to stop running.

Yes I suspect something with Kafka as output, if I comment it out and enable Elasticsearch as output its working fine. Only with Kafka its failing and also failed to start service message is coming while restarting .

Well what do the logs look like when you enable it?

You mean enable Elasticsearch ?

After enabling Elasticsearch as output

        2020-08-10T04:19:00.098-0400    INFO    instance/beat.go:544    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
    2020-08-10T04:19:00.101-0400    INFO    instance/beat.go:551    Beat UUID: e638f53f-84fc-47ea-875d-354872f82bef
    2020-08-10T04:19:00.101-0400    INFO    [seccomp]       seccomp/seccomp.go:93   Syscall filter could not be installed because the kernel does not support seccomp
    2020-08-10T04:19:00.101-0400    INFO    [beat]  instance/beat.go:768    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "e638f53f-84fc-47ea-875d-354872f82bef"}}}
    2020-08-10T04:19:00.101-0400    INFO    [beat]  instance/beat.go:777    Build info      {"system_info": {"build": {"commit": "34b4e2cc75fbbee5e7149f3916de72fb8892d070", "libbeat": "6.4.0", "time": "2018-08-17T22:20:20.000Z", "version": "6.4.0"}}}
    2020-08-10T04:19:00.101-0400    INFO    [beat]  instance/beat.go:780    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
    2020-08-10T04:19:00.105-0400    INFO    [beat]  instance/beat.go:784    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-08-05T09:20:14-04:00","containerized":true,"hostname":"rhel74","ips":["127.0.0.1/8","::1/128","100.64.24.211/22","2607:f2b1:f000:24:250:56ff:fe1e:a275/64","fe80::250:56ff:fe1e:a275/64"],"kernel_version":"3.10.0-693.17.1.el7.x86_64","mac_addresses":["00:50:56:1e:a2:75"],"os":{"family":"","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.4 (Maipo)","major":7,"minor":4,"patch":0,"codename":"Maipo"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"05136768af6046b0b100160cf1bdd62e"}}}
    2020-08-10T04:19:00.106-0400    INFO    [beat]  instance/beat.go:813    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 17774, "ppid": 1, "seccomp": {"mode":"disabled"}, "start_time": "2020-08-10T04:18:59.580-0400"}}}
    2020-08-10T04:19:00.106-0400    INFO    instance/beat.go:273    Setup Beat: filebeat; Version: 6.4.0
    2020-08-10T04:19:00.107-0400    INFO    elasticsearch/client.go:163     Elasticsearch url: http://100.102.128.20:9200
    2020-08-10T04:19:00.108-0400    INFO    pipeline/module.go:98   Beat name: rhel74
    2020-08-10T04:19:00.108-0400    WARN    [cfgwarn]       beater/filebeat.go:78   DEPRECATED: prospectors are deprecated, Use `inputs` instead. Will be removed in version: 7.0.0
    2020-08-10T04:19:00.109-0400    INFO    [monitoring]    log/log.go:114  Starting metrics logging every 30s
    2020-08-10T04:19:00.109-0400    INFO    instance/beat.go:367    filebeat start running.
    2020-08-10T04:19:00.110-0400    INFO    registrar/registrar.go:134      Loading registrar data from /var/lib/filebeat/registry
    2020-08-10T04:19:00.110-0400    INFO    registrar/registrar.go:141      States Loaded from registrar: 4
    2020-08-10T04:19:00.110-0400    INFO    crawler/crawler.go:72   Loading Inputs: 1
    2020-08-10T04:19:00.110-0400    WARN    [cfgwarn]       input/config.go:42      DEPRECATED: input_type input config is deprecated. Use type instead. Will be removed in version: 6.0.0
    2020-08-10T04:19:00.111-0400    INFO    log/input.go:138        Configured paths: [/var/log/dmesg* /var/log/messages*]
    2020-08-10T04:19:00.111-0400    INFO    input/input.go:114      Starting input of type: log; ID: 8669900041677695518
    2020-08-10T04:19:00.111-0400    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1
    2020-08-10T04:19:00.122-0400    INFO    log/harvester.go:251    Harvester started for file: /var/log/dmesg
    2020-08-10T04:19:00.124-0400    INFO    log/harvester.go:251    Harvester started for file: /var/log/messages
    2020-08-10T04:19:01.642-0400    INFO    elasticsearch/client.go:708     Connected to Elasticsearch version 6.6.0
    2020-08-10T04:19:01.905-0400    INFO    template/load.go:129    Template already exists and will not be overwritten.

No, kafka, that's the one with the issue?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.