FileBeat Propector issue

SJN8 · August 22, 2018, 6:31am

Hi All,

i am sending events of tomee via filebeat to kafka but getting below exceptions ,
can someone help me understanding it , looks like some configuration needs to be updated .
ticker time , crawler , prospector outlet .

2018/08/21 06:24:29.966584 crawler.go:109: INFO Stopping Crawler
2018/08/21 06:24:29.966687 crawler.go:119: INFO Stopping 20 prospectors
2018/08/21 06:24:29.966736 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.966771 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.966780 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.966834 prospector.go:138: INFO Stopping Prospector: 9481811630750621847
2018/08/21 06:24:29.966737 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.966873 prospector.go:138: INFO Stopping Prospector: 16250081816100377070
2018/08/21 06:24:29.966741 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.966899 prospector.go:138: INFO Stopping Prospector: 14995750911456661710
2018/08/21 06:24:29.966846 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.966949 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966961 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966749 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.966998 prospector.go:138: INFO Stopping Prospector: 10884091510794477696
2018/08/21 06:24:29.967022 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966964 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967064 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967111 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967009 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967138 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967155 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967213 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967258 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966748 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967297 prospector.go:138: INFO Stopping Prospector: 9524210098540359837
2018/08/21 06:24:29.966750 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967314 prospector.go:138: INFO Stopping Prospector: 2266296135282362560
2018/08/21 06:24:29.967320 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967298 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966762 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967366 prospector.go:138: INFO Stopping Prospector: 5071786216812631489
2018/08/21 06:24:29.967383 prospector.go:410: INFO Scan aborted because prospector stopped.
2018/08/21 06:24:29.967399 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967404 harvester.go:215: INFO Harvester started for file: /var/log//logs/catalina.out-20180814-20180816
2018/08/21 06:24:29.967445 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966758 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967494 prospector.go:138: INFO Stopping Prospector: 9476847815990620746
2018/08/21 06:24:29.967505 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.967513 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.967524 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.967533 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.967567 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967608 prospector.go:138: INFO Stopping Prospector: 4848512172117451415
2018/08/21 06:24:29.967614 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967625 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967638 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967515 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.967676 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.967676 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966770 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967713 prospector.go:138: INFO Stopping Prospector: 4870614798961431722
2018/08/21 06:24:29.967723 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967712 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966728 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967742 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966916 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.966763 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967788 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180814-20180816. Closing.
2018/08/21 06:24:29.967798 prospector.go:138: INFO Stopping Prospector: 9402531377110145866
2018/08/21 06:24:29.967802 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180820. Closing.
2018/08/21 06:24:29.967758 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967807 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180810-20180812-20180814. Closing.
2018/08/21 06:24:29.967846 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180810-20180812-20180814-20180816. Closing.
2018/08/21 06:24:29.967849 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180810-20180812-20180814-20180816-20180818. Closing.

SJN8 · August 22, 2018, 6:33am

2018/08/21 06:24:29.967846 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180810-20180812-20180814-20180816. Closing.
2018/08/21 06:24:29.967849 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180810-20180812-20180814-20180816-20180818. Closing.
2018/08/21 06:24:29.966988 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967783 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180811-20180813. Closing.
2018/08/21 06:24:29.967879 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180816. Closing.
2018/08/21 06:24:29.966770 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967867 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180816-20180818. Closing.
2018/08/21 06:24:29.967905 prospector.go:138: INFO Stopping Prospector: 2644950983898890120
2018/08/21 06:24:29.967859 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180817. Closing.
2018/08/21 06:24:29.967911 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180809-20180811-20180813-20180815-20180817-20180819. Closing.
2018/08/21 06:24:29.967727 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.967919 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.967991 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.967788 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968006 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967824 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180815-20180817-20180819. Closing.
2018/08/21 06:24:29.968003 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.967988 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968072 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967830 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967835 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180809-20180811-20180813. Closing.
2018/08/21 06:24:29.967836 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180818. Closing.
2018/08/21 06:24:29.968148 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967847 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out. Closing.
2018/08/21 06:24:29.967880 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180817-20180819. Closing.
2018/08/21 06:24:29.966762 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.968206 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968214 prospector.go:138: INFO Stopping Prospector: 649985330363606594
2018/08/21 06:24:29.966771 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.967886 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967849 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967727 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968257 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968274 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968291 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968295 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.968276 harvester.go:236: INFO Reader was closed: /var/log/logs/Server_HttpMonitoring_2018-08-13_05_0.log. Closing.
2018/08/21 06:24:29.968329 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.967715 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968339 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.967919 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180815-20180817. Closing.
2018/08/21 06:24:29.967749 prospector.go:138: INFO Stopping Prospector: 16207921420499893675
2018/08/21 06:24:29.967920 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180810-20180812. Closing.
2018/08/21 06:24:29.967942 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967779 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180808-20180810-20180812. Closing.
2018/08/21 06:24:29.967971 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.967991 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.966768 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.968463 prospector.go:138: INFO Stopping Prospector: 12971859759279375250
2018/08/21 06:24:29.968495 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968507 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.968520 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.968528 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.968536 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.967825 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180808-20180810-20180812-20180814. Closing.
2018/08/21 06:24:29.968241 prospector.go:138: INFO Stopping Prospector: 17204249658926138838
2018/08/21 06:24:29.967761 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.967901 harvester.go:236: INFO Reader was closed: /var/log//logs/catalina.out-20180811-20180813-20180815-20180817-20180819. Closing.
2018/08/21 06:24:29.966749 prospector.go:138: INFO Stopping Prospector: 11641839350454898397
2018/08/21 06:24:29.966792 prospector.go:138: INFO Stopping Prospector: 7607415600068520164

SJN8 · August 22, 2018, 6:33am

2018/08/21 06:24:29.968644 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.968655 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.968657 harvester.go:236: INFO Reader was closed: /var/log/logs/Server_SLF4J_BOS_2018-08-15_0.log. Closing.
2018/08/21 06:24:29.968376 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.968699 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968687 harvester.go:236: INFO Reader was closed: /var/log/logs/Server_SLF4J_BOS_2018-08-09_0.log. Closing.
2018/08/21 06:24:29.968664 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.968702 harvester.go:236: INFO Reader was closed: /var/log/logs/Server_SLF4J_BOS_2018-08-08_0.log. Closing.
2018/08/21 06:24:29.968703 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.967937 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.968743 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.968548 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.968754 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968760 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.968729 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.968750 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.968795 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.968804 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.968715 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.968883 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.968964 logger.go:29: DBG [publish] client: cancelled 0 events
018/08/21 06:24:29.969396 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.969410 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.969418 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.969541 prospector.go:410: INFO Scan aborted because prospector stopped.
2018/08/21 06:24:29.969597 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.969607 prospector.go:138: INFO Stopping Prospector: 5108894301380658126
2018/08/21 06:24:29.969620 harvester.go:215: INFO Harvester started for file: /var/log//logs/Server_KernelMonitoring_2018-08-10_05_0.log
2018/08/21 06:24:29.969637 forwarder.go:35: INFO Prospector outlet closed
2018/08/21 06:24:29.969641 harvester.go:236: INFO Reader was closed: /var/log//logs/Server_KernelMonitoring_2018-08-07_12_0.log. Closing.
2018/08/21 06:24:29.972130 prospector.go:410: INFO Scan aborted because prospector stopped.
2018/08/21 06:24:29.972175 prospector.go:121: INFO Prospector ticker stopped
2018/08/21 06:24:29.972184 prospector.go:138: INFO Stopping Prospector: 11496662325398572520
2018/08/21 06:24:29.972205 harvester.go:236: INFO Reader was closed: /var/log/logs/Server_KernelMonitoring_2018-08-17_19_0.log. Closing.
2018/08/21 06:24:29.972213 harvester.go:215: INFO Harvester started for file: /var/log/logs/Server_KernelMonitoring_2018-08-07_08_0.log
2018/08/21 06:24:29.976157 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.976175 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.976188 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:29.997936 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:29.997954 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:29.997964 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:30.015878 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:30.015895 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:30.015904 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:30.017812 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:30.017829 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:30.017838 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:30.048195 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:30.048212 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:30.048221 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:24:30.054319 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:24:30.054335 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:24:30.054344 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:29:34.980459 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:29:34.980476 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:29:34.980488 logger.go:29: DBG [publish] client: cancelled 0 events
2018/08/21 06:29:34.980506 crawler.go:135: INFO Crawler stopped
2018/08/21 06:29:34.980530 registrar.go:210: INFO Stopping Registrar
2018/08/21 06:29:34.980603 logger.go:18: DBG [publish] client: closing acker
2018/08/21 06:29:34.980632 logger.go:18: DBG [publish] client: done closing acker
2018/08/21 06:29:34.980644 logger.go:29: DBG [publish] client: cancelled 0 events

Thanks,
Shashank

pierhugues · August 22, 2018, 1:35pm

@SJN8 I did a quick look, the "prospector outlet closed" is triggered when we update the state of the file, to help me debug this I will need a few more information:

Filebeat version
You filebeat configuration.

Thanks.

SJN8 · August 23, 2018, 6:27am

Hi pier ,

It is resolved by deleting registry file as there were duplicate records for prospector .

Thanks for your help also i have one more query ,

as it is not a good practice to delete registry file always .
i change my filebeat configuration everyday to update send daily logs to new kafka topic .

So do we have any option in filebeat like logstash to config reload automatic so i dont need to restart Filebeat manually if configuration changes , it should pick new configuration automatically .

let me know if i need to provide more information .

Thanks,
Shashank

pierhugues · August 23, 2018, 3:52pm

@SJN8 Not everything can be reloaded in the beats configuration, but the prospectors are reloadable.

But I believe you can use the following to dynamically create the topic name based on the current date.

topic: "logs-%{+yyyy.MM.dd}"

SJN8 · August 27, 2018, 10:25am

@pierhugues :- Thanks for sharing this for filebeat configuration , it is working fine for me .

Now one query on Logstash which is taking input from kafka , the same topic where filebeat would be sending events ,

as above i tried to make below changes in my logstash config input section ,

input {
kafka {
bootstrap_servers => "serverA , serverB , serverC"
security_protocol => "SASL_PLAINTEXT"
sasl_mechanism => "PLAIN"
jaas_path => "/etc/logstash/jaas.conf"
topics => ["logs-%{+YYYY.MM.dd}"]
}
}

but above changes are not working , below are exception ,

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/file.conf --config.reload.automatic
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[ERROR] 2018-08-27 12:02:31.512 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 10, column 22 (byte 379) after # Kafka input doc\n# https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html\n\ninput {\n\tkafka {\n\t\tbootstrap_servers => "serA , serB , serC"\n\t\tsecurity_protocol => "SASL_PLAINTEXT"\n\t\tsasl_mechanism => "PLAIN"\n\t\tjaas_path => "/etc/logstash/jaas.conf"\n\t\ttopics => [PEOINT7-", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

can you help what i am missing here ?

Thanks,
Shashank

pierhugues · August 27, 2018, 3:22pm

This look like a malformed logstash config around line 10?

SJN8 · August 28, 2018, 7:18am

@pierhugues , configuration are good , it is not resolving metadata , like it did in Filebeat ,

i am getting below error ,

[WARN ] 2018-08-28 09:12:47.568 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 4 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[INFO ] 2018-08-28 09:12:47.570 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] AbstractCoordinator - Discovered coordinator vkafka01dsy:9092 (id: 2147483647 rack: null) for group logstash.
[INFO ] 2018-08-28 09:12:47.572 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] ConsumerCoordinator - Revoking previously assigned partitions [] for group logstash
[INFO ] 2018-08-28 09:12:47.573 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] AbstractCoordinator - (Re-)joining group logstash
[WARN ] 2018-08-28 09:12:47.673 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 9 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:47.774 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 11 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:47.876 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 12 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:47.976 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 13 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:48.076 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 14 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:48.177 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 15 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:48.278 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 16 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:48.379 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 17 : {logs-%{+YYYY-MM-dd}=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:12:48.480 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/

[WARN ] 2018-08-28 09:14:00.234 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 4 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}
[INFO ] 2018-08-28 09:14:00.236 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] AbstractCoordinator - Discovered coordinator vkafka01dsy:9092 (id: 2147483647 rack: null) for group logstash.
[INFO ] 2018-08-28 09:14:00.239 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] ConsumerCoordinator - Revoking previously assigned partitions [] for group logstash
[INFO ] 2018-08-28 09:14:00.239 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] AbstractCoordinator - (Re-)joining group logstash
[WARN ] 2018-08-28 09:14:00.340 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 10 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:14:00.441 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 11 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:14:00.542 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 12 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:14:00.643 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 13 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:14:00.744 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 14 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}
[WARN ] 2018-08-28 09:14:00.845 [Ruby-0-Thread-15: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.0.2/lib/logstash/inputs/kafka.rb:241] NetworkClient - Error while fetching metadata with correlation id 15 : {[logs-%{+YYYY-MM-dd}]=UNKNOWN_TOPIC_OR_PARTITION}

it is not resolving %{+YYYY.MM.dd} in current date .

pierhugues · August 28, 2018, 12:26pm

@SJN8 I am afraid that logstash doesn't support dynamic topic name, I didn't know that

Can I ask why do you want to have multiple topics for kafka?

SJN8 · August 28, 2018, 1:02pm

@pierhugues :- ohh okay , multiple topic is for managing the space like i can delete old data so i am implementing day wise topics in kafka and day wise index in elastic search .

i guess if metadata is supported in Filebeat and elastic search then should be in logstash , looks like i am missing some configuration .

please help in this or let know who might be knowing in community will add that person in this thread

Thanks,
Shashank

system · September 25, 2018, 1:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.