Connect Filebeat to logstash

Revathy · July 6, 2020, 4:23pm

Hi,

I am trying to setup filebeat to logstash and get below errors at filebeat and logstash end:

filebeat; Version: 7.7.0
logstash "number" : "7.8.0"

Modified /etc/filebeat/filebeat.yml:
enabled: true
paths:
commented output.elasticsearch
uncommented output.logstash and added hosts: ["hostname:5044"]
Modified /etc/logstash/conf.d/beats_elasticsearch.conf:
input {
beats {
port => 5044
}
}

#filter {
#}

output {
elasticsearch {
hosts => ["hostname:9200"]
}
}

I started filebeat and got below error:

2020-07-06T08:51:23.912-0700 ERROR [publisher_pipeline_output] pipeline/output.go:106 Failed to connect to backoff(elasticsearch(http://hostname:5044)): Get http://hostname:5044: dial tcp ip_address:5044: connect: connection refused

Started logstash and its log below:

[INFO ] 2020-07-06 09:00:20.562 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2020-07-06 09:00:20.835 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-07-06 09:00:45.266 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: x.x.x.x:5044, remote: x.x.x.x:53628] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
[WARN ] 2020-07-06 09:00:45.267 [nioEventLoopGroup-2-2] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71

Please explain what else I should do.

Started filebeat and logstash as:
sudo /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/beats_elasticsearch.conf

Thanks

Badger · July 6, 2020, 5:02pm

Failed to connect to elasticsearch? Are you sure you commented out the elasticsearch output and uncommented the logstash output in the filebeat configuration?

Revathy · July 6, 2020, 7:14pm

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
' # Array of hosts to connect to.
'# hosts: ["localhost:9200"]
'# Protocol - either .
'#protocol: "https"

'# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"

I have commented these lines in /etc/filebeat/filebeat.yml. Is there any other file that I need to modify? Don't know why the log is showing elasticsearch

Revathy · July 6, 2020, 7:59pm

Filebeat log:

2020-07-06T08:51:17.846-0700 INFO instance/beat.go:621 Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/fi
lebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
2020-07-06T08:51:18.004-0700 INFO instance/beat.go:629 Beat ID: b66a32ee-cd70-4b8e-a9bc-aa01963856c5
2020-07-06T08:51:18.009-0700 INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp
2020-07-06T08:51:18.009-0700 INFO [beat] instance/beat.go:957 Beat info {“system_info”: {“beat”: {“path”: {“config”: “/usr/share/filebeat/bin”, “data
“: “/usr/share/filebeat/bin/data”, “home”: “/usr/share/filebeat/bin”, “logs”: “/usr/share/filebeat/bin/logs”}, “type”: “filebeat”, “uuid”: “b66a32ee-cd70-4b8e-a9bc-a
a01963856c5”}}}
2020-07-06T08:51:18.009-0700 INFO [beat] instance/beat.go:966 Build info {“system_info”: {“build”: {“commit”: “5e69e25b920e3d93bec76a09a31da3ab35a5560
7”, “libbeat”: “7.7.0”, “time”: “2020-05-12T00:53:16.000Z”, “version”: “7.7.0”}}}
2020-07-06T08:51:18.009-0700 INFO [beat] instance/beat.go:969 Go runtime info {“system_info”: {“go”: {“os”:“linux”,“arch”:“amd64”,“max_procs”:4,“version”:”
go1.13.9"}}}
2020-07-06T08:51:18.010-0700 INFO [beat] instance/beat.go:973 Host info {“system_info”: {“host”: {“architecture”:“x86_64",“boot_time”:“2020-06-09T04:
40:03-07:00",“containerized”:false,“name”:“”,“ip”:[“127.0.0.1/8”,“10.102.153.239/20"],“kernel_version”:“2.6.32-754.28.1.el6.x86_64",
“mac”:[“00:50:56:a0:1f:61"],“os”:{“family”:“”,“platform”:“ol”,“name”:“Oracle Linux Server”,“version”:“6.10”,“major”:6,“minor”:10,“patch”:0},“timezone”:“PDT”,“timezon
e_offset_sec”:-25200,“id”:“6cd1ecbb634d20276a322ec00000000f”}}}
2020-07-06T08:51:18.010-0700 INFO [beat] instance/beat.go:1002 Process info {“system_info”: {“process”: {“capabilities”: {“inheritable”:null,“permitted”:
[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_
raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_confi
g”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”,“38”,“39",“40”,“41",“42”,“43
“,”44",“45”,“46",“47”,“48",“49”,“50",“51”,“52",“53”,“54",“55”,“56",“57”,“58",“59”,“60",“61”,“62",“63”],“effective”:[“chown”,“dac_override”,“dac_read_search”,“fowner”
,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux_immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawi
o”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin”,“sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,”
setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake_alarm”,“block_suspend”,“audit_read”,“38”,“39",“40”,“41",“42”,“43",“44”,“45",“46”,“47",“48”,“49",“50”,“51",“52”,“53
“,”54",“55”,“56",“57”,“58",“59”,“60",“61”,“62",“63”],“bounding”:[“chown”,“dac_override”,“dac_read_search”,“fowner”,“fsetid”,“kill”,“setgid”,“setuid”,“setpcap”,“linux
immutable”,“net_bind_service”,“net_broadcast”,“net_admin”,“net_raw”,“ipc_lock”,“ipc_owner”,“sys_module”,“sys_rawio”,“sys_chroot”,“sys_ptrace”,“sys_pacct”,“sys_admin
“,”sys_boot”,“sys_nice”,“sys_resource”,“sys_time”,“sys_tty_config”,“mknod”,“lease”,“audit_write”,“audit_control”,“setfcap”,“mac_override”,“mac_admin”,“syslog”,“wake
alarm”,“block_suspend”,“audit_read”,“38",“39”,“40",“41”,“42",“43”,“44",“45”,“46",“47”,“48",“49”,“50",“51”,“52",“53”,“54",“55”,“56",“57”,“58",“59”,“60",“61”,“62",“63”
],“ambient”:null}, “cwd”: “/var/log/filebeat”, “exe”: “/usr/share/filebeat/bin/filebeat”, “name”: “filebeat”, “pid”: 8941, “ppid”: 8940, “seccomp”: {“mode”:“”}, “sta
rt_time”: “2020-07-06T08:51:17.070-0700”}}}
2020-07-06T08:51:18.011-0700 INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.7.0
2020-07-06T08:51:18.011-0700 INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to ‘filebeat-7.7.0’ as ILM is enabled.
2020-07-06T08:51:18.011-0700 INFO eslegclient/connection.go:84 elasticsearch url: :5044
2020-07-06T08:51:18.011-0700 INFO [publisher] pipeline/module.go:110 Beat name:
2020-07-06T08:51:18.011-0700 ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-07-06T08:51:18.011-0700 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-07-06T08:51:18.011-0700 INFO instance/beat.go:438 filebeat start running.
2020-07-06T08:51:18.012-0700 INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/bin/data/registry/filebeat
2020-07-06T08:51:18.012-0700 INFO registrar/migrate.go:112 Initialize registry meta file
2020-07-06T08:51:18.022-0700 INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/bin/data/registry/filebeat/data.json. Creat
ing a new registry file.
2020-07-06T08:51:18.039-0700 INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/bin/data/registry/filebeat/data.json
2020-07-06T08:51:18.039-0700 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-07-06T08:51:18.039-0700 INFO beater/crawler.go:73 Loading Inputs: 1
2020-07-06T08:51:18.040-0700 INFO log/input.go:152 Configured paths: []
2020-07-06T08:51:18.040-0700 INFO input/input.go:114 Starting input of type: log; ID: 13172526264431030025
2020-07-06T08:51:18.040-0700 INFO beater/crawler.go:105 Loading and starting Inputs completed. Enabled inputs: 1
2020-07-06T08:51:18.040-0700 INFO cfgfile/reload.go:175 Config reloader started
2020-07-06T08:51:18.040-0700 INFO cfgfile/reload.go:235 Loading of config files completed.
2020-07-06T08:51:18.044-0700 INFO log/harvester.go:297 Harvester started for file:
2020-07-06T08:51:21.008-0700 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-07-06T08:51:22.009-0700 INFO [publisher_pipeline_output] pipeline/output.go:101 Connecting to backoff(elasticsearch(hostname:5044))
2020-07-06T08:51:23.912-0700 ERROR [publisher_pipeline_output] pipeline/output.go:106 Failed to connect to backoff(elasticsearch(hostname:5044)): Get x.x.x.x:5044: dial tcp x.x.x.x:5044: connect: connection refused
2020-07-06T08:51:45.927-0700 INFO [publisher] pipeline/retry.go:196 retryer: send unwait-signal to consumer
2020-07-06T08:51:45.927-0700 INFO [publisher] pipeline/retry.go:198 done
2020-07-06T08:51:45.927-0700 INFO [publisher] pipeline/retry.go:173 retryer: send wait signal to consumer
2020-07-06T08:51:45.927-0700 INFO [publisher] pipeline/retry.go:175 done
2020-07-06T08:51:48.013-0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {“monitoring”: {“metrics”: {“beat”:{“cpu”:{“system”:{“ticks”:40,“time”:{“ms”:47}},“total”:{“ticks”:150,“time”:{“ms”:165},“value”:150},“user”:{“ticks”:110,“time”:{“ms”:118}}},“handles”:{“limit”:{“hard”:32768,“soft”:32768},“open”:11},“info”:{“ephemeral_id”:“119506af-cd7d-4c96-b0cf-7818ead71d00",“uptime”:{“ms”:30243}},“memstats”:{“gc_next”:15232448,“memory_alloc”:10777104,“memory_total”:22170408,“rss”:44703744},“runtime”:{“goroutines”:27}},“filebeat”:{“events”:{“active”:1103,“added”:1104,“done”:1},“harvester”:{“files”:{“d68421b0-d1e7-4204-9819-89899f28259f”:{“last_event_published_time”:“2020-07-06T08:51:21.033Z”,“last_event_timestamp”:“2020-07-06T08:51:21.033Z”,“name”:“”,“read_offset”:121278,“size”:121278,“start_time”:“2020-07-06T08:51:18.040Z”}},“open_files”:1,“running”:1,“started”:1}},“libbeat”:{“config”:{“module”:{“running”:0},“reloads”:1,“scans”:1},“output”:{“type”:“elasticsearch”},“pipeline”:{“clients”:1,“events”:{“active”:1070,“filtered”:34,“published”:1070,“retry”:100,“total”:1104}}},“registrar”:{“states”:{“current”:1,“update”:1},“writes”:{“success”:2,“total”:2}},“system”:{“cpu”:{“cores”:4},“load”:{“1”:1.09,“15":1.01,“5”:1.06,“norm”:{“1”:0.2725,“15":0.2525,“5”:0.265}}}}}}
2020-07-06T08:52:12.514-0700 ERROR [publisher_pipeline_output] pipeline/output.go:106 Failed to connect to backoff(elasticsearch(http://:5044)): Get http://:5044: dial tcp 10.102.130.7:5044: connect: connection refused
2020-07-06T08:52:12.514-0700 INFO [publisher_pipeline_output] pipeline/output.go:99 Attempting to reconnect to backoff(elasticsearch(http://:5044)) with 5 reconnect attempt(s)

Revathy · July 6, 2020, 8:06pm

logstash log:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-07-06 09:00:16.453 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-07-06 09:00:16.471 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2020-07-06 09:00:16.754 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-07-06 09:00:16.758 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.8.0", "jruby.version"=>"jruby 9.2.11.1 (2.5.7) 2020-03-25 b1f55b1a40 Java HotSpot(TM) 64-Bit Server VM 25.212-b10 on 1.8.0_212-b10 +indy +jit [linux-x86_64]"}
[INFO ] 2020-07-06 09:00:16.777 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"bb727ea0-db3d-42f6-9533-e542f4789276", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2020-07-06 09:00:18.282 [Converge PipelineAction::Create] Reflections - Reflections took 39 ms to scan 1 urls, producing 21 keys and 41 values
[INFO ] 2020-07-06 09:00:18.972 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://:9200/, http://:9200/, http:// .net:9200/]}}
[WARN ] 2020-07-06 09:00:19.125 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http:// :9200/"}
[INFO ] 2020-07-06 09:00:19.288 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2020-07-06 09:00:19.290 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[WARN ] 2020-07-06 09:00:19.339 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http:// :9200/"}
[WARN ] 2020-07-06 09:00:19.399 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http:// :9200/"}
[INFO ] 2020-07-06 09:00:19.460 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//:9200", "//:9200", "//:9200"]}
[INFO ] 2020-07-06 09:00:19.498 [Ruby-0-Thread-6: :1] elasticsearch - Using default mapping template
[INFO ] 2020-07-06 09:00:19.552 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/beats_elasticsearch.conf"], :thread=>"#<Thread:0x18b6891f run>"}
[INFO ] 2020-07-06 09:00:19.598 [Ruby-0-Thread-6: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[INFO ] 2020-07-06 09:00:20.358 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2020-07-06 09:00:20.386 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2020-07-06 09:00:20.466 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[INFO ] 2020-07-06 09:00:20.562 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2020-07-06 09:00:20.835 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-07-06 09:00:45.266 [defaultEventExecutorGroup-4-1] BeatsHandler - [local: x.x.x.x.:5044, remote: x.x.x.x:53628] Handling exception: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
[WARN ] 2020-07-06 09:00:45.267 [nioEventLoopGroup-2-2] DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 71
at org.logstash.beats.Protocol.version(Protocol.java:22) ~[logstash-input-beats-6.0.9.jar:?]
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:62) ~[logstash-input-beats-6.0.9.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 8 more

Neha_Wagholikar · July 7, 2020, 7:27am

Hi,

1] Please check both version of logstash and filebeat and their compatibility
2] If its linux vm you are trying , check once firewall rules also check if its getting connected using telnet

Revathy · July 7, 2020, 10:13am

Hi Neha,

nc -zv 5044 shows successful and filebeat 7.7.0 is compatible with logstash 7.8.0.
Checking for firewall

Revathy · July 8, 2020, 6:23am

Worked fine after upgrading filebeat to 7.8.0

Neha_Wagholikar · July 13, 2020, 6:15am

Great!

system · August 10, 2020, 6:15am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.