Hi,
I have 2 servers where OASIS logs are getting monitored with filebeat.
I want them to be saved in the same index.
Should I port themm from different ports( Server 1 : 5044, server 2 : 5045)?
Or can i use the same port for both servers?
If I use different ports , can i map them to the same index?
Kindly help me out.
Hi!
If I understand your question, you can send events with Filebeat agents to the same port.
Since you run different agents, events will have the information of the host where the agent runs.
Hi @ChrsMark,
I did try that but my data only comes in for one Server logs. I had them both come into port 5044.
Hi,
Which's service are those ports? Could you explain the exact setup please? Are you using Filebeat to ship to Elasticsearch or Logstash?
Also you can check in the logs of Filebeats agents and see what is going wrong?
Regards.
Hi @ChrsMark, Apologies, If I wasn't clear,
So my set up goes like this I have 2 servers where i installed filebeat 7.2.0, and they are sending an application log to my logstash 7.2.0 which is a different server. the logstash pushes data to my ES instance later.
This is my logstash conf file.
input { beats { port => 5044 ssl => false } } output { elasticsearch { hosts => ["10.16.5.24:9200"] index => "oasis" } stdout { codec => rubydebug } }
I have configured logstash output as
output.logstash:
# The Logstash hosts
hosts: ["110.226.51.22:5044"]
in both application servers.
But i see only data from one of them passing when i check my ES index.
Thank you for providing the information!
Now we need to see what is the problem with Filebeat that does not send events. Some checks to do:
Regards.
@ChrsMark,
I do not find any errors or warnings in the filebeat logs.
As far as I can see, logstash also has general warnings.
[2019-12-11T06:41:53,380][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2019-12-11T06:41:58,598][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>30, "name"=>"[oasislogs]<beats", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.0-java/lib/logstash/inputs/beats.rb:204:in run'"}, {"thread_id"=>22, "name"=>"[oasislogs]>worker0", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>23, "name"=>"[oasislogs]>worker1", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>24, "name"=>"[oasislogs]>worker2", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>25, "name"=>"[oasislogs]>worker3", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>26, "name"=>"[oasislogs]>worker4", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>27, "name"=>"[oasislogs]>worker5", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>28, "name"=>"[oasislogs]>worker6", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}, {"thread_id"=>29, "name"=>"[oasislogs]>worker7", "current_call"=>"[...]/logstash-core/lib/logstash/java_pipeline.rb:239:in block in start_workers'"}]}}
[2019-12-11T06:41:58,602][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
[2019-12-11T06:41:59,876][INFO ][logstash.javapipeline ] Pipeline terminated {"pipeline.id"=>"oasislogs"} [2019-12-11T06:42:00,572][INFO ][logstash.runner ] Logstash shut down. [2019-12-11T06:42:10,757][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-12-11T06:42:14,859][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.16.5.24:9200/]}} [2019-12-11T06:42:14,997][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.16.5.24:9200/"}
[2019-12-11T06:42:15,033][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2019-12-11T06:42:15,035][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type` event field won't be used to determine the document _type {:es_version=>7}
[2019-12-11T06:42:15,058][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.16.5.24:9200"]}
[2019-12-11T06:42:15,111][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-12-11T06:42:15,135][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-12-11T06:42:15,138][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"oasislogs", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x67dc8344 run>"}
[2019-12-11T06:42:15,176][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-12-11T06:42:15,376][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-12-11T06:42:15,383][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"oasislogs"}
[2019-12-11T06:42:15,454][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:oasislogs], :non_running_pipelines=>}
[2019-12-11T06:42:15,467][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-12-11T06:42:15,702][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
And now,
the only server which was pushing data, stopped and no other data is getting uploaded as well. I tried to change the output to a file, which is of no data as well.
Lastly, I'm able to telnet to the logstash server from both the app servers.
Please let me know of what I can try out to resolve this.
Could you restart the Filebeat's, with filebeat -e -d "*" so as to have verbose output, and follow their logs?
We need to know why Filebeats are not sending events.
@ChrsMark Apologies for the delayed response,
below is the logs,
D:\ELK\filebeat720\filebeat720>filebeat -e -d "*" 2019-12-15T23:41:27.642-0500 INFO instance/beat.go:606 Home path: [D:\E LK\filebeat720\filebeat720] Config path: [D:\ELK\filebeat720\filebeat720] Data p ath: [D:\ELK\filebeat720\filebeat720\data] Logs path: [D:\ELK\filebeat720\filebe at720\logs] 2019-12-15T23:41:27.642-0500 DEBUG [beat] instance/beat.go:658 Beat met adata path: D:\ELK\filebeat720\filebeat720\data\meta.json 2019-12-15T23:41:27.643-0500 INFO instance/beat.go:614 Beat ID: e733c6b b-f5bd-435d-a073-f9d14fd0ce81 2019-12-15T23:41:27.649-0500 DEBUG [filters] add_cloud_metadata/add_c loud_metadata.go:164 add_cloud_metadata: starting to fetch metadata, timeout= 3s 2019-12-15T23:41:27.653-0500 DEBUG [filters] add_cloud_metadata/add_c loud_metadata.go:196 add_cloud_metadata: received disposition for qcloud afte r 2.9297ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Ge t http://metadata.tencentyun.com/meta-data/placement/zone: dial tcp: lookup meta data.tencentyun.com: no such host, metadata={}] 2019-12-15T23:41:30.675-0500 DEBUG [filters] add_cloud_metadata/add_c loud_metadata.go:196 add_cloud_metadata: received disposition for aws after 3 .0097656s. result=[provider:aws, error=failed requesting aws metadata: Get http: //169.254.169.254/2014-02-25/dynamic/instance-identity/document: net/http: reque st canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), metadata={}] 2019-12-15T23:41:30.675-0500 DEBUG [filters] add_cloud_metadata/add_c loud_metadata.go:203 add_cloud_metadata: timed-out waiting for all responses 2019-12-15T23:41:30.675-0500 DEBUG [filters] add_cloud_metadata/add_c loud_metadata.go:167 add_cloud_metadata: fetchMetadata ran for 3.0097656s 2019-12-15T23:41:30.675-0500 INFO add_cloud_metadata/add_cloud_metadata.go :347 add_cloud_metadata: hosting provider type not detected. 2019-12-15T23:41:30.675-0500 DEBUG [processors] processors/processor.go: 93 Generated new processors: add_host_metadata=[netinfo.enabled=[false], ca che.ttl=[5m0s]], add_cloud_metadata=null 2019-12-15T23:41:30.675-0500 DEBUG [seccomp] seccomp/seccomp.go:88 Syscall filtering is only supported on Linux 2019-12-15T23:41:30.675-0500 INFO [beat] instance/beat.go:902 Beat inf o {"system_info": {"beat": {"path": {"config": "D:\\ELK\\filebeat720\\file beat720", "data": "D:\\ELK\\filebeat720\\filebeat720\\data", "home": "D:\\ELK\\f ilebeat720\\filebeat720", "logs": "D:\\ELK\\filebeat720\\filebeat720\\logs"}, "t ype": "filebeat", "uuid": "e733c6bb-f5bd-435d-a073-f9d14fd0ce81"}}} 2019-12-15T23:41:30.675-0500 INFO [beat] instance/beat.go:911 Build in fo {"system_info": {"build": {"commit": "9ba65d864ca37cd32c25b980dbb4020975 288fc0", "libbeat": "7.2.0", "time": "2019-06-20T15:05:29.000Z", "version": "7.2 .0"}}} 2019-12-15T23:41:30.675-0500 INFO [beat] instance/beat.go:914 Go runti me info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"ver sion":"go1.12.4"}}} 2019-12-15T23:41:30.675-0500 INFO [beat] instance/beat.go:918 Host inf o {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-12-0 6T03:20:49.53-05:00","name":"ICDWPOASIWEB03","ip":["10.16.52.11/24","::1/128","1 27.0.0.1/8"],"kernel_version":"6.1.7601.24520 (win7sp1_ldr_escrow.190828-1732)", "mac":["00:50:56:b6:f3:32"],"os":{"family":"windows","platform":"windows","name" :"Windows Server 2008 R2 Standard","version":"6.1","major":1,"minor":0,"patch":0 ,"build":"7601.24511"},"timezone":"EST","timezone_offset_sec":-18000,"id":"58943 449-451c-4dc6-9787-7382404bf221"}}} 2019-12-15T23:41:30.675-0500 INFO [beat] instance/beat.go:947 Process info {"system_info": {"process": {"cwd": "D:\\ELK\\filebeat720\\filebeat720", "exe": "D:\\ELK\\filebeat720\\filebeat-7.2.0-windows-x86_64\\filebeat.exe", "na me": "filebeat.exe", "pid": 5808, "ppid": 6064, "start_time": "2019-12-15T23:41: 27.597-0500"}}} 2019-12-15T23:41:30.675-0500 INFO instance/beat.go:292 Setup Beat: file beat; Version: 7.2.0 2019-12-15T23:41:30.675-0500 DEBUG [beat] instance/beat.go:318 Initiali zing output plugins 2019-12-15T23:41:30.675-0500 DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer 2019-12-15T23:41:30.675-0500 INFO [publisher] pipeline/module.go:97 Beat name: ICDWPOASIWEB03 2019-12-15T23:41:30.675-0500 INFO instance/beat.go:421 filebeat start r unning. 2019-12-15T23:41:30.675-0500 DEBUG [test] registrar/migrate.go:159 isFile(D:\ELK\filebeat720\filebeat720\data\registry) -> false 2019-12-15T23:41:30.675-0500 DEBUG [test] registrar/migrate.go:159 isFile() -> false 2019-12-15T23:41:30.675-0500 DEBUG [test] registrar/migrate.go:152 isDir(D:\ELK\filebeat720\filebeat720\data\registry\filebeat) -> true 2019-12-15T23:41:30.675-0500 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s 2019-12-15T23:41:30.675-0500 DEBUG [service] service/service_windows. go:72 Windows is interactive: true 2019-12-15T23:41:30.676-0500 DEBUG [test] registrar/migrate.go:159 isFile(D:\ELK\filebeat720\filebeat720\data\registry\filebeat\meta.json) -> true 2019-12-15T23:41:30.677-0500 DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found 2019-12-15T23:41:30.678-0500 DEBUG [registrar] registrar/registrar.go:1 25 Registry file set to: D:\ELK\filebeat720\filebeat720\data\registry\fileb eat\data.json 2019-12-15T23:41:30.678-0500 INFO registrar/registrar.go:145 Loading registrar data from D:\ELK\filebeat720\filebeat720\data\registry\filebeat\data.j son 2019-12-15T23:41:30.679-0500 INFO registrar/registrar.go:152 States L oaded from registrar: 0 2019-12-15T23:41:30.679-0500 WARN beater/filebeat.go:358 Filebeat is unab le to load the Ingest Node pipelines for the configured modules because the Elas ticsearch output is not configured/enabled. If you have already loaded the Inges t Node pipelines or are using Logstash pipelines, you can ignore this warning. 2019-12-15T23:41:30.680-0500 INFO crawler/crawler.go:72 Loading Inputs: 1 2019-12-15T23:41:30.680-0500 DEBUG [cfgfile] cfgfile/reload.go:134 Checking module configs from: D:\ELK\filebeat720\filebeat720/modules.d/*.yml 2019-12-15T23:41:30.679-0500 DEBUG [registrar] registrar/registrar.go:2 78 Starting Registrar 2019-12-15T23:41:30.681-0500 DEBUG [cfgfile] cfgfile/reload.go:148 Number of module configs found: 0 2019-12-15T23:41:30.681-0500 INFO crawler/crawler.go:106 Loading and star ting Inputs completed. Enabled inputs: 0 2019-12-15T23:41:30.681-0500 INFO cfgfile/reload.go:172 Config reloader started 2019-12-15T23:41:30.682-0500 DEBUG [cfgfile] cfgfile/reload.go:198 Scan for new config files 2019-12-15T23:41:30.682-0500 DEBUG [cfgfile] cfgfile/reload.go:217 Number of module configs found: 0 2019-12-15T23:41:30.682-0500 DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0 2019-12-15T23:41:30.683-0500 DEBUG [reload] cfgfile/list.go:80 Start list: 0, Stop list: 0 2019-12-15T23:41:30.683-0500 INFO cfgfile/reload.go:227 Loading of confi g files completed. 2019-12-15T23:42:00.692-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"syst em":{"ticks":31,"time":{"ms":31}},"total":{"ticks":62,"time":{"ms":62},"value":6 2},"user":{"ticks":31,"time":{"ms":31}}},"handles":{"open":96},"info":{"ephemera l_id":"24d146df-4d09-4c81-b018-8fbd0944dbd9","uptime":{"ms":33060}},"memstats":{ "gc_next":4194304,"memory_alloc":2192440,"memory_total":6190544,"rss":20918272}, "runtime":{"goroutines":16}},"filebeat":{"harvester":{"open_files":0,"running":0 }},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"lo gstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{ "current":0}},"system":{"cpu":{"cores":2}}}}} 2019-12-15T23:42:30.693-0500 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"syst em":{"ticks":31},"total":{"ticks":62,"value":62},"user":{"ticks":31}},"handles": {"open":96},"info":{"ephemeral_id":"24d146df-4d09-4c81-b018-8fbd0944dbd9","uptim e":{"ms":63060}},"memstats":{"gc_next":4194304,"memory_alloc":2250616,"memory_to tal":6248720,"rss":32768},"runtime":{"goroutines":16}},"filebeat":{"harvester":{ "open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipel ine":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}
I also tried a file output with the Logstash conf file. There was no output there. Could this be a problem with Logstash maybe? Kindly help me out.
Thanks,
Katara
Hey!
From the Filebeat's output you sent we cannot see any events sent to Logstash. So this makes me guess that maybe Filebeat is not collecting logs?
I would suggest we first check this. You can test your Filebeat's part only by keeping out Logstash for now and sending events to console. See: https://www.elastic.co/guide/en/beats/filebeat/current/console-output.html
So with this we will see if Filebeat is collecting logs by checking if events are shown in the console.
@ChrsMark, Sure, In the mean time, in one of the servers, I reinstalled Filebeat entirely, and it seems to push data again. I will try the console output in another server, so that I can get to the root cause.
Nice! Fyi: usually you can just delete data folder in such cases so as to clean start Beats, instead of removing all the installation.
@ChrsMark, Thank you!
And I think i see something,
so when in this fresh installation as well, the data has stopped coming after a point.
there was no changes made in any of the components.
is there any reason with scheduling or automatic refresh that I'm missing?
Hi!
Not sure that I fully understand your question but I Have not seen something like this. Could you elaborate more?
Also did you manage to capture any logs of Filebeat?
@ChrsMark, I mean, there is a time when i see no data logs loading and after sometime, it will start loading again. It doesn't seem very stable. I was just trying to understand the background.
and yes I ran a console but I'm certainly getting a lot of outputs in my console now, given, the data is flowing again now.
Thanks for clarifying.
In order to know if it is Filebeat's problem we need to look into the logs.
Thanks.
@ChrsMark, I'll try and fetch the logs once the issue occurs again and post it down.
Thank you so much for helping me out so far 
No problem!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.