# Configuring Filebeat to send the logs to remote Logstash(ELK) **URL:** https://discuss.elastic.co/t/configuring-filebeat-to-send-the-logs-to-remote-logstash-elk/121355 **Category:** Beats **Tags:** filebeat **Created:** [February 24, 2018, 10:55pm UTC](https://discuss.elastic.co/t/configuring-filebeat-to-send-the-logs-to-remote-logstash-elk/121355 "2018-02-24T22:55:11Z") **Posts on this page:** 3 **Page:** 1
### Author: ![Henry\_Han](https://avatars.discourse-cdn.com/v4/letter/h/b19c9b/32.png) [@Henry\_Han](https://discuss.elastic.co/u/Henry_Han) #### Post date: [February 24, 2018, 10:55pm UTC](https://discuss.elastic.co/t/configuring-filebeat-to-send-the-logs-to-remote-logstash-elk/121355/1 "2018-02-24T22:55:12Z")
Hello, This is the first time I use Filebeat, and I learned some basic knowledge of ELK from the online tutorial. Basically, I would like to pass the log information from the web-tier instance to the ELK server instance on Amazon Web Service EC2. First, I set up my Tomcat server on a web-tier instance (AWS EC2), and the Tomcat server has generated several txt log files stored in `/opt/tomcat/logs/` I installed Filebeat based on the instruction [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html;](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html;) and the filebeat.yml is: ```auto filebeat.prospectors: - type: log paths: - /opt/tomcat/logs/*.txt #- /var/log/*.log #- c:\programdata\elasticsearch\logs\* #== Filebeat modules == filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false #== Elasticsearch template setting == setup.template.settings: index.number_of_shards: 3 #index.codec: best_compression #_source.enabled: false #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts, elk_server_ip is the public ip address of the instance (ELK server) hosts: ["elk_server_ip:5044"] index: filebeat_logs ``` Next, I installed Kibana, Logstash & ElasticSearch on another AWS EC2 instance. And I enable the ports: |5044|tcp|0.0.0.0/0|✔| |5601|tcp|0.0.0.0/0|✔| |9200|tcp|0.0.0.0/0|✔| And then I set up a config file for Logstash: ```auto input { beats { port => 5044 } } filter { grok { match => { "message" => "/EventAdv/search\?user_id=%{NUMBER:user_id}&lat=%{NUMBER:latitude}&lon=%{NUMBER:longitude}" } } if "_grokparsefailure" in [tags] { drop { } } mutate { add_field => ["[geoip][location]","%{[longitude]}"] add_field => ["[geoip][location]","%{[latitude]}"] } mutate { convert => ["[geoip][location]","float"] } } output { stdout { codec => rubydebug } elasticsearch { hosts => "localhost" } } ``` After I started Logstash (sudo /usr/share/logstash/bin/logstash -f eventadv.conf), the starting process stuck and I got following logs ```auto Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2018-02-24 22:46:14.724 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} [INFO] 2018-02-24 22:46:14.732 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} [WARN] 2018-02-24 22:46:15.201 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2018-02-24 22:46:15.406 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.2"} [INFO] 2018-02-24 22:46:15.537 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600} [INFO] 2018-02-24 22:46:16.430 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [INFO] 2018-02-24 22:46:16.641 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}} [INFO] 2018-02-24 22:46:16.644 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"} [WARN] 2018-02-24 22:46:16.729 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"} [INFO] 2018-02-24 22:46:16.886 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>nil} [WARN] 2018-02-24 22:46:16.886 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6} [INFO] 2018-02-24 22:46:16.887 [[main]-pipeline-manager] elasticsearch - Using mapping template from {:path=>nil} [INFO] 2018-02-24 22:46:16.890 [[main]-pipeline-manager] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}} [INFO] 2018-02-24 22:46:16.903 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]} [INFO] 2018-02-24 22:46:17.319 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} [INFO] 2018-02-24 22:46:17.402 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#"} [INFO] 2018-02-24 22:46:17.423 [[main]1, :pipelines=>["main"]}" ``` Could you guys tell me what did I miss or what did I set up wrong? Thank you so much! Henry ---
### Author: ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens) #### Post date: [February 27, 2018, 3:44am UTC](https://discuss.elastic.co/t/configuring-filebeat-to-send-the-logs-to-remote-logstash-elk/121355/2 "2018-02-27T03:44:48Z")
All I see in the logs is Logstash starting up. Beats config looks good. Have you checked filebeat logs for errors? If you think connection filebeat-\>LS is the problem, then only test these two components. E.g. have Logstash run only this config: ```auto input { beats { port => 5044 } } output { stdout { codec => rubydebug } } ``` ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [March 27, 2018, 3:45am UTC](https://discuss.elastic.co/t/configuring-filebeat-to-send-the-logs-to-remote-logstash-elk/121355/3 "2018-03-27T03:45:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.