# Need help getting Logstash index from EC2 instance to Elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [May 22, 2017, 7:22pm UTC](https://discuss.elastic.co/t/need-help-getting-logstash-index-from-ec2-instance-to-elasticsearch/86726 "2017-05-22T19:22:13Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![fuel](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fuel](https://discuss.elastic.co/u/fuel)\ **Post date:** [May 22, 2017, 7:22pm UTC](https://discuss.elastic.co/t/need-help-getting-logstash-index-from-ec2-instance-to-elasticsearch/86726/1 "2017-05-22T19:22:13Z")
I have activated Elasticsearch from the AWS dashboard. My whole goal is to be able to view apache logs from my EC2 instance in Elasticsearch. I have installed Logstash on the EC2 instance. I have been able to start up logstash with this command: ``` sudo initctl start logstash ``` This gives: ``` logstash start/running, process 32490 ``` At this time, nothing gets logged into /var/log/logstash/logstash-plain.log. Rather, if I run: ``` sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }' ``` This will hang for a minute and then output the following: ``` [ec2-user@ip-172-xx-xx-xxx logstash]$ sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }' WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console 18:41:02.270 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} 18:41:02.496 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"67eac888-a78d-4250-8b8d-b634b7d4b020", :path=>"/usr/share/logstash/data/uuid"} 18:41:03.928 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125} 18:41:04.194 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started The stdin plugin is now waiting for input: 18:41:04.829 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600} echo "Hello, bob" 2017-05-22T18:45:50.271Z ip-172-xx-xx-xxx echo "Hello, bob" tap tap is this thing on? 2017-05-22T18:47:38.457Z ip-172-xx-xx-xxx tap tap is this thing on? ``` This will create a `logstash-plain.log` file with this output: ``` [2017-05-22T18:41:30,594][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]} [2017-05-22T18:43:14,250][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]} ``` Okay, so there some clear fatal errors that need to be worked out. But.. nonetheless I still feel like I am missing something here. Here is the logstash.conf file: ``` input { tcp { port => 10000 } } filter{ grok { match => { "message" => "Hello, %{WORD:name}" } } } output { elasticsearch { hosts => "https://search-site-hsldkfjoiwehgxvu5am.us-east-1.es.amazonaws.com" } } ``` Side note, I have installed the `fingerprint` plugin to try and replicate the “Getting started with Logstash” demo here on [Elastic.co](http://Elastic.co). So with this `logstash.conf` file in place I have run this command to try and send some data over to Kibana via Elasticsearch on AWS. Here is the terminal output: ``` $ sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e /etc/logstash/conf.d/logstash.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console 19:04:00.763 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "} ``` Then when reviewing logstash-plain.log I get this: ``` [2017-05-22T19:02:00,406][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]} ``` Okay, so from all this there should be no big surprise that I am not able to load an index from this instance and view data in Kibana. So. what am I missing here? Clearly from the errors I have some permissions errors, but I don’t recall in the install notes that I have to go back and update permissions. I have run many variations, but I would greatly appreciate some direction here. Thank you for your time. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 25, 2017, 10:02am UTC](https://discuss.elastic.co/t/need-help-getting-logstash-index-from-ec2-instance-to-elasticsearch/86726/2 "2017-05-25T10:02:20Z")
What OS, what version of things? ---
**Author:** ![fuel](https://avatars.discourse-cdn.com/v4/letter/f/df705f/32.png) [@fuel](https://discuss.elastic.co/u/fuel)\ **Post date:** [May 25, 2017, 1:13pm UTC](https://discuss.elastic.co/t/need-help-getting-logstash-index-from-ec2-instance-to-elasticsearch/86726/3 "2017-05-25T13:13:44Z")
The server is: ``` Linux version 4.4.51-40.60.amzn1.x86_64 (mockbuild@gobi-build-64010) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Wed Mar 29 19:17:24 UTC 2017 ``` I installed: ``` $ sudo yum install logstash-5.4.0.rpm ``` I then had to create a symlink to allow logstash to view the config files: ``` $ sudo ln -s /etc/logstash /usr/share/logstash/config ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2017, 1:13pm UTC](https://discuss.elastic.co/t/need-help-getting-logstash-index-from-ec2-instance-to-elasticsearch/86726/4 "2017-06-22T13:13:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.