I have activated Elasticsearch from the AWS dashboard. My whole goal is to be able to view apache logs from my EC2 instance in Elasticsearch. I have installed Logstash on the EC2 instance. I have been able to start up logstash with this command:
sudo initctl start logstash
This gives:
logstash start/running, process 32490
At this time, nothing gets logged into /var/log/logstash/logstash-plain.log. Rather, if I run:
sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
This will hang for a minute and then output the following:
[ec2-user@ip-172-xx-xx-xxx logstash]$ sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console
18:41:02.270 [main] INFO logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
18:41:02.496 [LogStash::Runner] INFO logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"67eac888-a78d-4250-8b8d-b634b7d4b020", :path=>"/usr/share/logstash/data/uuid"}
18:41:03.928 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
18:41:04.194 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
18:41:04.829 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
echo "Hello, bob"
2017-05-22T18:45:50.271Z ip-172-xx-xx-xxx echo "Hello, bob"
tap tap is this thing on?
2017-05-22T18:47:38.457Z ip-172-xx-xx-xxx tap tap is this thing on?
This will create a logstash-plain.log file with this output:
[2017-05-22T18:41:30,594][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-05-22T18:43:14,250][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
Okay, so there some clear fatal errors that need to be worked out. But.. nonetheless I still feel like I am missing something here. Here is the logstash.conf file:
input {
tcp {
port => 10000
}
}
filter{
grok {
match => { "message" => "Hello, %{WORD:name}" }
}
}
output {
elasticsearch {
hosts => "https://search-site-hsldkfjoiwehgxvu5am.us-east-1.es.amazonaws.com"
}
}
Side note, I have installed the fingerprint plugin to try and replicate the “Getting started with Logstash” demo here on Elastic.co. So with this logstash.conf file in place I have run this command to try and send some data over to Kibana via Elasticsearch on AWS.
Here is the terminal output:
$ sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e /etc/logstash/conf.d/logstash.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console
19:04:00.763 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
Then when reviewing logstash-plain.log I get this:
[2017-05-22T19:02:00,406][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
Okay, so from all this there should be no big surprise that I am not able to load an index from this instance and view data in Kibana. So. what am I missing here?
Clearly from the errors I have some permissions errors, but I don’t recall in the install notes that I have to go back and update permissions. I have run many variations, but I would greatly appreciate some direction here. Thank you for your time.