Need help getting Logstash index from EC2 instance to Elasticsearch


#1

I have activated Elasticsearch from the AWS dashboard. My whole goal is to be able to view apache logs from my EC2 instance in Elasticsearch. I have installed Logstash on the EC2 instance. I have been able to start up logstash with this command:

sudo initctl start logstash

This gives:

logstash start/running, process 32490

At this time, nothing gets logged into /var/log/logstash/logstash-plain.log. Rather, if I run:

 sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'

This will hang for a minute and then output the following:

[ec2-user@ip-172-xx-xx-xxx logstash]$ sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console
18:41:02.270 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
18:41:02.496 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"67eac888-a78d-4250-8b8d-b634b7d4b020", :path=>"/usr/share/logstash/data/uuid"}
18:41:03.928 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
18:41:04.194 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
18:41:04.829 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
echo "Hello, bob"
2017-05-22T18:45:50.271Z ip-172-xx-xx-xxx echo "Hello, bob"
tap tap is this thing on?
2017-05-22T18:47:38.457Z ip-172-xx-xx-xxx tap tap is this thing on?

This will create a logstash-plain.log file with this output:

[2017-05-22T18:41:30,594][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-05-22T18:43:14,250][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

Okay, so there some clear fatal errors that need to be worked out. But.. nonetheless I still feel like I am missing something here. Here is the logstash.conf file:

input {
  tcp {
    port => 10000
  }
}
filter{
  grok {
    match => { "message" => "Hello, %{WORD:name}" }      
  }
}
output {
  elasticsearch {
    hosts => "https://search-site-hsldkfjoiwehgxvu5am.us-east-1.es.amazonaws.com"
  }
}

Side note, I have installed the fingerprint plugin to try and replicate the “Getting started with Logstash” demo here on Elastic.co. So with this logstash.conf file in place I have run this command to try and send some data over to Kibana via Elasticsearch on AWS.

Here is the terminal output:

$ sudo bin/logstash --path.settings=/etc/logstash/logstash.yml -e /etc/logstash/conf.d/logstash.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console
19:04:00.763 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}

Then when reviewing logstash-plain.log I get this:

[2017-05-22T19:02:00,406][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:433:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:216:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:132:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:131:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:217:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

Okay, so from all this there should be no big surprise that I am not able to load an index from this instance and view data in Kibana. So. what am I missing here?

Clearly from the errors I have some permissions errors, but I don’t recall in the install notes that I have to go back and update permissions. I have run many variations, but I would greatly appreciate some direction here. Thank you for your time.


(Mark Walkom) #2

What OS, what version of things?


#3

The server is:

Linux version 4.4.51-40.60.amzn1.x86_64 (mockbuild@gobi-build-64010) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Wed Mar 29 19:17:24 UTC 2017

I installed:

$ sudo yum install logstash-5.4.0.rpm 

I then had to create a symlink to allow logstash to view the config files:

$ sudo ln -s /etc/logstash /usr/share/logstash/config

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.