Logstash server errors

[rmelton]

installed ELK stack 5.2.2 on ubuntu 14.04 server image and put 5.2.2 filebeat on 5 other servers to send logs up to logstash.

I can curl elasticsearch and login to kibana but my new logstash is flooding errors into logstash-plain.log and i am not getting any new logs into elasticsearch.

here is the logstash error message in logstash-plain.log:

[2017-03-21T13:58:50,496][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}"}

i validated logstash config files and restarted logstash:
sudo bin/logstash --config.test_and_exit --path.settings /etc/logstash
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

logstash does not seem to be crashing and restarting - process ID is not changing.

Any idea what this error is about and how to fix it?

[warkolm (Mark Walkom)]

@rmelton

missing authentication token for REST request

Have you set auth details?

[rmelton]

i think API auth should just be basic auth. how can i set this up for REST APIs? do i put something in the output file for this? does user have to be logstash_internal user?

[warkolm (Mark Walkom)]

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-user
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-password

https://www.elastic.co/guide/en/x-pack/current/logstash.html

They should get you started

[rmelton]

I think i have followed the instrucitons in the links which you sent. i'm still getting a continual error (and log message) when logstash tries to process my input plugin.

Here's the error caused by my input plugin:
[2017-03-22T18:19:38,779][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, { at line 6, column 6 (byte 48) after input{\nbeats{\nport => 5044\ntype => logs\n}\nuser "}

It's flagging an error with the 'user' parameter. I tried many different forms of indentation, etc. nothing works.
if i comment out, user and password, the input plugin is processed correct.

here's my input plugin:cat -n 02-filebeat-input.conf
1 input{
2 beats{
3 port => 5044
4 type => logs
5 }
6 user => logstash_internal
7 password => changeme
8 }

I followed the instructions at this link https://www.elastic.co/guide/en/x-pack/current/logstash.html - but looks like logstash doesn't follow its own documentation.

When i comment out the user and password in input, i see same problem in filter plugin. if i comment out user and pw in input and filter, i see same problem in output plugin.
I verified that i am using logstash 5.2.2
lx=stack@login: os=admin /usr/share/logstash$ sudo bin/logstash --version
logstash 5.2.2

Any idea why it is rejecting my user and password parameters?

[warkolm (Mark Walkom)]

You don't need user details for inputs, just for outputs.

[rmelton]

ok, i removed user and password from input and filter plugins and only have it in output plugin.
still getting continual error in logstash log file when logstash starts:

[2017-03-24T15:40:42,791][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, { at line 26, column 6 (byte 469) after output {\nuser "}

It is still choking on 'user' even in the output plugin.

here is my output plugin. i have tried moving user and password various places within the plugin - no
success.

output {
user => logstash_internal
password => changeme
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

thanks for the help - any further ideas?

[warkolm (Mark Walkom)]

They need to go inside the Elasticsearch section, that is what they relate to.

[rmelton]

OK, it works when i put username and password in elasticsearch part of output plugin.
Here is final output plugin which made it work correctly.

lx=stack@login: os=admin /etc/logstash/conf.d$ cat 30-elasticsearch-output.conf
output {
elasticsearch { hosts => ["localhost:9200"] user => logstash_internal password => yourpassword}
stdout { codec => rubydebug }
}

thanks for the help.

[rmelton]

see above fix to logstash output plugin. logstash working correctly.