Logstash server errors

Elastic Stack Logstash
1

installed ELK stack 5.2.2 on ubuntu 14.04 server image and put 5.2.2 filebeat on 5 other servers to send logs up to logstash.

I can curl elasticsearch and login to kibana but my new logstash is flooding errors into logstash-plain.log and i am not getting any new logs into elasticsearch.

here is the logstash error message in logstash-plain.log:

[2017-03-21T13:58:50,496][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication token for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}"}

i validated logstash config files and restarted logstash:
sudo bin/logstash --config.test_and_exit --path.settings /etc/logstash
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

logstash does not seem to be crashing and restarting - process ID is not changing.

Any idea what this error is about and how to fix it?

2

Have you set auth details?

3

i think API auth should just be basic auth. how can i set this up for REST APIs? do i put something in the output file for this? does user have to be logstash_internal user?

4

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-user
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-password

https://www.elastic.co/guide/en/x-pack/current/logstash.html

They should get you started :slight_smile:

5

I think i have followed the instrucitons in the links which you sent. i'm still getting a continual error (and log message) when logstash tries to process my input plugin.

Here's the error caused by my input plugin:
[2017-03-22T18:19:38,779][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, { at line 6, column 6 (byte 48) after input{\nbeats{\nport => 5044\ntype => logs\n}\nuser "}

It's flagging an error with the 'user' parameter. I tried many different forms of indentation, etc. nothing works.
if i comment out, user and password, the input plugin is processed correct.

here's my input plugin:cat -n 02-filebeat-input.conf
1 input{
2 beats{
3 port => 5044
4 type => logs
5 }
6 user => logstash_internal
7 password => changeme
8 }

I followed the instructions at this link https://www.elastic.co/guide/en/x-pack/current/logstash.html - but looks like logstash doesn't follow its own documentation.

When i comment out the user and password in input, i see same problem in filter plugin. if i comment out user and pw in input and filter, i see same problem in output plugin.
I verified that i am using logstash 5.2.2
lx=stack@login: os=admin /usr/share/logstash$ sudo bin/logstash --version
logstash 5.2.2

Any idea why it is rejecting my user and password parameters?

6

You don't need user details for inputs, just for outputs.

7

ok, i removed user and password from input and filter plugins and only have it in output plugin.
still getting continual error in logstash log file when logstash starts:

[2017-03-24T15:40:42,791][ERROR][logstash.agent ] Cannot load an invalid configuration {:reason=>"Expected one of #, { at line 26, column 6 (byte 469) after output {\nuser "}

It is still choking on 'user' even in the output plugin.

here is my output plugin. i have tried moving user and password various places within the plugin - no
success.

output {
user => logstash_internal
password => changeme
elasticsearch { hosts => ["localhost:9200"] }
stdout { codec => rubydebug }
}

thanks for the help - any further ideas?

8

They need to go inside the Elasticsearch section, that is what they relate to.

9

OK, it works when i put username and password in elasticsearch part of output plugin.
Here is final output plugin which made it work correctly.

lx=stack@login: os=admin /etc/logstash/conf.d$ cat 30-elasticsearch-output.conf
output {
elasticsearch { hosts => ["localhost:9200"] user => logstash_internal password => yourpassword}
stdout { codec => rubydebug }
}

thanks for the help.

10

see above fix to logstash output plugin. logstash working correctly.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.