Error occurred after xpack setting

Elastic Stack Logstash
1

After xpack.security enabling, I received error below when I start logstash.

Nov 13 05:32:26 ip-172-31-60-170 logstash[25575]: [2019-11-13T14:32:26,597][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

setting

elasticsearch.yml
xpack.security.enabled: true

logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "logstash_system's password"
xpack.monitoring.elasticsearch.hosts: ["http://127.0.0.1:9200"]

I also set xpack.monitoring.enabled: false in logstash.yml. but in vain.

but I can log in kibana by elastic user.
Is this influenced to me?

I would appreciated to your answer.

2

Hi there,

Have you followed through all the steps in https://www.elastic.co/guide/en/logstash/7.4/ls-security.html ?

3

Thank you for your support.
I have set what you said besides of "PKI Authentication" and "TLS", but I have same problem.

The setting is below.

logstash.yml

4

Did you set the password for logstash_system by running elasticsearch-setup-passwords ? Can you try and authenticate with your logstash_system user with curl , i.e.

curl -u logstash_system http://127.0.0.1:9200

If you are not sure about the password for that user, you can change it ( since you know the password for the elastic user) by using the change password API

curl -u elastic -X POST "http://127.0.0.1:9200/_security/user/logstash_system/_password?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "the_new_password_here"
}
'
5

Thank you for your support.

I sent below command.

curl -u logstash_system http://127.0.0.1:9200

and server respond to enter password to me.

So,I sent below command.

curl -u xxxxxxxx:logstash_system http://127.0.0.1:9200curl

xxxxxxxx -> logstash_system’s password

and respond normally.

xxxxxxxx is set in logstash.yml

Is this a problem?

Best Regards,

Koikeda

6

I'll take it you mean you sent


curl -u logstash_system:xxxxxxxx http://127.0.0.1:9200

No, this is expected. I was just trying to see if we can verify that you are using the correct password and it seems you do.

Do you use the elasticsearch output plugin ? i.e. do you have any

 output {
      elasticsearch {
      ....
      }
 }

configuration sections ? If so, have you configured the appropriate username and password there ? Refer to the documentation I've already shared with you above

7

Yes, I define elasticsearch in INPUT and OUTPUT and set user and password.

I set user "logstash_internal" that was defined in Kibana with appropriated rights along your previous suggested link.

I set JDBC, too.

Is this a cause?

8

Share your full configuration please and make sure that you can authenticate with logstash_internal. Please share the output of

curl -u logstash_internal:<your_password_here> http://127.0.0.1:9200/_security/_authenticate
9

I got message like below.

{"username":"logstash_internal","roles":["logstash_writer","monitoring_user"],"full_name":"Internal Logstash User","email":null,"metadata":{},"enabled":true,"authentication_realm":{"name":"default_native","type":"native"},"lookup_realm":{"name":"default_native","type":"native"}}

Please confirm.

Best regards

Koikeda.

10

Please

12

I will resend.

(Attachment settingfile.zip is missing)

13

I will resend

(Attachment logstash.conf is missing)

15

I will send.

logstash.conf

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {

elasticsearch {

hosts => ["[127.0.0.1:9200](http://127.0.0.1:9200)"]

user => "elastic"

password => "wOedcSjME082qVMPckhR"

}

jdbc {

jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.12.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

jdbc_connection_string => 'jdbc:mysql://[153.126.196.248:3306/model_on?useSSL=true&requireSSL=true&verifyServerCertificate=true&trustCertificateKeyStoreUrl=file:///home/ubuntu/elastickey.jks&trustCertificateKeyStoreType=JKS&trustCertificateKeyStorePassword=MSeven11At8n](http://153.126.196.248:3306/model_on?useSSL=true&requireSSL=true&verifyServerCertificate=true&trustCertificateKeyStoreUrl=file:///home/ubuntu/elastickey.jks&trustCertificateKeyStoreType=JKS&trustCertificateKeyStorePassword=MSeven11At8n)'

jdbc_user => "elasticsearch"

jdbc_password => "MSeven11At8n"

statement => "SELECT request_d_c.*,[s1.name](http://s1.name),site.company,site.site_name,[account.name](http://account.name) as user_name,account.department as department_name,[device_type.name](http://device_type.name) as device_name,[simulator.name](http://simulator.name) as simulator_name,[mdc_quality.name](http://mdc_quality.name) as quality_name,[model_type.name](http://model_type.name) as type_name,[model_detail.name](http://model_detail.name) as detail_name,[source_type.name](http://source_type.name) as source_name FROM model_on.request_d_c

left join site on request_d_c.site_id = site.site_id

inner join account on request_d_c.user_id=account.id

left join device_type on request_d_c.device_type_id = device_type.id

left join simulator on request_d_c.simulator_id = simulator.id

left join mdc_quality on request_d_c.mdc_quality_id = mdc_quality.id

left join model_type on request_d_c.model_type_id = model_type.id

left join model_detail on request_d_c.model_detail_id = model_detail.id

left join source_type on request_d_c.source_type_id = source_type.id

left join device_type_map on request_d_c.device_id = device_type_map.device_id

left join device_type as s1 on device_type_map.device_type_id = s1.id where account.name <> 'Roman Reigns' and request_d_c.site_id <> '' and request_d_c.subscription_id <> '' and site.company <> '株式会社モーデック' and request_d_c.request_type='1';"

type => "download"

}

}

output {

if [type] == "download" {

elasticsearch {

  manage_template => false

  hosts => ["[127.0.0.1:9200](http://127.0.0.1:9200)"]

  user => "elastic"

  password => "wOedcSjME082qVMPckhR"

  index => "modelon"

  document_id => "%{id}"

}

stdout {codec => rubydebug {metadata => true }}

}

}

16

I will send setting.

elasticsearch.yml

======================== Elasticsearch Configuration =========================

17

I will send setting.

kibana.yml

Kibana is served by a back end server. This setting specifies the port to use.

#server.port: 5601

Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

The default is 'localhost', which usually means remote machines will not be able to connect.

To allow connections from remote users, set this parameter to a non-loopback address.

server.host: "0.0.0.0"

Enables you to specify a path to mount Kibana at if you are running behind a proxy.

Use the server.rewriteBasePath setting to tell Kibana if it should remove the basePath

from requests it receives, and to prevent a deprecation warning at startup.

This setting cannot end in a slash.

#server.basePath: ""

Specifies whether Kibana should rewrite requests that are prefixed with

server.basePath or require that they are rewritten by your reverse proxy.

This setting was effectively always false before Kibana 6.3 and will

default to true starting in Kibana 7.0.

#server.rewriteBasePath: false

The maximum payload size in bytes for incoming server requests.

#server.maxPayloadBytes: 1048576

The Kibana server's name. This is used for display purposes.

server.name: 127.0.0.1

The URLs of the Elasticsearch instances to use for all your queries.

elasticsearch.hosts: "http://127.0.0.1:9200"

When this setting's value is true Kibana uses the hostname specified in the server.host

setting. When the value of this setting is false, Kibana uses the hostname of the host

that connects to this Kibana instance.

#elasticsearch.preserveHost: true

Kibana uses an index in Elasticsearch to store saved searches, visualizations and

dashboards. Kibana creates a new index if the index doesn't already exist.

#kibana.index: ".kibana"

The default application to load.

#kibana.defaultAppId: "home"

If your Elasticsearch is protected with basic authentication, these settings provide

the username and password that the Kibana server uses to perform maintenance on the Kibana

index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

is proxied through the Kibana server.

elasticsearch.username: "kibana"

elasticsearch.password: "RdyuOL6Dqc7hrbD2TzTf"

Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

These settings enable SSL for outgoing requests from the Kibana server to the browser.

#server.ssl.enabled: false

#server.ssl.certificate: /path/to/your/server.crt

#server.ssl.key: /path/to/your/server.key

Optional settings that provide the paths to the PEM-format SSL certificate and key files.

These files validate that your Elasticsearch backend uses the same key files.

#elasticsearch.ssl.certificate: /path/to/your/client.crt

#elasticsearch.ssl.key: /path/to/your/client.key

Optional setting that enables you to specify a path to the PEM file for the certificate

authority for your Elasticsearch instance.

#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

To disregard the validity of SSL certificates, change this setting's value to 'none'.

#elasticsearch.ssl.verificationMode: full

Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of

the elasticsearch.requestTimeout setting.

#elasticsearch.pingTimeout: 1500

Time in milliseconds to wait for responses from the back end or Elasticsearch. This value

must be a positive integer.

#elasticsearch.requestTimeout: 30000

List of Kibana client-side headers to send to Elasticsearch. To send no client-side

headers, set this value to (an empty list).

#elasticsearch.requestHeadersWhitelist: [ authorization ]

Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten

by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

#elasticsearch.customHeaders: {}

Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

#elasticsearch.shardTimeout: 30000

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.

#elasticsearch.startupTimeout: 5000

Logs queries sent to Elasticsearch. Requires logging.verbose set to true.

#elasticsearch.logQueries: false

Specifies the path where Kibana creates the process ID file.

#pid.file: /var/run/kibana.pid

Enables you specify a file where Kibana stores log output.

#logging.dest: stdout

Set the value of this setting to true to suppress all logging output.

#logging.silent: false

Set the value of this setting to true to suppress all logging output other than error messages.

#logging.quiet: false

Set the value of this setting to true to log all events, including system usage information

and all requests.

#logging.verbose: false

Set the interval in milliseconds to sample system and process performance

metrics. Minimum is 100ms. Defaults to 5000.

#ops.interval: 5000

Specifies locale to be used for all localizable strings, dates and number formats.

Supported languages are the following: English - en , by default , Chinese - zh-CN .

#i18n.locale: "en"

18

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

20

Dear

Could you read this?

Is this not enough?

21

Hi, no, we can't read this still. Please do not send attachments in an email answer this doesn't work. You need to access the thread on your browser and copy paste your settings as I pointed out in an earlier answer. We are only interested in your logstash configuration at this point.

24

Please, this is not the first time we ask this of you in the current thread: don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

25 
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    user => "elastic"
    password => "wOedcSjME082qVMPckhR"
  }
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-8.0.12.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => 'jdbc:mysql://153.126.196.248:3306/model_on?useSSL=true&requireSSL=true&verifyServerCertificate=true&trustCertificateKeyStoreUrl=file:///home/ubuntu/elastickey.jks&trustCertificateKeyStoreType=JKS&trustCertificateKeyStorePassword=MSeven11At8n'
    jdbc_user => "elasticsearch"
    jdbc_password => "MSeven11At8n"
    statement => "SELECT request_d_c.*,s1.name,site.company,site.site_name,account.name as user_name,account.department as department_name,device_type.name as device_name,simulator.name as simulator_name,mdc_quality.name as quality_name,model_type.name as type_name,model_detail.name as detail_name,source_type.name as source_name FROM model_on.request_d_c
left join site on request_d_c.site_id = site.site_id
inner join account on request_d_c.user_id=account.id
left join device_type on request_d_c.device_type_id = device_type.id
left join simulator on request_d_c.simulator_id = simulator.id
left join mdc_quality on request_d_c.mdc_quality_id = mdc_quality.id
left join model_type on request_d_c.model_type_id = model_type.id
left join model_detail on request_d_c.model_detail_id = model_detail.id
left join source_type on request_d_c.source_type_id = source_type.id
left join device_type_map on request_d_c.device_id = device_type_map.device_id
left join device_type as s1 on device_type_map.device_type_id = s1.id where account.name <> 'Roman Reigns' and request_d_c.site_id <> '' and request_d_c.subscription_id <> '' and site.company <> '株式会社モーデック' and request_d_c.request_type='1';"
    type => "download"
  }
}

output {
  if [type] == "download" {
    elasticsearch {
      manage_template => false
      hosts => ["127.0.0.1:9200"]
      user => "elastic"
      password => "wOedcSjME082qVMPckhR"
      index => "modelon"
      document_id => "%{id}"
    }
    stdout {codec => rubydebug {metadata => true }}
  }
}