Logstash not starting after enabling xpac security

[Tek_Chand]

Hello Team,

We are using ELK with 3 node cluster. Earlier we were using elasticsearch6.4 but now we upgraded to version6.8, because we want to use role base access feature.

We upgraded successfully our Elasticsaerch node, kibana node and logstash node.

We enable the xpac security by adding the below line in elasticsaerch.yml file over 3 elasticsearch nodes

xpack.security.enabled: true

We created the cert using below command over master node:

bin/elasticsearch-certutil cert

And we define the below line in our elasticsaerch.yml file:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificate.p12
xpack.security.transport.ssl.truststore.path: elastic-certificate.p12

Now we copied the same cert file i.e elastic-certificate.p12 and define same config in elasticsaerch.yml file.

Over kibana we define the the below line in kibana.yml

elasticsearch.username: "kibana"
elasticsearch.password: "password"

The above username and passowrd we got when we run the below command on elasticsaerch master node:

bin/elasticsearch-setup-passwords auto

And start the kibana service. Now we can see the user, role tab inside Management->Security tab and we are able to create roles and users.

But now when we are starting logstash we are getting error:

We defined the below config in logstash output filter:

output {
  elasticsearch {
    hosts => ["192.168.57.4:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    ssl => "true"
    cacert => "/etc/ssl/certs/elastic-certificate.p12"
    user => "elastic"
    password => "QjzmdpVIw47TrwTvOuo1"
  }
}

The user, passowrd and cert are we using for logstash are same whcih we got from the below commands:

bin/elasticsearch-certutil cert
bin/elasticsearch-setup-passwords auto

The error we are getting in logstash logs:

[ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Ch
eck the logs for more information.

Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x14550
909>", :error=>"signed fields invalid"

We tried to search the solution for both the error but didn't get anything.

Can you please help us?

Thank You

[Tek_Chand]

Hello Team,

Now this issue is resolved. Earlier i enabled SSL in logstash output filter. But after disabling the SSL and cacert flag fromlogstash is working fine.

New output plugin in logstash:

output {
  elasticsearch {
    hosts => ["192.168.57.4:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "QjzmdpVIw47TrwTvOuo1"
  }
}

Thank You