# Got response code '401' after Securing the Stack

**URL:** <https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [September 25, 2019, 5:41pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087 "2019-09-25T17:41:32Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 25, 2019, 5:41pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/1 "2019-09-25T17:41:32Z")

</div>

I recently used this guide [https://www.elastic.co/blog/getting-started-with-elasticsearch-security](https://www.elastic.co/blog/getting-started-with-elasticsearch-security) to secure our ELK stack.

I am able to log in to Kibana and TSL is working correctly. However my syslogs from logstash have stopped adding to the indices in elasticsearch.

I checked the logs for the logstash service and saw multiple lines with this error.  
[2019-09-25T12:24:43,390][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"[http://localhost:9200/](http://localhost:9200/)", :error\_type=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=\>"Got response code '401' contacting Elasticsearch at URL '[http://localhost:9200/](http://localhost:9200/)'"}

my logstash-sample.conf looks like this:

input {  
beats {  
port =\> 5044  
}  
}

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
user =\> "logstash\_system"  
password =\> "PASS from bin/elasticsearch-setup-passwords auto"  
}  
}

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2019, 5:44pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/2 "2019-09-25T17:44:00Z")

</div>

> [@mfisher](#):
>
> hosts =\> ["[http://localhost:9200](http://localhost:9200)"]

If you have TLS enabled I suppose this should be `https` and not `http`?

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 25, 2019, 5:46pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/3 "2019-09-25T17:46:36Z")

</div>

I thought that at first but I configured TLS a few days prior and things were running fine. Out of curiosity I changed it to https and it didn't resolve the issue.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 25, 2019, 5:50pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/4 "2019-09-25T17:50:37Z")

</div>

The built in `logstash_system` role does not have privileges to write to the indices. Have a look at [the documentation](https://www.elastic.co/guide/en/logstash/7.3/ls-security.html) and create a new user and role with the correct primileges.

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 25, 2019, 6:57pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/5 "2019-09-25T18:57:13Z")

</div>

I created a user "logstash"\_writer with the role "logstash\_writer\_role".

The role has cluster privs "manage\_index\_templates, monitor, manage\_ilm" and index privs "write, create, delete, create\_index, manage, manage\_ilm" for all indices.

Then added those credentials to the logstash-sample.conf and still get the same error.

Also I was mistaken about TLS that was only configured between Kibana and the browser. So that shouldn't be an issue.

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 25, 2019, 10:25pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/6 "2019-09-25T22:25:47Z")

</div>

I thought elasticsearch may be having an issue with the characters I used for the password so I changed the "logstash\_writer" user password to "testing" and got this.

> [2019-09-25T17:12:56,202][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"**[http://logstash\_writer:xxxxxx@localhost:9200/](http://logstash_writer:xxxxxx@localhost:9200/)**"}

however now there is a new error in the log

```
    [2019-09-25T17:15:36,500][WARN][logstash.outputs.elasticsearch] Overwriting supplied index logstash-asa with rollover alias cisco-asa
[2019-09-25T17:15:36,738][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]}
[2019-09-25T17:15:36,847][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

```

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 26, 2019, 8:00am UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/7 "2019-09-26T08:00:16Z")

</div>

Please show us your current elasticsearch configuration and the elasticsearch output plugin configuration from logstash. If you have enabled TLS for the http layer there is no way this will ever work with

```auto
hosts => ["http://localhost:9200"]

```

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 26, 2019, 2:47pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/8 "2019-09-26T14:47:14Z")

</div>

elasticsearch.yml

:/etc/elasticsearch# grep -v "#" elasticsearch.yml  
path.data: /var/lib/elasticsearch  
path.logs: /var/log/elasticsearch  
xpack.security.enabled: true

output from logstash.conf file:

output {

elasticsearch {  
hosts =\> ["localhost:9200"]  
manage\_template =\> true  
ilm\_enabled =\> "auto"  
ilm\_rollover\_alias =\> "cisco-asa"  
ilm\_pattern =\> "000001"  
ilm\_policy =\> "cisco\_asa\_rollover\_policy"  
index =\> "logstash-asa"  
user =\> logstash\_writer  
password =\> testing  
}  
}

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 26, 2019, 2:59pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/9 "2019-09-26T14:59:29Z")

</div>

can you authenticate to Elastiscsearch with `logstash_writer` and `testing`? i.e. can you share the output of

```auto
curl -ulogstash_writer:testing http://localhost:9200/_security/_authenticate

```

and the error still remains above as you have shared it ? If so I will proceed to move this to the Logstash subforum in the hopes that you get some more targeted help 🙂

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 26, 2019, 4:47pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/10 "2019-09-26T16:47:57Z")

</div>

output of "curl -ulogstash\_writer:testing [http://localhost:9200/\_security/\_authenticate](http://localhost:9200/_security/_authenticate)"

{  
"username" : "logstash\_writer",  
"roles" : [  
"logstash\_writer\_role"  
],  
"full\_name" : "Logstash Writer",  
"email" : "",  
"metadata" : { },  
"enabled" : true,  
"authentication\_realm" : {  
"name" : "default\_native",  
"type" : "native"  
},  
"lookup\_realm" : {  
"name" : "default\_native",  
"type" : "native"  
}  
}

I noticed in the logstash logs there was an entry referencing that ssl was required to be configured for security.

generated the .p12 with **bin/elasticsearch-certutil cert -out config/elastic-certificates.p12 -pass ""**

Added this to elasticsearch.yml:  
xpack.security.transport.ssl.enabled: true  
xpack.security.transport.ssl.verification\_mode: certificate  
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12  
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

now logstash logs show pipeline aborted due to this error :  
exception=\>#\<Manticore::UnknownException: **Unsupported or unrecognized SSL message** \>

I used **bin/elasticsearch-certutil cert --pem** to generate the cert for logstash and set the path in the .conf file

ssl =\> true  
cacert =\> "/etc/logstash/config/logstash.crt"

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 26, 2019, 10:10pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/11 "2019-09-26T22:10:08Z")

</div>

Could you close this thread, there were multiple configuration errors that I've found myself that have made the original issue moot.

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 27, 2019, 6:25pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/12 "2019-09-27T18:25:03Z")

</div>

It turns out this error was due to privs within my "logstash\_writer\_role" role.

I had added the following privs to the role

> privs " **manage\_index\_templates** , **monitor** , **manage\_ilm**" and index privs " **write** , **create** , **delete** , **create\_index** , **manage** , **manage\_ilm**" for all indices.

However whenever logstash tried to overwrite the index name with the rollover alias it encountered that error and killed the pipeline.

```
[2019-09-25T17:15:36,500][WARN][logstash.outputs.elasticsearch] Overwriting supplied index logstash-asa with rollover alias cisco-asa
[2019-09-25T17:15:36,738][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]}
[2019-09-25T17:15:36,847][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

```

Changing the role of the logstash\_writer to superuser fixed the issue.

What role privs would allow logstash to fully manage the indices without hitting this error?

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 29, 2019, 9:37am UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/13 "2019-09-29T09:37:23Z")

</div>

Those should be enough. Can you please verify the privileges that `logstash_writer_role` role has with

```auto
GET /_security/role/logstash_writer_role

```

?

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 30, 2019, 2:56pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/14 "2019-09-30T14:56:41Z")

</div>

GET \_security/roles/logstash\_writer\_role  
Can't connect to \_security:80 (Temporary failure in name resolution)

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [September 30, 2019, 3:32pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/15 "2019-09-30T15:32:33Z")

</div>

Apologies, I wasn't clear probably.

```auto
GET /_security/role/logstash_writer_role

```

should be run from the dev tools in Kibana. If this is not avaialble, you should use curl

```auto
curl -u elastic -X GET "localhost:9200/_security/role/logstash_writer_role?pretty"

```

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [September 30, 2019, 3:45pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/16 "2019-09-30T15:45:40Z")

</div>

> [@ikakavas](#):
>
> GET /\_security/role/logstash\_writer\_role

Gotcha heres the output:

````auto
  "logstash_writer_role" : {
    "cluster" : [
      "manage_ilm",
      "manage_index_templates",
      "monitor"
    ],
    "indices" : [
      {
        "names" : [
          "cisco-ios-*",
          "logstash-asa*",
          "cisco-asa-*",
          "apm-*"
        ],
        "privileges" : [
          "write",
          "create",
          "delete",
          "create_index",
          "manage",
          "manage_ilm"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [],
    "run_as" : [],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}```
````

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [October 1, 2019, 5:53am UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/17 "2019-10-01T05:53:58Z")

</div>

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with \</\> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

Your privilege list looks right. You previously mentioned that

> [@mfisher](#):
>
> I had added the following privs to the role
> 
> > privs " **manage\_index\_templates** , **monitor** , **manage\_ilm**" and index privs " **write** , **create** , **delete** , **create\_index** , **manage** , **manage\_ilm**" for all indices.

but it looks like those are applied to only the ones below:

> [@mfisher](#):
>
> "names" : [  
> "cisco-ios- _",  
> "logstash-asa_ ",  
> "cisco-asa- _",  
> "apm-_ "  
> ],

Are all the indices where logstash tries to write to covered by they above list ? Is this actually the indices list you saw or is this some formatting mishap ? You seem to be missing a few `*` from your index patterns

---

<div class="post-metadata">

**Author:** ![mfisher](https://avatars.discourse-cdn.com/v4/letter/m/ed655f/32.png) [@mfisher](https://discuss.elastic.co/u/mfisher)\
**Post date:** [October 1, 2019, 2:09pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/18 "2019-10-01T14:09:00Z")

</div>

They all have \*'s and those are all indices on the stack.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 29, 2019, 2:09pm UTC](https://discuss.elastic.co/t/got-response-code-401-after-securing-the-stack/201087/19 "2019-10-29T14:09:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.