Sporadic failures with X-Pack/Shield

Elastic Stack Logstash
1

We started using xpack/shield with ElasticSearch 5.2. Unfortunately, we get random auth errors and can't even get logstash running because of it (it was running fine before).

When requesting
http://kibana:9200/_template/my-index-template with
correct auth headers via Postman, we sometimes get a successful result, the
rest of the time we get the following error. It's the exact same request both
times, once it works, once it doesn't.

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "error attempting to authenticate request",
        "header": {
          "WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type": "security_exception",
    "reason": "error attempting to authenticate request",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "authentication is already present in the context"
    },
    "header": {
      "WWW-Authenticate": "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status": 401
}

Logs:

[2017-02-23T15:37:14,191] [transport] [access_granted]	origin_type=[rest], origin_address=[10.15.66.199], principal=[logstash_internal], action=[indices:admin/template/get], request=[GetIndexTemplatesRequest]
[2017-02-23T15:37:15,365] [transport] [access_granted]	origin_type=[rest], origin_address=[10.15.66.199], principal=[logstash_internal], action=[indices:admin/template/get], request=[GetIndexTemplatesRequest]
[2017-02-23T15:37:15,936] [rest] [authentication_failed]	origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template]
[2017-02-23T15:37:16,375] [rest] [authentication_failed]	origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template]

We use the "logstash_internal" user with role "logstash_writer", taken from this setup. I've temporarily given logstash_writer all privileges I could find. Given the sporadic nature of the problem, I guess that's not the cause.

_xpack/security/role/logstash_writer:

{"logstash_writer":{"cluster":["manage_index_templates","monitor","all","manage","manage_security"],"indices":[{"names":["logstash-*","otherindex1-*","thing-requests-new-*","thing-requests-*"],"privileges":["write","delete","create_index","all"],"field_security":{"grant":["*"]}}],"run_as":[],"metadata":{},"transient_metadata":{"enabled":true}}}

_xpack/security/user/logstash_internal:

{"logstash_internal":{"username":"logstash_internal","roles":["logstash_writer"],"full_name":"Internal Logstash User","email":"devnull@dev-null.at","metadata":{},"enabled":true}}

Elasticsearch config:

path.data: /var/lib/elasticsearch/data
network.host: localhost
network.bind_host: 0.0.0.0
http.port: 9200
node.max_local_storage_nodes: 1
action.destructive_requires_name: true
indices.fielddata.cache.size:  20%

xpack.security.audit.enabled: true

xpack:
  security:
    authc:
      realms:
        ldap1:
          type: ldap
          order: 0
          url: "ldaps://XXXXXXXXXX:636"
          bind_dn: "uid=kibana,ou=XXXXXXX"
          bind_password: XXXXXXXXX
          user_search:
            base_dn: "XXXXXXX"
            attribute: uid
          group_search:
            base_dn: "XXXXXXXXX"
          files:
            role_mapping: "CONFIG_DIR/x-pack/role_mapping.yml"
          unmapped_groups_as_roles: true

xpack:
  security:
    authc:
      realms:
        native1:
          type: native
          order: 1

action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*
2

Hi @Greek0,

I am sorry that you're seeing this error. Do you happen to have access to the elasticsearch log file? If so there should hopefully be a stacktrace for this. Also, is this 5.2.0 or 5.2.1?

3

Hi @jaymode,
thanks a lot for your reply! Neither elasticsearch.log nor elasticsearch_access.log contained much interesting information:

[2017-02-27T15:16:38,369][INFO ][o.e.p.PluginsService     ] [KB_AfTW] loaded plugin [x-pack]
[2017-02-27T15:16:39,764][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.XPackPlugin
[2017-02-27T15:16:40,961][INFO ][o.e.n.Node               ] initialized

access.log:

[2017-02-27T15:17:37,449] [transport] [access_granted]      origin_type=[rest], origin_address=[10.15.66.199], principal=[logstash_internal], action=[indices:admin/template/get], request=[GetIndexTemplatesRequest] 
[2017-02-27T15:17:42,136] [transport] [access_granted]      origin_type=[rest], origin_address=[10.15.66.199], principal=[logstash_internal], action=[indices:admin/template/get], request=[GetIndexTemplatesRequest] 
[2017-02-27T15:17:42,301] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:17:42,492] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:17:42,687] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:17:42,859] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:17:43,048] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:17:43,224] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:17:43,418] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:18:18,237] [rest] [authentication_failed]    origin_address=[10.15.66.199], principal=[logstash_internal], uri=[/_template/my-index-template] 
[2017-02-27T15:22:06,227] [transport] [access_granted]      origin_type=[rest], origin_address=[10.15.66.199], principal=[logstash_internal], action=[indices:admin/template/get], request=[GetIndexTemplatesRequest]

All those times, I sent exactly the same request with postman. Puzzling :frowning:

4

Do you mind trying to reproduce with the ?error_trace=true appended to the URL?

5

I didn't know about error_trace, thanks. I added it and got this (content too long for Discourse).

6

Thanks for the stack trace.

Edit: looks like you already added your config. Sorry. We're looking into this.

7

@Greek0 Can you try adding user_search.pool.enabled: false to your LDAP realm's configuration?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.