# Can not open Stack monitoring

**URL:** <https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring, elastic-stack-security\
**Created:** [June 19, 2020, 9:21am UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772 "2020-06-19T09:21:52Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 31, 2020, 2:34pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/21 "2020-08-31T14:34:42Z")

</div>

Can you include the HAR here so I can take a look too? Thanks

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [August 31, 2020, 3:09pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/22 "2020-08-31T15:09:05Z")

</div>

Download HAR please from [URL](https://egeon.bpa.ru/~s7Ekz)

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 31, 2020, 4:57pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/23 "2020-08-31T16:57:59Z")

</div>

The HAR doesn't contain any XHR requests from the browser to the server. That's what we really need to be able to debug this. Can you please capture a new one and ensure it includes XHR requests from the Stack Monitoring plugin?

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 4, 2020, 12:13pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/24 "2020-09-04T12:13:41Z")

</div>

New HAR can be downloaded from this [URL](https://fs.ctdl.ru:44434/index.php/s/jW9abjq2sNssMwf).  
Password 02799163

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 7, 2020, 7:20am UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/25 "2020-09-07T07:20:58Z")

</div>

Did you have enough time to check if the new HAR contains required requests?

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 21, 2020, 3:08pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/26 "2020-09-21T15:08:52Z")

</div>

Did you have enough time to check if the new HAR contains required requests?

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 29, 2020, 11:16am UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/27 "2020-09-29T11:16:19Z")

</div>

@v.n

Sorry for the delay. I tried to download from the link but it says the file is no longer available.

Does your situation look similar to this? [https://github.com/elastic/kibana/issues/57079](https://github.com/elastic/kibana/issues/57079)

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 29, 2020, 2:17pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/28 "2020-09-29T14:17:08Z")

</div>

Hi, yes it looks similar.  
But we have four node cluster and secure config was partly implemented some months ago. After that monitoring worked well and what caused problems i dont know and cannot remember.  
We have also nginx in front of kibana but even after disabling it and connecting to tcp 5601 directly nothing changes.

**Regards,**

Vladimir

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 29, 2020, 2:41pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/29 "2020-09-29T14:41:19Z")

</div>

Uploaded again  
[https://fs.ctdl.ru:44434/index.php/s/CE5KAq55kZeenxK](https://fs.ctdl.ru:44434/index.php/s/CE5KAq55kZeenxK)  
Pass 75426974

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 29, 2020, 2:55pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/30 "2020-09-29T14:55:27Z")

</div>

Are you seeing the same behavior as [Can not open Stack monitoring](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/15)?

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 29, 2020, 7:42pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/31 "2020-09-29T19:42:35Z")

</div>

I see the same behavior as written [here](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/16).  
But I don't know how to reproduce checks from previous [post](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/15).

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 29, 2020, 8:07pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/32 "2020-09-29T20:07:38Z")

</div>

How does this one fail exactly?

`POST /*:.monitoring-es-7-*,.monitoring-es-7-*/_search`

Can you paste the entire response?

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 30, 2020, 10:11am UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/33 "2020-09-30T10:11:04Z")

</div>

```
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.24",
        "node" : "Zy4jPFsyR_S6mvlUQJOxDg",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.25",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.26",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.27",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.28",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.29",
        "node" : "0HC5iE8ITcqRY7kFPem6rQ",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.30",
        "node" : "Zy4jPFsyR_S6mvlUQJOxDg",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      }
    ]
  },
  "status" : 403
}
```

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 30, 2020, 1:12pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/34 "2020-09-30T13:12:06Z")

</div>

Thanks @v.n

How are you issuing this request? Through Kibana dev tools? Curl?

I want to ensure we remove the possibility of any intermediate service that might be transforming the request (like a proxy or LB)

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 30, 2020, 1:41pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/35 "2020-09-30T13:41:19Z")

</div>

> [@chrisronline](#):
>
> Through Kibana dev tools?

Exactly through this one.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 30, 2020, 2:17pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/36 "2020-09-30T14:17:48Z")

</div>

Is there a proxy or LB between Kibana and Elasticsearch?

Can you try running the query as a curl to Elasticsearch from the same machine running Elasticsearch?

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 30, 2020, 2:58pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/37 "2020-09-30T14:58:04Z")

</div>

```
[user@myhost ~]$ curl -XPOST -u elastic:password 'https://myhost:9200/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}
```

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 30, 2020, 3:28pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/38 "2020-09-30T15:28:49Z")

</div>

Forgot to say that there is nothing between kibana and ES. I made request on node with kibana and es, its a coordinating node.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 30, 2020, 4:07pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/39 "2020-09-30T16:07:56Z")

</div>

I'm assuming the answer is yes, but let's make sure this works:

`curl -XPOST -u elastic:password 'https://myhost:9200/.monitoring-es-7-*/_search?pretty'`

---

<div class="post-metadata">

**Author:** ![v.n](https://avatars.discourse-cdn.com/v4/letter/v/5daacb/32.png) [@v.n](https://discuss.elastic.co/u/v.n)\
**Post date:** [September 30, 2020, 5:49pm UTC](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772/40 "2020-09-30T17:49:29Z")

</div>

Sorry, I made a mistake in password for elastic user earlier and response was wrong.  
So new queries and responses are below.

```auto
curl -XPOST -u elastic:password 'https://myhost:9200/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.24",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.25",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.26",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.27",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.28",
        "node" : "0HC5iE8ITcqRY7kFPem6rQ",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.29",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.30",
        "node" : "Zy4jPFsyR_S6mvlUQJOxDg",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      }
    ]
  },
  "status" : 403
}

```

and response for

```auto
curl -XPOST -u elastic:password 'https://myhost:9200/.monitoring-es-7-*/_search?pretty'

```

is too long and looks as expected.

[Previous page](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772.md?page=1)

[Next page](https://discuss.elastic.co/t/can-not-open-stack-monitoring/237772.md?page=3)