Can not open Stack monitoring

Elastic Stack Elasticsearch
,
21

Can you include the HAR here so I can take a look too? Thanks

22

Download HAR please from URL

23

The HAR doesn't contain any XHR requests from the browser to the server. That's what we really need to be able to debug this. Can you please capture a new one and ensure it includes XHR requests from the Stack Monitoring plugin?

24

New HAR can be downloaded from this URL.
Password 02799163

25

Did you have enough time to check if the new HAR contains required requests?

26

Did you have enough time to check if the new HAR contains required requests?

27

@v.n

Sorry for the delay. I tried to download from the link but it says the file is no longer available.

Does your situation look similar to this? https://github.com/elastic/kibana/issues/57079

28

Hi, yes it looks similar.
But we have four node cluster and secure config was partly implemented some months ago. After that monitoring worked well and what caused problems i dont know and cannot remember.
We have also nginx in front of kibana but even after disabling it and connecting to tcp 5601 directly nothing changes.

Regards,

Vladimir

29

Uploaded again
https://fs.ctdl.ru:44434/index.php/s/CE5KAq55kZeenxK
Pass 75426974

30

Are you seeing the same behavior as Can not open Stack monitoring?

31

I see the same behavior as written here.
But I don't know how to reproduce checks from previous post.

32

How does this one fail exactly?

POST /*:.monitoring-es-7-*,.monitoring-es-7-*/_search

Can you paste the entire response?

33 
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.24",
        "node" : "Zy4jPFsyR_S6mvlUQJOxDg",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.25",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.26",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.27",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.28",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.29",
        "node" : "0HC5iE8ITcqRY7kFPem6rQ",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.30",
        "node" : "Zy4jPFsyR_S6mvlUQJOxDg",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      }
    ]
  },
  "status" : 403
}
34

Thanks @v.n

How are you issuing this request? Through Kibana dev tools? Curl?

I want to ensure we remove the possibility of any intermediate service that might be transforming the request (like a proxy or LB)

35

Exactly through this one.

36

Is there a proxy or LB between Kibana and Elasticsearch?

Can you try running the query as a curl to Elasticsearch from the same machine running Elasticsearch?

37 
[user@myhost ~]$ curl -XPOST -u elastic:password 'https://myhost:9200/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}
38

Forgot to say that there is nothing between kibana and ES. I made request on node with kibana and es, its a coordinating node.

39

I'm assuming the answer is yes, but let's make sure this works:

curl -XPOST -u elastic:password 'https://myhost:9200/.monitoring-es-7-*/_search?pretty'

40

Sorry, I made a mistake in password for elastic user earlier and response was wrong.
So new queries and responses are below.

curl -XPOST -u elastic:password  'https://myhost:9200/*:.monitoring-es-7-*,.monitoring-es-7-*/_search?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      },
      {
        "type" : "security_exception",
        "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.24",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.25",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.26",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.27",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.28",
        "node" : "0HC5iE8ITcqRY7kFPem6rQ",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.29",
        "node" : "9FcVASLhQaecOi41I6Nu0g",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      },
      {
        "shard" : 0,
        "index" : ".monitoring-es-7-2020.09.30",
        "node" : "Zy4jPFsyR_S6mvlUQJOxDg",
        "reason" : {
          "type" : "security_exception",
          "reason" : "action [indices:data/read/search[phase/query]] is unauthorized for user [elastic]",
          "caused_by" : {
            "type" : "illegal_state_exception",
            "reason" : "There are no external requests known to support wildcards that don't support replacing their indices"
          }
        }
      }
    ]
  },
  "status" : 403
}

and response for

curl -XPOST -u elastic:password 'https://myhost:9200/.monitoring-es-7-*/_search?pretty'

is too long and looks as expected.