Centralized Kibana to fetch logstash data from multiple DC

So you got a remote and local to connect on a local subnet is that correct could you do a cross cluster search etc?

From local to remote is connected. But when I try to do a cross cluster search on devtools, I am getting "index not found exception" as shown below .

On LOCAL

GET /remote-kibana:filebeat-2023.10.05/_search
{
  "query": {
    "match": {
      "host.name.keyword": "<<servername>>"
    }
  }
}

and output is

{
  "error": {
    "root_cause": [
      {
        "type": "index_not_found_exception",
        "reason": "no such index [filebeat-2023.10.05]",
        "index_uuid": "_na_",
        "resource.type": "index_or_alias",
        "resource.id": "filebeat-2023.10.05",
        "index": "filebeat-2023.10.05"
      }
    ],
    "type": "index_not_found_exception",
    "reason": "no such index [filebeat-2023.10.05]",
    "index_uuid": "_na_",
    "resource.type": "index_or_alias",
    "resource.id": "filebeat-2023.10.05",
    "index": "filebeat-2023.10.05"
  },
  "status": 404
}

And on REMOTE for the same index, I get the results.


GET /filebeat-2023.10.05/_search
{
  "query": {
    "match": {
      "host.name.keyword": "<<servername>>"
    }
  }
}

and output is

{
  "took": 1308,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": 2.8521688,
    "hits": [
      {
        "_index": "filebeat-2023.10.05",
        "_id": "fZA1_YoB8kd8DQIYCFp2",
        "_score": 2.8521688,
        "_ignored": [
          "event.original.keyword",
          "message.keyword"
        ],
        "_source": {
          "event": {
            "original": """2023-10-05 02:20:17,040	447179434	INFO  	MsgHandler7    	Declared media; tenant=1, id=3694845, externalMediaId=\\<<servermame>>\Incoming\2023\10\05\10-04-2023_15-22-36_sid_2081654661_504.wav, uri=http://10.112.33.6:25022/tenant/1/media/354875	[correlationId=d526e31d-3ea0-47b8-ac9e-8affbce06b26]	c.n.j.g.r.m.c.MediaCreationTask"""
          },
          "tags": [
            "filebeat",
            "beats_input_codec_plain_applied"
          ],

So... No it is not working....

Are you logged in as elastic user on LOCAL and REMOTE / same users when you run the search?

Did you try different indices?

Is there some elastic Role Based Controls on the Indices on Remote?

What error do you see on REMOTE when you run that command that fails.

Good thing it looks like LOCAL sees REMOTE cluster on the _search otherwise I am pretty sure you would get a different error like

"type": "no_such_remote_cluster_exception",

yes, elastic is the user both on LOCAL and REMOTE.

yes, tried different indices.

I have not created any specific role based controls on any indices.

Before I move forward with other troubleshooting, to confirm - we got Platinum license. That should be enough for cross cluster Search and replication with remote cluster, correct ?

Also, do we need to create any specific roles or users for cross cluster search specific or for replication ?

LOCAL (central kibana) elastic logs showed below error msg :

--------+-------------------------------------------------+----------------+
[2023-10-05T17:28:49,733][TRACE][o.e.t.TransportLogger    ] [centrKibana-node-1] Netty4TcpChannel{localAddress=/552.16.95.53:61197, remoteAddress=10.112.11.194/10.112.11.194:9300, profile=default} [length: 20684, request id: 670046, type: response, version: 8.6.2] READ: 20684B
[2023-10-05T17:28:49,733][TRACE][o.e.t.T.tracer           ] [centrKibana-node-1] [670046][indices:data/read/search] received response from [{EU-LSKib-node-1}{2v5zcxN9QP6_CQ-ty1qWGA}{xi_zUFj5Q_CD_4llRl6ySw}{EU-LSKib-node-1}{10.112.11.194}{10.112.11.194:9300}{cdfhilmrstw}{ml.allocated_processors_double=16.0, ml.max_jvm_size=17179869184, ml.allocated_processors=16, ml.machine_memory=34358702080, xpack.installed=true}]
[2023-10-05T17:28:49,734][DEBUG][r.suppressed             ] [centrKibana-node-1] path: /EU-remote-kibana%3Awinlogbeat-2023.10.04/_search, params: {pretty=true, index=EU-remote-kibana:winlogbeat-2023.10.04}
org.elasticsearch.transport.RemoteTransportException: [error while communicating with remote cluster [remote-kibana]]
Caused by: org.elasticsearch.transport.RemoteTransportException: [EU-LSKib-node-1][20.442.11.194:9300][indices:data/read/search]
Caused by: org.elasticsearch.index.IndexNotFoundException: no such index [winlogbeat-2023.10.04]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.notFoundException(IndexNameExpressionResolver.java:467) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.ensureAliasOrIndexExists(IndexNameExpressionResolver.java:481) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:1235) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:1141) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.resolveExpressions(IndexNameExpressionResolver.java:251) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:339) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:310) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:899) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:961) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:769) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:390) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-8.6.2.jar:?]

Good, Good, Good

With respect to License please confirm both are simple self-managed clusters, Not ECE, ECK or Elastic Cloud. If that is correct, then Basic for CCS and Platinum for CCR

If you are using ECE, ECK or Elastic Cloud any combination that crosses deployments/installations then requires Enterprise License.

I am confused why your GET says
GET /remote-kibana:filebeat-2023.10.05/_search

But the logs say

index=EU-remote-kibana:winlogbeat-2023.10.04

Not sure if you are showing me same thing? It helps if you are consistent and precise when running into tough problems.

try this and see what happens you need to use the name of the remote cluster you set up

GET /EU-remote-kibana:_all/_search

Try that and look at the logs / results

Says it can't find that index...

Also try this

GET /_remote/info from the local

I guess "self-managed clusters" means you have manually deployed and manage your Elastic Stack components (Elasticsearch, Logstash, Kibana, etc.) on your own infrastructure or servers. if that is right, then it is "self-managed clusters".

For, GET /_remote/info

{
  "EU-remote-kibana": {
    "connected": true,
    "mode": "sniff",
    "seeds": [
      "<< IP >> :9300"
    ],
    "num_nodes_connected": 1,
    "max_connections_per_cluster": 1,
    "initial_connect_timeout": "30s",
    "skip_unavailable": false
  }

And for, GET /EU-remote-kibana:_all/_search - I got below :

{
  "took": 114,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "_clusters": {
    "total": 1,
    "successful": 1,
    "skipped": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0,
    "hits": []
  }
}

Yes those are self-managed.

The _search results...

That says there is no indices on the remote cluster that can be seen by the local cluster.

Try this ... look carefully

GET /*:_all/_search

Are you sure there are no restrictions on the remote indices?

It seems like perhaps there is a privilege issue

Perhaps just create a test index on the remote cluster

POST /my-test-index/_doc
{
  "foo" : " bar"
}

The try to search from local

GET EU-remote-kibana:_all/_search

Make sure the roles are the same both sides

Are you sure there are no restrictions on the remote indices?

I have not put any restrictions manually. but is there a way to confirm if there are any restrictions on the remote indices? I am using "elastic" user which got superuser role.

For both, "GET /*:_all/_search" and "GET EU-remote-kibana:_all/_search", I got the below:

{
  "took": 109,
  "timed_out": false,
  "_shards": {
    "total": 0,
    "successful": 0,
    "skipped": 0,
    "failed": 0
  },
  "_clusters": {
    "total": 1,
    "successful": 1,
    "skipped": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": 0,
    "hits": []
  }
}

@Kvoyce2023

I poked the engineers... I am running out of ideas...

Did you try my suggestion of creating a new index manually on the remote?

Did you change any roles on the local side? It is that role that is passed to the remote as far as I understand

I just learned about this command...

From Engineering

I would start with this:

GET /_resolve/index/*:*

Resolve index API | Elasticsearch Guide [8.11] | Elastic If that shows nothing (or less than expected) that probably indicates some security/permissions issue (which I'm not skilled at investigating).

yes, I did create test index on remote cluster. And output of "GET EU-remote-kibana:_all/_search" is what is shown above with no hits.

I am using "elastic" user which got "superuser" role - which is what I am using on both LOCAL and REMOTE.

And for GET /_resolve/index/*:*

{
  "indices": [],
  "aliases": [],
  "data_streams": []
}

To be clear you ran

GET /_resolve/index/*:*? That is not what is shown

To put plain text use the backtick and the beginning and end I Edited your post take a look ... help be precise.

I have no clue at this point... there is something bad / basic going on.
I set up 2 clusters and got them working in 15 mins..

When you run this do you see any logs on the local or remote?

GET /_resolve/index/EU-remote-kibana:*

sorry, I used GET /_resolve/index/: only for the above output. Missed the *.* at the end.

for GET /_resolve/index/EU-remote-kibana:* -LOCAL give

GET /_resolve/index/EU-remote-kibana:*
{
  "indices": [],
  "aliases": [],
  "data_streams": []
}

on remote gives

GET /_resolve/index/EU-remote-kibana:*
{
 "error": {
   "root_cause": [
     {
       "type": "no_such_remote_cluster_exception",
       "reason": "no such remote cluster: [EU-remote-kibana]"
     }
   ],
   "type": "security_exception",
   "reason": "action [indices:admin/resolve/index] is unauthorized for user [elastic] with effective roles [superuser], this action is granted by the index privileges [view_index_metadata,manage,read,all]",
   "caused_by": {
     "type": "no_such_remote_cluster_exception",
     "reason": "no such remote cluster: [EU-remote-kibana]"
   }
 },
 "status": 403
}

I edited the post above to put the commands with the results. Can you please do that going forward.

When you write them into the sentence I can't tell what part is the sentence and what part is the command.

The command you ran makes sense

The local you're asking about the remote...

It's still saying it couldn't find any indices.

You wouldn't run the same command on the remote so the error makes total sense because there is no remote from the remote cluster... So that totally makes sense. All the commands I'm giving you I'll try to be better to be specific of which to run.

At the end we are still the same place, for some reason from the local to the remote, it doesn't see any indices.

If I were you at this point since we spent so much time

I would just set up two single node clusters from scratch and see if you can get it work.
There's something fundamentally wrong and I can't see it

Perhaps silly question. How many nodes are on the remote cluster? I'm surprised that it only says that there is one. Is the remote cluster only a single node?

I also asked when you ran this from the local
GET /_resolve/index/EU-remote-kibana:*

what logs / errors did you see in both the local and remotes elasticsearch logs.

So, remote cluster server got Kibana + logstash + Elasticsearch.
This remote server's Kibana.yml got the URLs of the Elasticsearch instances (which is 3 separate ES nodes) to use for the queries.

Just to be clear Cross Cluster Search does not flow through/ pass through Kibana ... It is from elasticsearch to elasticsearch.

ok let me share my server stack below :-

  1. Central server - with Kibana, Elasticsearch

  2. 3 Data center

    • each data center got below
      a) Elasticsearch on all 3 elasticsearch nodes and Elasticsearch nodes joined the Elastic
      cluster.
      b) Logstash on 2 LS nodes
      c) Kibana on one LS 1 node. This is the server where I am considering as Remoter Cluster server. so for cross cluster search to work, I installed Elasticsearch on this sever. This is the server, wher we are able to visualize the beats data, which gets send to elasticSearch via logstash.

I appreciate that I think I have the general understanding what you're trying to accomplish.

I have set up cross cluster search many times on many different types of environments.

My point was Kibana Logstash. All those things don't have anything to do with cross cluster search.

In the end you need to have correct connectivity, correct certs correct Remote cluster correct users roles indices. All those things are only related to elasticsearch nodes.

Also what I keep suggesting.

Did you look at the logs when you ran those remote searches

I understand you have this large architecture What I keep trying to suggest and I think you've tried but it's hard for me to tell

You cannot replicate cross cluster search on two simple single node clusters on the same subnet you'll never get it to work across the data centers.

Did you check your direct messages.

I'm not sure I can really help anymore through this thread.

A quick question.

Do I need to connect to one of the ES nodes server (item "a") as "Remote cluster" for the cross cluster search to work ? instead of making item "c" server as "remote cluster ?

If you have reversed the Local and Remote concepts that could explain the odd results you are seeing.

And just to be clear....

Logstash has nothing to do with CCS

Kibana's Only Role is it is a UI To Help Setup The Remote Cluster, which could all be done via Elasticsearch REST API

With Respect to Search Kibana is just a UI to Create / Send the CCS Search, Which could Also be accomplished via the Elasticsearch REST API

Perhaps take another look at this: