# Local + cross-cluster search as described in docs is not working

**URL:** <https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271>\
**Category:** Kibana\
**Created:** [January 30, 2020, 8:36pm UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271 "2020-01-30T20:36:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![matoro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matoro/32/61791_2.png) [@matoro](https://discuss.elastic.co/u/matoro)\
**Post date:** [January 30, 2020, 8:36pm UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271/1 "2020-01-30T20:36:19Z")

</div>

Hi, I am trying to create an index pattern in Kibana that will search both local and remote clusters, but I can only ever get it to do one or the other. The approach mentioned in the documentation and in the below forum thread does not seem to be working:

> **[Index patterns has been renamed to data views. | Kibana Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/kibana/current/index-patterns.html)**

> [@How to run cross cluster search across local and remote indices](https://discuss.elastic.co/t/how-to-run-cross-cluster-search-across-local-and-remote-indices/117997):
>
> I am trying to setup my environment so that I can switch from using tribe nodes to use cross cluster search and running in to a few problems After some useful feedback in this forum I have configured my system with kibana pointing to a coordinating node in cluster\_1 with cross cluster search configured so I can also search indices in cluster\_2 and cluster\_3. I can configure index patterns in Kibana to search for logs as follows:- logstash-\* returns all local logs in clu…

Here is an index pattern to match the local indices (86 total):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/9/79be13037be769275153d62ace651961850d585f.png)

Now here's an index pattern that matches remote indices (100 total):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/9/4909280b1753d4ab1ee2b4213cd2790d7edf4686.png)

What pattern should I put to match both? I tried as described `sessions2-*,*:sessions2-*` but that returns the same results at the remote-only query. Same for if I reverse the order.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78307e63cf4bca3504a3bfcd7cfbf91d11f688e4.png)

I have not tried the other solution mentioned in the forum post (defining the local cluster as a remote one), but I feel like that is not the correct solution.

This is on ES/Kibana 7.5.1.

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [January 30, 2020, 9:15pm UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271/2 "2020-01-30T21:15:03Z")

</div>

Do you think you happen to have exactly 100 indices that match the `*:..` pattern? Or could it be that the query is limiting to 100 results?

Looking at the not-yet-released 7.6.0, I can see the size is limited to 200, but might have been 100 in previous releases (you can check if you open your browser console, go to the network tab, clear it just before typing the name of any new index pattern, and look at the bottom of the request tab);

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/5/85e93654aab49b6cf214fddd6f42041572f0bc7f.jpeg)

So I think we may need to find some other way to verify if your index pattern is working. The comma-separated `local,remote` pattern should work.

---

<div class="post-metadata">

**Author:** ![matoro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matoro/32/61791_2.png) [@matoro](https://discuss.elastic.co/u/matoro)\
**Post date:** [January 31, 2020, 2:48pm UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271/3 "2020-01-31T14:48:18Z")

</div>

Thank you for clarifying that, the wording should definitely be clarified then when it finds more than the max number of indices: "Your pattern matches 100 _or more_ indices".

I noticed while trying to create that index pattern that it would sometimes time out on the query. Once it matched I created it anyway, and it pulls up some data in Discover, but any query I try to run times out and a message box with the following pops up:  
`Request to Elasticsearch failed: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}`

Clicking on "See the full error gives me the following traceback:

```auto
Wrapper@https://<redacted>/bundles/commons.bundle.js:1:1372222
KbnError@https://<redacted>/bundles/commons.bundle.js:1:1373768
RequestFailure@https://<redacted>/bundles/commons.bundle.js:1:2479966
callResponseHandlers/<@https://<redacted>/bundles/commons.bundle.js:1:2477427
Promise.try@https://<redacted>/bundles/commons.bundle.js:1:1252221
Promise.map/<@https://<redacted>/bundles/commons.bundle.js:1:1251571
Promise.map@https://<redacted>/bundles/commons.bundle.js:1:1251531
callResponseHandlers@https://<redacted>/bundles/commons.bundle.js:1:2475929
fetchSearchResults/<@https://<redacted>/bundles/commons.bundle.js:1:2458273
processQueue@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:204190
scheduleProcessQueue/<@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:205166
$digest@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:215161
$evalAsync/<@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:217871
completeTask@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:229772
Browser/self.defer/timeoutId<@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:67787

```

This error shows up in my logs (sensitive info redacted) and JSON prettified:

```auto
{
  "type": "response",
  "@timestamp": "2020-01-31T14:24:57Z",
  "tags": [],
  "pid": 32549,
  "method": "post",
  "statusCode": 500,
  "req": {
    "url": "/elasticsearch/sessions2-*,*:sessions2-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1580480141260&timeout=30000ms",
    "method": "post",
    "headers": {
      "host": "<redacted>",
      "x-real-ip": "<redacted>",
      "x-forwarded-for": "<redacted>",
      "x-forwarded-proto": "https",
      "x-forwarded-user": "<redacted>",
      "connection": "close",
      "content-length": "692",
      "user-agent": "<redacted>",
      "accept": "application/json, text/plain, */*",
      "accept-language": "en-US,en;q=0.5",
      "accept-encoding": "gzip, deflate, br",
      "referer": "<redacted>",
      "content-type": "application/json",
      "kbn-version": "7.5.1",
      "origin": "<redacted>",
      "dnt": "1"
    },
    "remoteAddress": "127.0.0.1",
    "userAgent": "127.0.0.1",
    "referer": "<redacted>"
  },
  "res": {
    "statusCode": 500,
    "responseTime": 30004,
    "contentLength": 9
  },
  "message": "POST /elasticsearch/sessions2-*,*:sessions2-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1580480141260&timeout=30000ms 500 30004ms - 9.0B"
}
Jan 31 09:25:27 kibana[32549]: Debug: internal, implementation, error
Jan 31 09:25:27 kibana[32549]: SyntaxError: Unexpected token u in JSON at position 0
Jan 31 09:25:27 kibana[32549]: at JSON.parse (<anonymous>)
Jan 31 09:25:27 kibana[32549]: at server.route.handler (/usr/share/kibana/src/legacy/core_plugins/elasticsearch/lib/create_proxy.js:85:21)
Jan 31 09:25:27 kibana[32549]: at process._tickCallback (internal/process/next_tick.js:68:7)

```

So I tried manually sending some queries to individual nodes and it seems that ES is spitting out the following traceback (again, sensitive info redacted):

```auto
Jan 31 09:42:46 elasticsearch[1867]: [2020-01-31T09:42:45,815][DEBUG][o.e.a.s.TransportSearchAction] [<redacted>] [sessions2-200113][2], node[ySyv3LTKR1KytsIGSakdxQ], [P], s[STARTED], a[id=43IVb5T-Qbuvt5_EGmQRbw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[sessions2-*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='1580480141260', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=, getOrCreateAbsoluteStartMillis=1580481735804, ccsMinimizeRoundtrips=true, source={from:0,size:500,timeout:30000ms,query:{bool:{filter:[{bool:{should:[{query_string:{query:atlr\-*,fields:[node^1.0],type:best_fields,default_operator:or,max_determinized_states:10000,enable_position_increments:true,fuzziness:AUTO,fuzzy_prefix_length:0,fuzzy_max_expansions:50,phrase_slop:0,escape:false,auto_generate_synonyms_phrase_query:true,fuzzy_transpositions:true,boost:1.0}}],adjust_pure_negative:true,minimum_should_match:1,boost:1.0}}],adjust_pure_negative:true,boost:1.0}},version:true,_source:{includes:[],excludes:[]},stored_fields:*,docvalue_fields:[{field:cert.notAfter,format:date_time},{field:cert.notBefore,format:date_time},{field:firstPacket,format:date_time},{field:lastPacket,format:date_time},{field:timestamp,format:date_time}],script_fields:{},sort:[{_score:{order:desc}}],track_total_hits:2147483647,highlight:{pre_tags:[@kibana-highlighted-field@],post_tags:[@/kibana-highlighted-field@],fragment_size:2147483647,fields:{*:{}}}}}] lastShard [true]
Jan 31 09:42:46 elasticsearch[1867]: org.elasticsearch.transport.RemoteTransportException: [<redacted>][<redacted>:9300][indices:data/read/search[phase/query]]
Jan 31 09:42:46 elasticsearch[1867]: Caused by: java.lang.IllegalStateException: Task cancelled before it started: by user request
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.tasks.TaskManager.registerCancellableTask(TaskManager.java:141) ~[elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.tasks.TaskManager.register(TaskManager.java:122) ~[elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:60) ~[elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:746) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.access00(TransportService.java:74) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:127) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:692) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:646) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:637) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:136) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:54) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda(AbstractSearchAsyncAction.java:227) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.tryRun(AbstractSearchAsyncAction.java:643) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.finishAndRunNext(AbstractSearchAsyncAction.java:637) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.doRun(AbstractSearchAsyncAction.java:284) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.ThreadContext.doRun(ThreadContext.java:773) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]

```

At this point I am kind of lost. Thank you for any help you can provide.

---

<div class="post-metadata">

**Author:** ![matoro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matoro/32/61791_2.png) [@matoro](https://discuss.elastic.co/u/matoro)\
**Post date:** [January 31, 2020, 2:51pm UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271/4 "2020-01-31T14:51:12Z")

</div>

Oh, and one more thing, manual queries with the same index pattern in the console _do succeed_ e.g. the following:

```auto
POST sessions2-*,*:sessions2-*/_search
{
  "query": {
    "query_string": {
      "query": "node: atlr-*"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Srikanth\_Suresh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srikanth_suresh/32/62111_2.png) [@Srikanth\_Suresh](https://discuss.elastic.co/u/Srikanth_Suresh)\
**Post date:** [February 5, 2020, 9:54am UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271/5 "2020-02-05T09:54:20Z")

</div>

Hello Team,

I too am facing the same issue with Kibana 7.5.2 with an Elasticsearch cluster running on 7.5.2.

The error is alerted when I search for multiple days as the following:  
Request to Elasticsearch failed: {"error":{}}

```
Wrapper@http://REDACTED:5601/bundles/commons.bundle.js:3:2622021      
KbnError@http://REDACTED:5601/bundles/commons.bundle.js:3:2623567
RequestFailure@http://REDACTED:5601/bundles/commons.bundle.js:3:4903680
callResponseHandlers/<@http://REDACTED:5601/bundles/commons.bundle.js:3:4901144
Promise.try@http://REDACTED:5601/bundles/commons.bundle.js:3:2504121
Promise.map/<@http://REDACTED:5601/bundles/commons.bundle.js:3:2503492
Promise.map@http://REDACTED:5601/bundles/commons.bundle.js:3:2503452
callResponseHandlers@http://REDACTED:5601/bundles/commons.bundle.js:3:4899658
fetchSearchResults/<@http://REDACTED:5601/bundles/commons.bundle.js:3:4882019
processQueue@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:204190
scheduleProcessQueue/<@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:205154
$digest@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:215159
$evalAsync/<@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:217871
completeTask@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:229772
Browser/self.defer/timeoutId<@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:67787

```

There are no rejected threads seen in the /\_cat/thread\_pool?v.

However, I am able to see this error message in the Elasticsearch logs:  
[DEBUG][o.e.a.s.TransportSearchAction] Failed to execute fetch phase  
org.elasticsearch.transport.RemoteTransportException: [indices:data/read/search[phase/fetch/id]]  
Caused by: java.lang.IllegalStateException: Task cancelled before it started: by user request

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 4, 2020, 9:56am UTC](https://discuss.elastic.co/t/local-cross-cluster-search-as-described-in-docs-is-not-working/217271/6 "2020-03-04T09:56:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.