Local + cross-cluster search as described in docs is not working

Elastic Stack Kibana
1

Hi, I am trying to create an index pattern in Kibana that will search both local and remote clusters, but I can only ever get it to do one or the other. The approach mentioned in the documentation and in the below forum thread does not seem to be working:

Here is an index pattern to match the local indices (86 total):

image
image1172×690 38.8 KB

Now here's an index pattern that matches remote indices (100 total):

image
image1172×685 42 KB

What pattern should I put to match both? I tried as described sessions2-*,*:sessions2-* but that returns the same results at the remote-only query. Same for if I reverse the order.

image
image1166×685 42.1 KB

I have not tried the other solution mentioned in the forum post (defining the local cluster as a remote one), but I feel like that is not the correct solution.

This is on ES/Kibana 7.5.1.

2

Do you think you happen to have exactly 100 indices that match the *:.. pattern? Or could it be that the query is limiting to 100 results?

Looking at the not-yet-released 7.6.0, I can see the size is limited to 200, but might have been 100 in previous releases (you can check if you open your browser console, go to the network tab, clear it just before typing the name of any new index pattern, and look at the bottom of the request tab);

image
image1281×1008 180 KB

So I think we may need to find some other way to verify if your index pattern is working. The comma-separated local,remote pattern should work.

3

Thank you for clarifying that, the wording should definitely be clarified then when it finds more than the max number of indices: "Your pattern matches 100 or more indices".

I noticed while trying to create that index pattern that it would sometimes time out on the query. Once it matched I created it anyway, and it pulls up some data in Discover, but any query I try to run times out and a message box with the following pops up:
Request to Elasticsearch failed: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

Clicking on "See the full error gives me the following traceback:

Wrapper@https://<redacted>/bundles/commons.bundle.js:1:1372222
KbnError@https://<redacted>/bundles/commons.bundle.js:1:1373768
RequestFailure@https://<redacted>/bundles/commons.bundle.js:1:2479966
callResponseHandlers/<@https://<redacted>/bundles/commons.bundle.js:1:2477427
Promise.try@https://<redacted>/bundles/commons.bundle.js:1:1252221
Promise.map/<@https://<redacted>/bundles/commons.bundle.js:1:1251571
Promise.map@https://<redacted>/bundles/commons.bundle.js:1:1251531
callResponseHandlers@https://<redacted>/bundles/commons.bundle.js:1:2475929
fetchSearchResults/<@https://<redacted>/bundles/commons.bundle.js:1:2458273
processQueue@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:204190
scheduleProcessQueue/<@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:205166
$digest@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:215161
$evalAsync/<@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:217871
completeTask@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:229772
Browser/self.defer/timeoutId<@https://<redacted>/built_assets/dlls/vendors.bundle.dll.js:327:67787

This error shows up in my logs (sensitive info redacted) and JSON prettified:

{
  "type": "response",
  "@timestamp": "2020-01-31T14:24:57Z",
  "tags": [],
  "pid": 32549,
  "method": "post",
  "statusCode": 500,
  "req": {
    "url": "/elasticsearch/sessions2-*,*:sessions2-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1580480141260&timeout=30000ms",
    "method": "post",
    "headers": {
      "host": "<redacted>",
      "x-real-ip": "<redacted>",
      "x-forwarded-for": "<redacted>",
      "x-forwarded-proto": "https",
      "x-forwarded-user": "<redacted>",
      "connection": "close",
      "content-length": "692",
      "user-agent": "<redacted>",
      "accept": "application/json, text/plain, */*",
      "accept-language": "en-US,en;q=0.5",
      "accept-encoding": "gzip, deflate, br",
      "referer": "<redacted>",
      "content-type": "application/json",
      "kbn-version": "7.5.1",
      "origin": "<redacted>",
      "dnt": "1"
    },
    "remoteAddress": "127.0.0.1",
    "userAgent": "127.0.0.1",
    "referer": "<redacted>"
  },
  "res": {
    "statusCode": 500,
    "responseTime": 30004,
    "contentLength": 9
  },
  "message": "POST /elasticsearch/sessions2-*,*:sessions2-*/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1580480141260&timeout=30000ms 500 30004ms - 9.0B"
}
Jan 31 09:25:27 kibana[32549]: Debug: internal, implementation, error
Jan 31 09:25:27 kibana[32549]: SyntaxError: Unexpected token u in JSON at position 0
Jan 31 09:25:27 kibana[32549]: at JSON.parse (<anonymous>)
Jan 31 09:25:27 kibana[32549]: at server.route.handler (/usr/share/kibana/src/legacy/core_plugins/elasticsearch/lib/create_proxy.js:85:21)
Jan 31 09:25:27 kibana[32549]: at process._tickCallback (internal/process/next_tick.js:68:7)

So I tried manually sending some queries to individual nodes and it seems that ES is spitting out the following traceback (again, sensitive info redacted):

Jan 31 09:42:46 elasticsearch[1867]: [2020-01-31T09:42:45,815][DEBUG][o.e.a.s.TransportSearchAction] [<redacted>] [sessions2-200113][2], node[ySyv3LTKR1KytsIGSakdxQ], [P], s[STARTED], a[id=43IVb5T-Qbuvt5_EGmQRbw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[sessions2-*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='1580480141260', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=, getOrCreateAbsoluteStartMillis=1580481735804, ccsMinimizeRoundtrips=true, source={from:0,size:500,timeout:30000ms,query:{bool:{filter:[{bool:{should:[{query_string:{query:atlr\-*,fields:[node^1.0],type:best_fields,default_operator:or,max_determinized_states:10000,enable_position_increments:true,fuzziness:AUTO,fuzzy_prefix_length:0,fuzzy_max_expansions:50,phrase_slop:0,escape:false,auto_generate_synonyms_phrase_query:true,fuzzy_transpositions:true,boost:1.0}}],adjust_pure_negative:true,minimum_should_match:1,boost:1.0}}],adjust_pure_negative:true,boost:1.0}},version:true,_source:{includes:[],excludes:[]},stored_fields:*,docvalue_fields:[{field:cert.notAfter,format:date_time},{field:cert.notBefore,format:date_time},{field:firstPacket,format:date_time},{field:lastPacket,format:date_time},{field:timestamp,format:date_time}],script_fields:{},sort:[{_score:{order:desc}}],track_total_hits:2147483647,highlight:{pre_tags:[@kibana-highlighted-field@],post_tags:[@/kibana-highlighted-field@],fragment_size:2147483647,fields:{*:{}}}}}] lastShard [true]
Jan 31 09:42:46 elasticsearch[1867]: org.elasticsearch.transport.RemoteTransportException: [<redacted>][<redacted>:9300][indices:data/read/search[phase/query]]
Jan 31 09:42:46 elasticsearch[1867]: Caused by: java.lang.IllegalStateException: Task cancelled before it started: by user request
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.tasks.TaskManager.registerCancellableTask(TaskManager.java:141) ~[elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.tasks.TaskManager.register(TaskManager.java:122) ~[elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:60) ~[elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:746) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.access00(TransportService.java:74) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:127) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:692) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:646) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:637) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:136) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:54) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda(AbstractSearchAsyncAction.java:227) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.tryRun(AbstractSearchAsyncAction.java:643) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.finishAndRunNext(AbstractSearchAsyncAction.java:637) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.action.search.AbstractSearchAsyncAction.doRun(AbstractSearchAsyncAction.java:284) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.ThreadContext.doRun(ThreadContext.java:773) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
Jan 31 09:42:46 elasticsearch[1867]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]

At this point I am kind of lost. Thank you for any help you can provide.

4

Oh, and one more thing, manual queries with the same index pattern in the console do succeed e.g. the following:

POST sessions2-*,*:sessions2-*/_search
{
  "query": {
    "query_string": {
      "query": "node: atlr-*"
    }
  }
}
5

Hello Team,

I too am facing the same issue with Kibana 7.5.2 with an Elasticsearch cluster running on 7.5.2.

The error is alerted when I search for multiple days as the following:
Request to Elasticsearch failed: {"error":{}}

Wrapper@http://REDACTED:5601/bundles/commons.bundle.js:3:2622021      
KbnError@http://REDACTED:5601/bundles/commons.bundle.js:3:2623567
RequestFailure@http://REDACTED:5601/bundles/commons.bundle.js:3:4903680
callResponseHandlers/<@http://REDACTED:5601/bundles/commons.bundle.js:3:4901144
Promise.try@http://REDACTED:5601/bundles/commons.bundle.js:3:2504121
Promise.map/<@http://REDACTED:5601/bundles/commons.bundle.js:3:2503492
Promise.map@http://REDACTED:5601/bundles/commons.bundle.js:3:2503452
callResponseHandlers@http://REDACTED:5601/bundles/commons.bundle.js:3:4899658
fetchSearchResults/<@http://REDACTED:5601/bundles/commons.bundle.js:3:4882019
processQueue@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:204190
scheduleProcessQueue/<@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:205154
$digest@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:215159
$evalAsync/<@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:217871
completeTask@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:229772
Browser/self.defer/timeoutId<@http://REDACTED:5601/built_assets/dlls/vendors.bundle.dll.js:435:67787

There are no rejected threads seen in the /_cat/thread_pool?v.

However, I am able to see this error message in the Elasticsearch logs:
[DEBUG][o.e.a.s.TransportSearchAction] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [indices:data/read/search[phase/fetch/id]]
Caused by: java.lang.IllegalStateException: Task cancelled before it started: by user request

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.