Issue with search in Bitbucket Data Center

Elastic Stack Elasticsearch
1

I was able to install and configure Elasticsearch 5.5.3 on Ubuntu 14.04. I was able to connect to Bitbucket Data Center. However, when running a search I get: Server Busy The server couldn't process your request, please try again later.

The Elasticsearch logs show the following
[2018-10-11T13:36:27,610][WARN ][o.e.b.Natives ] cannot check if running as root because JNA is not available$
[2018-10-11T13:36:27,610][WARN ][o.e.b.Natives ] cannot install system call filter because JNA is not available$
[2018-10-11T13:36:27,611][WARN ][o.e.b.Natives ] cannot register console handler because JNA is not available$
[2018-10-11T13:36:27,612][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_NPROC because JNA is not available$
[2018-10-11T13:36:27,612][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_AS beacuse JNA is not available$
[2018-10-11T13:36:27,700][INFO ][o.e.n.Node ] [node-1] initializing ...$
[2018-10-11T13:36:27,770][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/var (/dev/sda6)]], net usable_space [1.8gb], net total_space [4.5gb], spins? [possibly], types [ext4]$
[2018-10-11T13:36:27,771][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1.9gb], compressed ordinary object pointers [true]$
[2018-10-11T13:36:27,779][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [8tIptIXfT6S_5cw8YX9Jew]$
[2018-10-11T13:36:27,779][INFO ][o.e.n.Node ] [node-1] version[5.5.3], pid[26519], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.4.0-59-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]$
[2018-10-11T13:36:27,780][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]$
[2018-10-11T13:36:28,708][INFO ][c.a.e.b.c.BucklerConfig ] Basic authentication for HTTP transport enabled (auth.basic.http.enabled): true$
[2018-10-11T13:36:28,708][INFO ][c.a.e.b.c.BucklerConfig ] Basic authentication for TCP transport enabled (auth.basic.tcp.enabled): false$
[2018-10-11T13:36:28,709][INFO ][c.a.e.b.c.BucklerConfig ] TLS for HTTP transport enabled (tls.http.enabled): false$
[2018-10-11T13:36:28,709][INFO ][c.a.e.b.c.BucklerConfig ] TLS for TCP transport enabled (tls.tcp.enabled): false$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-groovy]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty3]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded plugin [buckler]$
[2018-10-11T13:36:30,277][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.atlassian.elasticsearch.buckler.BucklerPlugin$
[2018-10-11T13:36:30,359][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen]$
[2018-10-11T13:36:30,858][INFO ][o.e.n.Node ] [node-1] initialized$
[2018-10-11T13:36:30,858][INFO ][o.e.n.Node ] [node-1] starting ...$
[2018-10-11T13:36:31,002][INFO ][o.e.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}$
[2018-10-11T13:36:31,013][WARN ][o.e.b.BootstrapChecks ] [node-1] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk$
[2018-10-11T13:36:34,074][INFO ][o.e.c.s.ClusterService ] [node-1] new_master {node-1}{8tIptIXfT6S_5cw8YX9Jew}{35hyG7IJRnGP-c3zk7Q4QQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)$
[2018-10-11T13:36:34,093][INFO ][c.a.e.b.SecureHttpServerTransport] [node-1] publish_address {10.11.183.110:9200}, bound_addresses {10.11.183.110:9200}$
[2018-10-11T13:36:34,093][INFO ][o.e.n.Node ] [node-1] started$
[2018-10-11T13:36:34,255][INFO ][o.e.g.GatewayService ] [node-1] recovered [1] indices into cluster_state$
[2018-10-11T13:36:34,825][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[bitbucket-search-v1][3]]

In the Bitbucket logs I see the following:

2018-10-12 00:21:22,016 WARN [Caesium-1-1] c.a.a.c.h.r.RemoteHashingInstructionsReader Unable to read remote instructions with key 'uid.onewayhash'.$
2018-10-12 00:21:22,017 WARN [Caesium-1-1] c.a.a.c.h.BcryptAnalyticsEmailHasher No instructions for hashing could be found.$
2018-10-12 08:28:08,653 ERROR [https-jsse-nio-7990-exec-9] sseidlit *JQ0YL6x508x27188x0 1lie9dm 10.19.67.20,10.80.134.70 "POST /rest/search/latest/search HTTP/1.1" c.a.b.s.internal.rest.SearchResource Unexpected response code from Elasticsearch: 400$

Any help would be most appreciated.

2

The warnings there are definitely not coming from elasticsearch, but from an extension not written or part of Elasticsearch itself, but rather from the bitbucket server.

I think your best bet is opening an issue in one of the bitbucket server issues to get this resolved.

Hope this helps!

3

Thanks for your response!! I did put a ticket in the Atlassian, but of course, they pointed back to Elasticsearch having the problem. I think it is something on the Bitbucket side too. I will throw it back in their court.
Thanks again...

4

you already pinpointed to the correct source snippet. the c.a.a.c.h.r prefix already points to com.atlassian rather than the regular o.e. which means org.elasticsearch

Also the last error comes from a jetty logger, not from an elasticsearch logger.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.