Issue with search in Bitbucket Data Center

I was able to install and configure Elasticsearch 5.5.3 on Ubuntu 14.04. I was able to connect to Bitbucket Data Center. However, when running a search I get: Server Busy The server couldn't process your request, please try again later.

The Elasticsearch logs show the following
[2018-10-11T13:36:27,610][WARN ][o.e.b.Natives ] cannot check if running as root because JNA is not available$
[2018-10-11T13:36:27,610][WARN ][o.e.b.Natives ] cannot install system call filter because JNA is not available$
[2018-10-11T13:36:27,611][WARN ][o.e.b.Natives ] cannot register console handler because JNA is not available$
[2018-10-11T13:36:27,612][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_NPROC because JNA is not available$
[2018-10-11T13:36:27,612][WARN ][o.e.b.Natives ] cannot getrlimit RLIMIT_AS beacuse JNA is not available$
[2018-10-11T13:36:27,700][INFO ][o.e.n.Node ] [node-1] initializing ...$
[2018-10-11T13:36:27,770][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/var (/dev/sda6)]], net usable_space [1.8gb], net total_space [4.5gb], spins? [possibly], types [ext4]$
[2018-10-11T13:36:27,771][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1.9gb], compressed ordinary object pointers [true]$
[2018-10-11T13:36:27,779][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [8tIptIXfT6S_5cw8YX9Jew]$
[2018-10-11T13:36:27,779][INFO ][o.e.n.Node ] [node-1] version[5.5.3], pid[26519], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.4.0-59-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_181/25.181-b13]$
[2018-10-11T13:36:27,780][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]$
[2018-10-11T13:36:28,708][INFO ][c.a.e.b.c.BucklerConfig ] Basic authentication for HTTP transport enabled (auth.basic.http.enabled): true$
[2018-10-11T13:36:28,708][INFO ][c.a.e.b.c.BucklerConfig ] Basic authentication for TCP transport enabled (auth.basic.tcp.enabled): false$
[2018-10-11T13:36:28,709][INFO ][c.a.e.b.c.BucklerConfig ] TLS for HTTP transport enabled (tls.http.enabled): false$
[2018-10-11T13:36:28,709][INFO ][c.a.e.b.c.BucklerConfig ] TLS for TCP transport enabled (tls.tcp.enabled): false$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-groovy]$
[2018-10-11T13:36:28,714][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty3]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]$
[2018-10-11T13:36:28,715][INFO ][o.e.p.PluginsService ] [node-1] loaded plugin [buckler]$
[2018-10-11T13:36:30,277][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.atlassian.elasticsearch.buckler.BucklerPlugin$
[2018-10-11T13:36:30,359][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen]$
[2018-10-11T13:36:30,858][INFO ][o.e.n.Node ] [node-1] initialized$
[2018-10-11T13:36:30,858][INFO ][o.e.n.Node ] [node-1] starting ...$
[2018-10-11T13:36:31,002][INFO ][o.e.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}$
[2018-10-11T13:36:31,013][WARN ][o.e.b.BootstrapChecks ] [node-1] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk$
[2018-10-11T13:36:34,074][INFO ][o.e.c.s.ClusterService ] [node-1] new_master {node-1}{8tIptIXfT6S_5cw8YX9Jew}{35hyG7IJRnGP-c3zk7Q4QQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)$
[2018-10-11T13:36:34,093][INFO ][c.a.e.b.SecureHttpServerTransport] [node-1] publish_address {10.11.183.110:9200}, bound_addresses {10.11.183.110:9200}$
[2018-10-11T13:36:34,093][INFO ][o.e.n.Node ] [node-1] started$
[2018-10-11T13:36:34,255][INFO ][o.e.g.GatewayService ] [node-1] recovered [1] indices into cluster_state$
[2018-10-11T13:36:34,825][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[bitbucket-search-v1][3]]

In the Bitbucket logs I see the following:

2018-10-12 00:21:22,016 WARN [Caesium-1-1] c.a.a.c.h.r.RemoteHashingInstructionsReader Unable to read remote instructions with key 'uid.onewayhash'.$
2018-10-12 00:21:22,017 WARN [Caesium-1-1] c.a.a.c.h.BcryptAnalyticsEmailHasher No instructions for hashing could be found.$
2018-10-12 08:28:08,653 ERROR [https-jsse-nio-7990-exec-9] sseidlit *JQ0YL6x508x27188x0 1lie9dm 10.19.67.20,10.80.134.70 "POST /rest/search/latest/search HTTP/1.1" c.a.b.s.internal.rest.SearchResource Unexpected response code from Elasticsearch: 400$

Any help would be most appreciated.

The warnings there are definitely not coming from elasticsearch, but from an extension not written or part of Elasticsearch itself, but rather from the bitbucket server.

I think your best bet is opening an issue in one of the bitbucket server issues to get this resolved.

Hope this helps!

Thanks for your response!! I did put a ticket in the Atlassian, but of course, they pointed back to Elasticsearch having the problem. I think it is something on the Bitbucket side too. I will throw it back in their court.
Thanks again...

you already pinpointed to the correct source snippet. the c.a.a.c.h.r prefix already points to com.atlassian rather than the regular o.e. which means org.elasticsearch

Also the last error comes from a jetty logger, not from an elasticsearch logger.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.