How to resolve failed requests to ES database after rebuilding the site

stan4o · May 13, 2023, 10:13pm

After our website (system) was rebuilt on a new server (Digital Ocean) all the requests to the Elastic search are failing = we cannot access the Elastic search. How to resolve this issue? I am not a programmer. This is what the developer that built the site says: "The issue is coming from ES. It is likely the query we are using.

I am not sure what the root cause is as the data looks perfectly fine, but from a guess, I would highly expect it to be related to the version of the software. We would most likely need to test the new build against any older versions of Elasticsearch we used and possibly work out any small deviations with the data structure.

Also it is useful to note that small corrections were made to the ES settings in order for the instance to work correctly. The issues have been documented however and can easily be used for reference purposes."

Do you know someone that can resolve this issue so we would be able to use our existing data (13M records)? As I said I'm not a programmer so I need someone experiences that can solve this for me.

Appreciate your help,
Stan

dadoonet · May 14, 2023, 1:41am

Welcome!

Did you change the Elasticsearch version? What were the initial version and the new one?
What are the error messages?
What are the Elasticsearch logs?

stan4o · May 15, 2023, 4:22am

Hi David,
Thank you for your quick response.
Let me clarify I am not a tech person, still trying to figure out how to access the new serve that someone else created...

To your questions:

Did you change the Elasticsearch version? - No, I've been told that the Elastic Search version was not changed.
What were the initial version and the new one? - According to my notes - Elastic Search original version 2.4.2 most probably or maybe 2.4.6
What are the error messages? - I have not seen the error messages, not a clue...
What are the Elasticsearch logs? - I am trying to access the server with WinSCP, but it asks me for a private key, I have to figure how to generate such a key etc... again I'm not a tech guy, will take time to figure out

David, do you know someone that can fix it. I will provide access to the server and of course should pay for the time and effort involved.
That might be a better solution rather than me learning server stuff...

Thanks,
Stan

dadoonet · May 15, 2023, 6:53am

We do have support at Elastic, but I don't think this will work because this version is not supported anymore for years.
And definitely, you must upgrade the cluster. We are now at 8.7.1.
But this implies a lot of changes on your side, so you probably need a consulting team to write the needed changes in your code. I can check if we have a partner in your area. Please send me a private message with your details (email, company name, location).

I'd suggest to have your cluster running at cloud.elastic.co. That will make your life easier in the future.

Coming back to the questions. We need to understand what is happening to help you. But if you don't have access to the source code, to the logs, there's barely nothing to help us understanding what is the actual problem.

stan4o · May 24, 2023, 1:54am

Hi David,
here are my findings so far:
installed on the server Elastic Search 6.1.1 - no change
found only one log file as below:

[2023-04-07T16:08:18,190][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2023-04-07T16:08:18,197][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
[2023-04-07T16:08:18,197][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 8388608, hard limit: 8388608
[2023-04-07T16:08:18,198][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
	# allow user 'elasticsearch' mlockall
	elasticsearch soft memlock unlimited
	elasticsearch hard memlock unlimited
[2023-04-07T16:08:18,198][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2023-04-07T16:08:18,880][INFO ][o.e.n.Node               ] [node1] initializing ...
[2023-04-07T16:08:19,155][INFO ][o.e.e.NodeEnvironment    ] [node1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda)]], net usable_space [181.2gb], net total_space [396.6gb], types [ext4]
[2023-04-07T16:08:19,156][INFO ][o.e.e.NodeEnvironment    ] [node1] heap size [1.9gb], compressed ordinary object pointers [true]
[2023-04-07T16:08:19,192][INFO ][o.e.n.Node               ] [node1] node name [node1], node ID [s8iRMIM6Q1mWVOSv3g5BYw]
[2023-04-07T16:08:19,193][INFO ][o.e.n.Node               ] [node1] version[6.1.1], pid[1], build[bd92e7f/2017-12-17T20:23:25.338Z], OS[Linux/5.19.0-35-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2023-04-07T16:08:19,193][INFO ][o.e.n.Node               ] [node1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2023-04-07T16:08:22,685][INFO ][o.e.p.PluginsService     ] [node1] loaded module [aggs-matrix-stats]
[2023-04-07T16:08:22,685][INFO ][o.e.p.PluginsService     ] [node1] loaded module [analysis-common]
[2023-04-07T16:08:22,685][INFO ][o.e.p.PluginsService     ] [node1] loaded module [ingest-common]
[2023-04-07T16:08:22,685][INFO ][o.e.p.PluginsService     ] [node1] loaded module [lang-expression]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [lang-mustache]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [lang-painless]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [mapper-extras]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [parent-join]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [percolator]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [reindex]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [repository-url]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [transport-netty4]
[2023-04-07T16:08:22,686][INFO ][o.e.p.PluginsService     ] [node1] loaded module [tribe]
[2023-04-07T16:08:22,687][INFO ][o.e.p.PluginsService     ] [node1] loaded plugin [ingest-geoip]
[2023-04-07T16:08:22,687][INFO ][o.e.p.PluginsService     ] [node1] loaded plugin [ingest-user-agent]
[2023-04-07T16:08:29,809][INFO ][o.e.d.DiscoveryModule    ] [node1] using discovery type [zen]
[2023-04-07T16:08:30,591][INFO ][o.e.n.Node               ] [node1] initialized
[2023-04-07T16:08:30,596][INFO ][o.e.n.Node               ] [node1] starting ...
[2023-04-07T16:08:30,765][INFO ][o.e.t.TransportService   ] [node1] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2023-04-07T16:08:30,786][WARN ][o.e.b.BootstrapChecks    ] [node1] memory locking requested for elasticsearch process but memory is not locked
[2023-04-07T16:08:33,848][INFO ][o.e.c.s.MasterService    ] [node1] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {node1}{s8iRMIM6Q1mWVOSv3g5BYw}{9cjFui0YRGeArVOR0nNvxw}{127.0.0.1}{127.0.0.1:9300}
[2023-04-07T16:08:33,868][INFO ][o.e.c.s.ClusterApplierService] [node1] new_master {node1}{s8iRMIM6Q1mWVOSv3g5BYw}{9cjFui0YRGeArVOR0nNvxw}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {node1}{s8iRMIM6Q1mWVOSv3g5BYw}{9cjFui0YRGeArVOR0nNvxw}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2023-04-07T16:08:33,929][INFO ][o.e.h.n.Netty4HttpServerTransport] [node1] publish_address {172.21.0.3:9200}, bound_addresses {0.0.0.0:9200}
[2023-04-07T16:08:33,929][INFO ][o.e.n.Node               ] [node1] started
[2023-04-07T16:08:34,391][INFO ][o.e.c.s.ClusterSettings  ] [node1] updating [action.destructive_requires_name] from [false] to [true]
[2023-04-07T16:08:34,444][INFO ][o.e.g.GatewayService     ] [node1] recovered [2] indices into cluster_state
[2023-04-07T16:08:35,719][INFO ][o.e.c.r.a.AllocationService] [node1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[app][4]] ...]).
[2023-05-22T05:46:30,447][WARN ][o.e.d.c.ParseField       ] Deprecated field [all_fields] used, replaced by [Set [fields] to `*` instead]
[2023-05-22T05:46:30,509][WARN ][o.e.d.c.ParseField       ] Deprecated field [all_fields] used, replaced by [Set [fields] to `*` instead]
[2023-05-22T05:46:30,515][WARN ][o.e.d.c.ParseField       ] Deprecated field [inline] used, expected [source] instead
[2023-05-22T05:46:30,520][WARN ][o.e.d.c.ParseField       ] Deprecated field [inline] used, expected [source] instead

system · June 21, 2023, 1:55am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.