hello,
deleted ca.crt. I'm getting the below error in logstash
[2024-06-06T18:17:02,766][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://elastic:xxxxxx@******9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://*****:9200/][Manticore::UnknownException] Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty"}
and logstash is not restarting
i have tried to scp those file from the another node. Still it was not working
both logstash and elasticsearch running in same machine
hi thanks for you reply
curl -X GET "ip-address:9200/_cluster/health?pretty"
curl: (52) Empty reply from server
[2024-06-07T10:07:29,590][WARN ][o.e.h.AbstractHttpServerTransport] [freeipa.izmo.local] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/ip-address:9200, remoteAddress=/ip-adress:45194}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: internal_error
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLException: Received fatal alert: internal_error
at sun.security.ssl.Alert.createSSLException(Alert.java:132) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
... 16 more
i'm getting the above error in elasticsearch.log file
i was trying the different ways now i'm getting the below error
the services are also not running
tail -f /var/log/elasticsearch/elasticsearch.log
[2024-06-07T10:53:32,517][INFO ][o.e.x.w.WatcherService ] [free.local] stopping watch service, reason [shutdown initiated]
[2024-06-07T10:53:32,519][INFO ][o.e.x.w.WatcherLifeCycleService] [free.local] watcher has stopped and shutdown
[2024-06-07T10:53:32,791][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [free.local] [controller/143755] [Main.cc@176] ML controller exiting
[2024-06-07T10:53:32,841][INFO ][o.e.x.m.p.NativeController] [fre.local] Native controller process has stopped - no new native processes can be started
[2024-06-07T10:53:36,196][INFO ][o.e.n.Node ] [free.local] stopped
[2024-06-07T10:53:36,196][INFO ][o.e.n.Node ] [free.local] closing ...
[2024-06-07T10:53:36,244][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [free.local] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-7705864664188545806/geoip-databases/HWGj2gJ9SNK9g96vY9d-ng/GeoLite2-Country.mmdb]
[2024-06-07T10:53:36,245][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [free.local] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-7705864664188545806/geoip-databases/HWGj2gJ9SNK9g96vY9d-ng/GeoLite2-ASN.mmdb]
[2024-06-07T10:53:36,245][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [free.local] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-7705864664188545806/geoip-databases/HWGj2gJ9SNK9g96vY9d-ng/GeoLite2-City.mmdb]
[2024-06-07T10:53:36,249][INFO ][o.e.n.Node ] [free.local] closed
thanks for your reply! as i was a newbie for elk IDK where this parameter present the trustAnchors parameter must be non-empt
as for now i have copied the http_ca.crt file from another server it was not empty and i have given root:elasticsearch permission
how to make the logstash user to access the file
tail -f /var/log/elasticsearch/elasticsearch.log
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2097) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228) ~[?:?]
at java.security.KeyStore.load(KeyStore.java:1500) ~[?:?]
at org.elasticsearch.common.ssl.KeyStoreUtil.readKeyStore(KeyStoreUtil.java:72) ~[?:?]
at org.elasticsearch.common.ssl.StoreTrustConfig.readKeyStore(StoreTrustConfig.java:94) ~[?:?]
at org.elasticsearch.common.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:82) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:479) ~[?:?]
at java.util.HashMap.computeIfAbsent(HashMap.java:1228) ~[?:?]
at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:618) ~[?:?]
... 24 more
Enter password for the elasticsearch keystore : Exception in thread "main" java.lang.IllegalStateException: unable to read from standard input; is standard input open and a tty attached?
idk why i have created a passwd and added that in elasticsearch.yml
like:
xpack.security.transport.ssl.keystore.password: Passwd
i have deleted the elk and installed it again because idk what to do
it got installed successfully!
but elasticsearch is failing to start! idk why
in my-application.log the error was
[2024-06-11T19:13:28,250][ERROR][o.e.x.c.t.IndexTemplateRegistry] [freeipa.izmo.local] error adding ingest pipeline template [behavioral_analytics-events-final_pipeline] for [enterprise_search]
org.elasticsearch.node.NodeClosedException: node closed {freeipa.izmo.local}{faDlnzz4R8qituH-_C33PA}{yumqwfSxQuC_aGDFu_sLzQ}{freeipa.izmo.local}{10.125.103.30}{10.125.103.30:9300}{cdfhilmrstw}{8.14.0}{7000099-8505000}{ml.max_jvm_size=8359247872, ml.allocated_processors_double=4.0, ml.allocated_processors=4, ml.machine_memory=16717844480, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=12.0.0}
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onClusterServiceClose(TransportMasterNodeAction.java:317) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:382) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:280) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.ClusterApplierService.addTimeoutListener(ClusterApplierService.java:255) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:174) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:115) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.retry(TransportMasterNodeAction.java:308) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.retryOnNextState(TransportMasterNodeAction.java:283) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:230) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:186) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:191) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.ingest.IngestService$PipelineClusterStateUpdateTask.onFailure(IngestService.java:173) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Entry.onRejection(MasterService.java:1594) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.onRejection(MasterService.java:1615) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService.drainQueueOnRejection(MasterService.java:1316) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService.forkQueueProcessor(MasterService.java:1292) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$PerPriorityQueue.execute(MasterService.java:1347) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue.submitTask(MasterService.java:1565) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.ingest.IngestService.lambda$putPipeline$5(IngestService.java:450) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:202) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:196) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:289) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.lambda$newResponseAsync$2(TransportNodesAction.java:213) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.newResponseAsync(TransportNodesAction.java:213) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction$1.lambda$onCompletion$4(TransportNodesAction.java:164) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.nodes.TransportNodesAction.lambda$doExecute$0(TransportNodesAction.java:176) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:245) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:39) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.0.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1570) ~[?:?]
[2024-06-11T19:13:28,313][INFO ][o.e.n.Node ] [freeipa.izmo.local] stopped
[2024-06-11T19:13:28,314][INFO ][o.e.n.Node ] [freeipa.izmo.local] closing ...
journalctl -xeu elasticsearch.service
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit elasticsearch.service has finished with a failure.
░░
░░ The job identifier is 1561723 and the job result is failed.
Jun 11 16:51:43 freeipa.izmo.local systemd[1]: elasticsearch.service: Consumed 47.888s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit elasticsearch.service completed and consumed the indicated resources.
Jun 11 18:42:23 freeipa.izmo.local systemd[1]: Starting elasticsearch.service - Elasticsearch...
░░ Subject: A start job for unit elasticsearch.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit elasticsearch.service has begun execution.
░░
░░ The job identifier is 1651820.
Jun 11 18:42:28 freeipa.izmo.local systemd-entrypoint[484749]: Jun 11, 2024 6:42:28 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Jun 11 18:42:28 freeipa.izmo.local systemd-entrypoint[484749]: WARNING: COMPAT locale provider will be removed in a future release
Jun 11 18:43:09 freeipa.izmo.local systemd-entrypoint[484749]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/elasticsearch.log
Jun 11 18:43:09 freeipa.izmo.local systemd-entrypoint[484749]: ERROR: Elasticsearch died while starting up, with exit code 1
Jun 11 18:43:10 freeipa.izmo.local systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit elasticsearch.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Jun 11 18:43:10 freeipa.izmo.local systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
Jun 11 18:43:10 freeipa.izmo.local systemd[1]: Failed to start elasticsearch.service - Elasticsearch.
░░ Subject: A start job for unit elasticsearch.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit elasticsearch.service has finished with a failure.
░░
░░ The job identifier is 1651820 and the job result is failed.
Jun 11 18:43:10 freeipa.izmo.local systemd[1]: elasticsearch.service: Consumed 44.405s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit elasticsearch.service completed and consumed the indicated resources.
the above one is from jounalctl
please help me to solve this issuse
Actullay the below thing was not came as default in elasticsearch.yml
mine is fedora os I have followed the installations steps from the offcial document it slef. then i have copied this from another server and paste in my main server
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 11-06-2024 10:58:51
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ip-172-31-44-251"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
If encountering directories on the Java library path the Elasticsearch process does not have permission to access, the process fails to boot with a NullPointerException: Cannot invoke "org.elasticsearch.nativeaccess.Systemd.notify_ready()" because "this.systemd" is null.
This only affects on-prem installations of Elasticsearch, environments running Elasticsearch in a container are not affected, nor is Elastic Cloud.
The workaround is to grant Elasticsearch read access to the directory mentioned in the java.nio.file.AccessDeniedException as observed in the logs and repeat this until the process starts up properly, see here for further details. With 8.14.1 the workaround won’t be necessary anymore and original permissions can be restored.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
