Errors from the virtual machine which hosts the ES

Elastic Stack Elasticsearch
1

Hello,

Could you please help me out, I am trying to understand why this errors are generated, but without any success. Thanks in advance!

AppIdentifier:onli

Culture:en-US

Environment:prd-eu

EventCode:100007

EventDetailCode:50301

EventId:ac941766-aa66-4bc9-bcec-a1bdcb3354d9

EventType:ReadSoft.ApplicationServices.Logging.FailureAuditEvent

Exception:ReadSoft.Online.Index.Exceptions.IndexSearchException: Failed to search in index masterdata-index-supplier2-eu. ---> Elasticsearch7.Net.ElasticsearchClientException: Maximum number of retries reached, failed over to all the known alive nodes before failing. Call: Status code unknown from: POST /masterdata-index-supplier2-eu/_search?typed_keys=true ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 20.123.75.169:9200 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at Elasticsearch7.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) --- End of inner exception stack trace --- --- End of inner exception stack trace --- at ReadSoft.Online.Index.Queries.QueryBase8.Throw`1.ThrowIndexSearchException[TRequest](IElasticSearchClient client, String indexName, Object initialFilter, TRequest request, IResponse response) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\Queries\QueryBase8.cs:line 302 at ReadSoft.Online.Index.Queries.QueryBase8.PagedSearch[T](IElasticSearchClient client, SearchDescriptor`1 searchDescriptor, Int32 getFrom, Int32 maxNumberOfResults, String indexName, Object initialFilter, Action`1 validate, Boolean useSearchAfter) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\Queries\QueryBase8.cs:line 49 at ReadSoft.Online.Index.Queries.MasterDataQueries8.SearchMasterDataSupplierWithBankAccountsItems(Object eventSource, IElasticSearchClient client, ICorporateGroupValidator groupValidator, MasterDataSupplierSearchQuery query, String indexName) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\Queries\MasterDataQueries8.cs:line 272 at ReadSoft.Online.Index.MasterDataIndexController8.SearchMasterDataSupplierIncludingBankAccounts(MasterDataSupplierSearchQuery query) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\MasterDataIndexController8.cs:line 635 at ReadSoft.Online.Index.MasterDataIndexController8.PagedResultSearchMasterSuppliers(MasterDataSupplierSearchQuery query) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\MasterDataIndexController8.cs:line 466 at ReadSoft.Online.MasterData.Impl.Repository.MasterSupplierRepository.SearchSuppliers(Guid corporateGroupId, SupplierSearchQuery query) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.MasterData.Impl\Repository\MasterSupplierRepository.cs:line 256 at ReadSoft.Online.MasterData.Impl.Repository.MasterSupplierRepository.GetSupplier(Guid corporateGroupId, Guid supplierId, Boolean includeBankAccounts) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.MasterData.Impl\Repository\MasterSupplierRepository.cs:line 101 at ReadSoft.Online.MasterData.Impl.MasterDataController.GetSupplier(Guid organizationId, Guid supplierId, Boolean includeBankAccounts) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.MasterData.Impl\MasterDataController.cs:line 135 at ReadSoft.Online.Core.Verification.DocumentEditingController.LoadSupplierData(Guid customerId, Nullable`1 buyerId, Document document, Boolean includeBankDetails) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Core\Verification\DocumentEditingController.cs:line 32 at ReadSoft.Online.Workflows.General.ValidateDocumentActivity.Execute(CodeActivityContext context) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Workflows\General\ValidateDocumentActivity.cs:line 37 at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation) Request: POST https://aps-prd-eu-es-ap2.northeurope.cloudapp.azure.com:9200/masterdata-index-supplier2-eu/_search?typed_keys=true {"from":0,"query":{"bool":{"filter":[{"terms":{"masterCorporateGroupId":["c5268dd2-c389-45df-a978-c9793bec8999"]}},{"bool":{"filter":[{"query_string":{"fields":["internalId"],"minimum_should_match":"100%","query":"34091917-a0a5-41ca-8254-686e722a8200"}}]}}]}},"size":1,"sort":[{"organizationNameLowerCase":{"order":"asc"}},{"nameLowerCase":{"order":"asc"}}],"track_total_hits":true} Initial filter: {"PageIndex":0,"PageSize":1,"SortField":"BuyerName","SortAscending":true,"Filter":{"MasterCorporateGroupId":{"Value":["c5268dd2-c389-45df-a978-c9793bec8999"],"Enabled":true},"Id":{"Value":"34091917-a0a5-41ca-8254-686e722a8200","Enabled":true},"Name":{"Value":null,"Enabled":false},"NameExactMatch":{"Value":null,"Enabled":false},"BuyerName":{"Value":null,"Enabled":false},"BuyerNameExactMatch":{"Value":null,"Enabled":false},"SupplierNumber":{"Value":null,"Enabled":false},"SupplierNumberExactMatch":{"Value":null,"Enabled":false},"Description":{"Value":null,"Enabled":false},"DescriptionExactMatch":{"Value":null,"Enabled":false},"TaxRegistrationNumber":{"Value":null,"Enabled":false},"TaxRegistrationNumberExactMatch":{"Value":null,"Enabled":false},"OrganizationNumber":{"Value":null,"Enabled":false},"OrganizationNumberExactMatch":{"Value":null,"Enabled":false},"Street":{"Value":null,"Enabled":false},"StreetExactMatch":{"Value":null,"Enabled":false},"PostalCode":{"Value":null,"Enabled":false},"PostalCodeExactMatch":{"Value":null,"Enabled":false},"City":{"Value":null,"Enabled":false},"CityExactMatch":{"Value":null,"Enabled":false},"CountryName":{"Value":null,"Enabled":false},"CountryNameExactMatch":{"Value":null,"Enabled":false},"PaymentTerm":{"Value":null,"Enabled":false},"PaymentTermExactMatch":{"Value":null,"Enabled":false},"PaymentMethod":{"Value":null,"Enabled":false},"PaymentMethodExactMatch":{"Value":null,"Enabled":false},"CurrencyCode":{"Value":null,"Enabled":false},"CurrencyCodeExactMatch":{"Value":null,"Enabled":false},"Location":{"Value":null,"Enabled":false},"LocationExactMatch":{"Value":null,"Enabled":false},"State":{"Value":null,"Enabled":false},"StateExactMatch":{"Value":null,"Enabled":false},"Blocked":{"Value":false,"Enabled":false},"TelephoneNumber":{"Value":null,"Enabled":false},"TelephoneNumberExactMatch":{"Value":null,"Enabled":false},"FaxNumber":{"Value":null,"Enabled":false},"FaxNumberExactMatch":{"Value":null,"Enabled":false},"TaxCode":{"Value":null,"Enabled":false},"TaxCodeExactMatch":{"Value":null,"Enabled":false},"BankAccountsCollection":{"Enabled":false,"BankAccountFilters":[]},"FreeQuery":{"Value":null,"Enabled":false},"BypassVerification":{"Value":false,"Enabled":false},"ExcludeFromIdentification":{"Value":false,"Enabled":false},"LockOnlineLearning":{"Value":false,"Enabled":false}},"IncludeBankAccounts":true,"ScoreBasedSorting":false,"UseSearchAfter":false}

Add to Search

InstanceId:ReadSoft.Online.WorkerRole_IN_29

IsAuditLog:false

LogLevel:Warn

MachineName:RD00155D4557A9

Message:Workflow aborted due to exception. Instance Id: ca8a3b2b-5723-46a6-95ad-0182dfd29d3b

Timestamp:2024-09-10T15:48:18.1645218Z
2

Could you check your elasticsearch logs? It seems this line is containing a full stacktrace, but badly formatted as a oneliner (please add the proper formatted one).

As far as I can read it, it is trying to do this operation:

POST /masterdata-index-supplier2-eu/_search?typed_keys=true

And fails due to some node being unavailable:

3

Hello Christof H,

Thank you for your response!

I’ve attached the one-liner logs in the .png file. I attempted to read the Elasticsearch logs generated locally at %ES_HOME%\logs, but encountered an error stating that “Auditing logging is DISABLED because the currently active license [BASIC] does not permit it.

logs1
logs11362×637 58.2 KB

Thanks in advance! :saluting_face:

4

Hi

Please attach logs as text. Beside that, this is still not properly formatted (so unreadable). You should also be able to find these logs at source (seems to me elasticsearch).

Did you validate the machine is able to reach "20.123.75.169:9200"?

Best regards
Christof

5

Hello, could you please check if you can reach the logs here:

The specified IP is the affected machine. While RDP is functioning correctly, I cannot ping 20.123.75.169:9200 via cmd. I ran netstat -an | find /i “listening” and did not find port 9200 listening, but port 9300 was active. According to the discussion here, port 9300 is a custom port and should work fine.

Thanks :slight_smile:

6

Hi

I added some indentation myself:

ReadSoft.Online.Index.Exceptions.IndexSearchException: Failed to search in index masterdata-index-supplier2-eu. 
---> Elasticsearch7.Net.ElasticsearchClientException: Maximum number of retries reached, failed over to all the known alive nodes before failing. Call: Status code unknown from: POST /masterdata-index-supplier2-eu/_search?typed_keys=true 
	---> System.Net.WebException: Unable to connect to the remote server 
		---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 20.123.75.169:9200
		at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
		at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
		--- End of inner exception stack trace ---
	at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
	at System.Net.HttpWebRequest.GetRequestStream()
	at Elasticsearch7.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData)
	--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at ReadSoft.Online.Index.Queries.QueryBase8.Throw`1.ThrowIndexSearchException[TRequest](IElasticSearchClient client, String indexName, Object initialFilter, TRequest request, IResponse response) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\Queries\QueryBase8.cs:line 302
at ReadSoft.Online.Index.Queries.QueryBase8.PagedSearch[T](IElasticSearchClient client, SearchDescriptor`1 searchDescriptor, Int32 getFrom, Int32 maxNumberOfResults, String indexName, Object initialFilter, Action`1 validate, Boolean useSearchAfter) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\Queries\QueryBase8.cs:line 49
at ReadSoft.Online.Index.Queries.MasterDataQueries8.SearchMasterDataSupplierWithBankAccountsItems(Object eventSource, IElasticSearchClient client, ICorporateGroupValidator groupValidator, MasterDataSupplierSearchQuery query, String indexName) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\Queries\MasterDataQueries8.cs:line 272
at ReadSoft.Online.Index.MasterDataIndexController8.SearchMasterDataSupplierIncludingBankAccounts(MasterDataSupplierSearchQuery query) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\MasterDataIndexController8.cs:line 635
at ReadSoft.Online.Index.MasterDataIndexController8.PagedResultSearchMasterSuppliers(MasterDataSupplierSearchQuery query) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Index\MasterDataIndexController8.cs:line 466
at ReadSoft.Online.MasterData.Impl.Repository.MasterSupplierRepository.SearchSuppliers(Guid corporateGroupId, SupplierSearchQuery query) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.MasterData.Impl\Repository\MasterSupplierRepository.cs:line 256
at ReadSoft.Online.MasterData.Impl.Repository.MasterSupplierRepository.GetSupplier(Guid corporateGroupId, Guid supplierId, Boolean includeBankAccounts) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.MasterData.Impl\Repository\MasterSupplierRepository.cs:line 101
at ReadSoft.Online.MasterData.Impl.MasterDataController.GetSupplier(Guid organizationId, Guid supplierId, Boolean includeBankAccounts) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.MasterData.Impl\MasterDataController.cs:line 135
at ReadSoft.Online.Core.Verification.DocumentEditingController.LoadSupplierData(Guid customerId, Nullable`1 buyerId, Document document, Boolean includeBankDetails) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Core\Verification\DocumentEditingController.cs:line 32
at ReadSoft.Online.Workflows.General.ValidateDocumentActivity.Execute(CodeActivityContext context) in C:\BuildAgent\work\e9026033474a65c0\Webeye\ReadSoft.Online.Workflows\General\ValidateDocumentActivity.cs:line 37
at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)
at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation) Request: POST https://aps-prd-eu-es-ap2.northeurope.cloudapp.azure.com:9200/masterdata-index-supplier2-eu/_search?typed_keys=true {"from":0,"query":{"bool":{"filter":[{"terms":{"masterCorporateGroupId":["c5268dd2-c389-45df-a978-c9793bec8999"]}},{"bool":{"filter":[{"query_string":{"fields":["internalId"],"minimum_should_match":"100%","query":"34091917-a0a5-41ca-8254-686e722a8200"}}]}}]}},"size":1,"sort":[{"organizationNameLowerCase":{"order":"asc"}},{"nameLowerCase":{"order":"asc"}}],"track_total_hits":true} Initial filter: {"PageIndex":0,"PageSize":1,"SortField":"BuyerName","SortAscending":true,"Filter":{"MasterCorporateGroupId":{"Value":["c5268dd2-c389-45df-a978-c9793bec8999"],"Enabled":true},"Id":{"Value":"34091917-a0a5-41ca-8254-686e722a8200","Enabled":true},"Name":{"Value":null,"Enabled":false},"NameExactMatch":{"Value":null,"Enabled":false},"BuyerName":{"Value":null,"Enabled":false},"BuyerNameExactMatch":{"Value":null,"Enabled":false},"SupplierNumber":{"Value":null,"Enabled":false},"SupplierNumberExactMatch":{"Value":null,"Enabled":false},"Description":{"Value":null,"Enabled":false},"DescriptionExactMatch":{"Value":null,"Enabled":false},"TaxRegistrationNumber":{"Value":null,"Enabled":false},"TaxRegistrationNumberExactMatch":{"Value":null,"Enabled":false},"OrganizationNumber":{"Value":null,"Enabled":false},"OrganizationNumberExactMatch":{"Value":null,"Enabled":false},"Street":{"Value":null,"Enabled":false},"StreetExactMatch":{"Value":null,"Enabled":false},"PostalCode":{"Value":null,"Enabled":false},"PostalCodeExactMatch":{"Value":null,"Enabled":false},"City":{"Value":null,"Enabled":false},"CityExactMatch":{"Value":null,"Enabled":false},"CountryName":{"Value":null,"Enabled":false},"CountryNameExactMatch":{"Value":null,"Enabled":false},"PaymentTerm":{"Value":null,"Enabled":false},"PaymentTermExactMatch":{"Value":null,"Enabled":false},"PaymentMethod":{"Value":null,"Enabled":false},"PaymentMethodExactMatch":{"Value":null,"Enabled":false},"CurrencyCode":{"Value":null,"Enabled":false},"CurrencyCodeExactMatch":{"Value":null,"Enabled":false},"Location":{"Value":null,"Enabled":false},"LocationExactMatch":{"Value":null,"Enabled":false},"State":{"Value":null,"Enabled":false},"StateExactMatch":{"Value":null,"Enabled":false},"Blocked":{"Value":false,"Enabled":false},"TelephoneNumber":{"Value":null,"Enabled":false},"TelephoneNumberExactMatch":{"Value":null,"Enabled":false},"FaxNumber":{"Value":null,"Enabled":false},"FaxNumberExactMatch":{"Value":null,"Enabled":false},"TaxCode":{"Value":null,"Enabled":false},"TaxCodeExactMatch":{"Value":null,"Enabled":false},"BankAccountsCollection":{"Enabled":false,"BankAccountFilters":[]},"FreeQuery":{"Value":null,"Enabled":false},"BypassVerification":{"Value":false,"Enabled":false},"ExcludeFromIdentification":{"Value":false,"Enabled":false},"LockOnlineLearning":{"Value":false,"Enabled":false}},"IncludeBankAccounts":true,"ScoreBasedSorting":false,"UseSearchAfter":false}

I still believe you should somewhere have elasticsearch logs. Although I'm not using any cloud service (we use on prem).

What is the IP "20.123.75.169"? Is it the same host as you are getting the error on? Did you try localhost or 0.0.0.0 in that case?
I see the IP is coming from below hostname. Is the dns mapping correct?

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 20.123.75.169:9200

Request: POST https://aps-prd-eu-es-ap2.northeurope.cloudapp.azure.com:9200/masterdata-index-supplier2-eu/_search?typed_keys=true

Could you elaborate on what you are trying to achieve. I understand your elasticsearch is returning errors. What is trying to connect? Is it running on the same machine? ...?

Is your http.port bound to a range? Or a static value?

Christof

1 Like
8

Hi Christof,

Hope this email finds you well!

I found logs at the path %ES_HOME%\logs indicating that auditing logging is DISABLED due to the current [BASIC] license not permitting it.

The public IP of the host 20.123.75.169, where our Elasticsearch (ES) is hosted, is encountering this issue. I have already opened a Microsoft Azure ticket regarding the DNS mapping. If you can provide guidance on what needs to be checked, I can request further investigation from them.

Our goal is to determine if this error is affecting our productivity.

9

=> Your set up is still not clear. Is it just a standalone elasticsearch in Azure cloud?

10

Correct it is hosted only in Azure cloud.