How to identify the ip address of application which sends the bulk updates to elasticsearch cluster?

Elastic Stack Elasticsearch
1

How to identify the IP address of the application which sends the bulk updates to elasticsearch cluster?

2

Can you elaborate?
Your question is missing crucial information.

3

Have a look at Enable audit logging | Elasticsearch Guide [7.12] | Elastic

1 Like
4

Thank you for your response.

Cluster configuration:

ES Version: 2.4
Master node: 5
Data node: 17
Number of indices: 1 (5 primary and 10 replica shards).

So, one shard is allocated to one data node at any point of time.

We are having performance issue due to high load on one of our data nodes. We identified it is due to high number of bulk Insert operation (probably UPDATE API).

When we move the shard from problematic node to a data node with no shard on it, the load does not transfer to the destination node (after the move). Instead some other node picks up the load. We always see the load on the node which hosts replica shard.

There are several app servers connecting to the es cluster using transport client and it uses sniffing to connect to data node.

We want to identify which app server is doing all the bulk Insert operation.

cc: @dadoonet

5

That's a way too old. You should upgrade.

You don't have that information within the documents?

6
  1. We are planning to upgrade es to latest version.
  2. No, the document doesn't have information on which app server updates it.
7

So documents are coming from different apps to the same index.

Could you describe a bit the use case so we have a better picture?

How do you use then the indexed data? What kind of search are you running from each app?

Is it the same app which is deployed on multiple servers?

8

Yes, documents are coming from different apps to the same indices. Same app is deployed on multiple servers. I am not sure about the use case,

One other issue I have noticed is the index queue is hitting beyond 2k (limit set up in .yml and cluster settings). It's was approximately 300k (index queue). Do we know why es is not honoring the index queue limit?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.