Audit logs stop on certain nodes in cluster

Elastic Stack Elasticsearch
1

Hello!

I have a production cluster running 20 nodes that I would like to configure audit logs for. Everything seems to be running correctly on 3 nodes, on the rest it stops logging events. When I restart the service on that particular node, it logs for a while and then stops. Am I missing something here? How should I debug this?

The version is 7.16.2

Thanks

2

Hello @ryuujin , welcome to the community !
Have you compared the last logs that can be seen in ES index with the actual log messages present in audit log files for those servers where it is missing ? It could be the case that other nodes are handling any request hence no logs are generated.

3

I think you are right, thank you for pointing me in this direction. So if I understand this correctly, normally requests can be handled by a small number of nodes, and the rest is just transporting shards between each other and does not need to handle user requests?

4

Well, its partially correct. Essentially the bulk writing request is handled by nodes which have primary shards for the target indices.
However, when it comes to search requests, all the nodes which have either primary or replica shards are involved to lookup for requested data. The result sets are then aggregated and sent as response by coordinator node, which happens to be the center of accepting and forwarding any request hitting ES cluster.

5

Thank you for explaining. So when a node is chosen as a coordinator node, it will then handle any incoming requests, and there is no need to handle anything explicitly. I should just aggregate audit logs from all nodes, because there is a chance that coordinator role will be assigned to any of the data nodes. Or are only master eligible nodes chosen for that?

6

By default all nodes are coordinator nodes but you explicitly set node roles for this as well.

7

Welcome to our community! :smiley:

You should/could reach out to the Support team to ask about this as well.

8

Thank you Ayush for clearing it up for me, I think this is enough :slight_smile: If I have some concerns about this, I will reach out to support.

Have a nice day!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.