I'm looking for a little guidance on how to build out the windows.yml module in Metricbeat. If this is being built out correctly we may need 3-4 lines to ship the Active Directroy perfmon out. The issue that I'm having is that I'm not able to find a lot of specified info on AD perfmon so if anyone has any links that I could be missing that would be appreciated. From my research it seems like these perfmon counters are also under NTDS so I'm not sure how this fits into the syntax as well. I should be able to replicate what the memory section but that might not necessarily be the case. Overall were trying to replicate some Splunk dashboards in Kibana but there isnt a specific app like there is in Splunk.
Cool, I'm really just not sure whats needed and whats not. Obviously the query part is needed but other than that I'm a bit lost. I also don't have access to the servers that we'd be running this on so I'm trying to be prepared when I hand off the info to another agency to implement and test. Otherwise I'd just type things in and see what works :).
A little delayed on my response due to the holiday break. I appreciate you starting the GitHub thread for me and for looking into this. Due to the pricing structure of Splunk vs Elasticsearch I could foresee a lot of people that want to port similar functionality out of Splunk. I havent taken a deep dive into what other Splunk apps are out there, metrics wise, but it could make the transition easier for people if some of the functionality was mirrored. At the time being I've got a basic metricbeat perfmon counter as;
Hello, the link mentionned is very useful, and quite complete.
For AD , you pointed out the intersesting section "DirectoryServices" for the counters.
More counters : DNS and DFS section (instance SYSVOL related)
I was using this below config file for some tests with metribeat.
I need to warn you, that actual Metricbeat is creating 1 document per counter.
With this config, for 6 Domain Controllers, pulling every 5 seconds, the daily index size is around 290-320 MB for 3.2M documents.
Extending this to a big environnement, it can be weird. (more than 100 Domain Controllers, and adding all the counters from the DirectoryServices/DNS/DFS/System sections)
Unless the metricbeat agent can create only 1 document per "perfmon pull", you must be aware of that (but it's another subject actually).
@mazoutte Thank you for taking the time to help me out with this and for the detailed explanation of what to watch out for. We should be deploying this configuration early this week and will follow up with our results.
We've got a lot of different agencies that could potentially be looking for lots of different counters. Whats the best way to allow the Elasticsearch community to add to the windows.yml in regards to the counters? It'd be nice if we started a yaml and people just slowly added to it so if someone was looking for ntds, processor, server, etc they'd have a completed list to chose from. Obviously willing to add what I have in ours to start.
Your post gave me the idea to put my hands back on this metricbeat subject.
In the past I wasn't able to have 1 single document with all the metrics for NTDS.
With the 7.x metricbeat agent version, I was able to make it working as I wanted.
So here is the 'complete' config file 'windows.yml' with all the NTDS counters found on a 2012r2 DCs and metricbeat agent 7.4.2.
I use "instance_name" NTDS to have all counters in 1 document.
You can add counters for CPU/Memory/Network ... on the same instance name or use another instance name if you prefer to have seperate documents for different category of counters.
I put some examples with some DNS and System Counters, with this config file , you will have 3 documents per pull per metricbeat agent.
Note : 1 counter is not pulled, and doesn't seem to be working on the server "DRA Threads Getting NC Changes Holding Semaphore".
It's NOT related to metricbeat, but to the OS itself (this behavior happens on other counters like DNS).
@mazoutte Thanks for following up on this! I'm sure this took a while to put together and I appreciate the extra effort. This is looking pretty good on our end with two hosts reporting the NTDS data and many more servers to come after that. We were able to replicate the Splunk dashboards that the customer would like fairly easily which was a bonus. Each one needed some small tweaks but nothing too major.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.