Metricbeat Active Directory (AD) performance metrics (perfmon) yaml build

Ryan_Downey · November 27, 2019, 1:21pm

I'm looking for a little guidance on how to build out the windows.yml module in Metricbeat. If this is being built out correctly we may need 3-4 lines to ship the Active Directroy perfmon out. The issue that I'm having is that I'm not able to find a lot of specified info on AD perfmon so if anyone has any links that I could be missing that would be appreciated. From my research it seems like these perfmon counters are also under NTDS so I'm not sure how this fits into the syntax as well. I should be able to replicate what the memory section but that might not necessarily be the case. Overall were trying to replicate some Splunk dashboards in Kibana but there isnt a specific app like there is in Splunk.

# ########## Memory
    - instance_label: memory.name      
      instance_name: memory.page.total.reads_sec
      measurement_label: memory.page.total.reads_sec
      query: '\Memory\Page Reads/sec'

# ########## Active Directory
    - instance_label: ntds.name
      instance_name: ntds.ds.directory.reads_sec
      measurement_label: ?
      query: '\DS Directory Reads/sec'

Mario_Castro · November 27, 2019, 4:41pm

Hi @Ryan_Downey

Let me summon @MarianaD that maybe she knows something about this (definitely she knows more than me )

Ryan_Downey · November 27, 2019, 4:44pm

Cool, I'm really just not sure whats needed and whats not. Obviously the query part is needed but other than that I'm a bit lost. I also don't have access to the servers that we'd be running this on so I'm trying to be prepared when I hand off the info to another agency to implement and test. Otherwise I'd just type things in and see what works :).

MarianaD · November 28, 2019, 9:54am

hi @Ryan_Downey, this is an interesting approach. It does seem that the NTDS performance object counters are a good start. I have not tried this yet but we are interested to have AD supported in Metricbeat.
I found https://www.itprotoday.com/active-directory/jsi-tip-5454-how-do-i-monitor-performance-active-directory which lists the NTDS related counters and provides more information on how to enable it.
I have also opened https://github.com/elastic/beats/issues/14836 to explore this idea further.
Feel free to add your thoughts there.

Ryan_Downey · December 2, 2019, 1:38pm

Mariana,

A little delayed on my response due to the holiday break. I appreciate you starting the GitHub thread for me and for looking into this. Due to the pricing structure of Splunk vs Elasticsearch I could foresee a lot of people that want to port similar functionality out of Splunk. I havent taken a deep dive into what other Splunk apps are out there, metrics wise, but it could make the transition easier for people if some of the functionality was mirrored. At the time being I've got a basic metricbeat perfmon counter as;

- instance_label: ntds.name
      instance_name: ntds.ds_directory.reads_sec
      measurement_label: ntds.ds_directory.reads_sec
      query: '\NTDS\DS Directory Reads/sec'

I'm going to give that a try early this week and see what happens.

mazoutte · December 2, 2019, 3:58pm

Hello, the link mentionned is very useful, and quite complete.
For AD , you pointed out the intersesting section "DirectoryServices" for the counters.
More counters : DNS and DFS section (instance SYSVOL related)

-> Interesting counters : 
    \Security System-Wide Statistics\Digest Authentications
    \Security System-Wide Statistics\KDC AS Requests
    \Security System-Wide Statistics\KDC TGS Requests
    \Security System-Wide Statistics\Kerberos Authentications
    \Security System-Wide Statistics\NTLM Authentications

I was using this below config file for some tests with metribeat.
I need to warn you, that actual Metricbeat is creating 1 document per counter.
With this config, for 6 Domain Controllers, pulling every 5 seconds, the daily index size is around 290-320 MB for 3.2M documents.
Extending this to a big environnement, it can be weird. (more than 100 Domain Controllers, and adding all the counters from the DirectoryServices/DNS/DFS/System sections)
Unless the metricbeat agent can create only 1 document per "perfmon pull", you must be aware of that (but it's another subject actually).

  # NTDS
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.anr.sec"
  query: '\DirectoryServices(NTDS)\AB ANR/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.browses.sec"
  query: '\DirectoryServices(NTDS)\AB Browses/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.client.sessions"
  query: '\DirectoryServices(NTDS)\AB Client Sessions'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.matches.sec"
  query: '\DirectoryServices(NTDS)\AB Matches/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.prop.reads.sec"
  query: '\DirectoryServices(NTDS)\AB Property Reads/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.proxy.lookups.sec"
  query: '\DirectoryServices(NTDS)\AB Proxy Lookups/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ab.searches.sec"
  query: '\DirectoryServices(NTDS)\AB Searches/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.db.searches.sec"
  query: '\DirectoryServices(NTDS)\Base searches/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.db.adds.sec"
  query: '\DirectoryServices(NTDS)\Database adds/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.db.deletes.sec"
  query: '\DirectoryServices(NTDS)\Database deletes/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.db.modifys.sec"
  query: '\DirectoryServices(NTDS)\Database modifys/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.db.recycles.sec"
  query: '\DirectoryServices(NTDS)\Database recycles/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ds.client.binds.sec"
  query: '\DirectoryServices(NTDS)\DS Client Binds/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ds.client.name.translations.sec"
  query: '\DirectoryServices(NTDS)\DS Client Name Translations/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ds.directory.reads.sec"
  query: '\DirectoryServices(NTDS)\DS Directory Reads/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ds.directory.searches.sec"
  query: '\DirectoryServices(NTDS)\DS Directory Searches/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ds.directory.writes.sec"
  query: '\DirectoryServices(NTDS)\DS Directory Writes/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ds.threads"
  query: '\DirectoryServices(NTDS)\DS Threads in Use'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.active.threads"
  query: '\DirectoryServices(NTDS)\LDAP Active Threads'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.bind.time"
  query: '\DirectoryServices(NTDS)\LDAP Bind Time'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.client.sessions"
  query: '\DirectoryServices(NTDS)\LDAP Client Sessions'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.closed.connections.sec"
  query: '\DirectoryServices(NTDS)\LDAP Closed Connections/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.new.connections.sec"
  query: '\DirectoryServices(NTDS)\LDAP New Connections/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.new.ssl.connections.sec"
  query: '\DirectoryServices(NTDS)\LDAP New SSL Connections/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.searches.sec"
  query: '\DirectoryServices(NTDS)\LDAP Searches/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.successful.binds.sec"
  query: '\DirectoryServices(NTDS)\LDAP Successful Binds/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.udp.operations.sec"
  query: '\DirectoryServices(NTDS)\LDAP UDP operations/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ldap.writes.sec"
  query: '\DirectoryServices(NTDS)\LDAP Writes/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.ntlm.binds.sec"
  query: '\DirectoryServices(NTDS)\NTLM Binds/sec'
- instance_label: "ntds.name"
  instance_name: "NTDS"
  measurement_label: "ntds.subtree.searches.sec"
  query: '\DirectoryServices(NTDS)\Subtree searches/sec'

Ryan_Downey · December 3, 2019, 2:27pm

@mazoutte Thank you for taking the time to help me out with this and for the detailed explanation of what to watch out for. We should be deploying this configuration early this week and will follow up with our results.

Ryan_Downey · December 6, 2019, 6:58pm

We've got a lot of different agencies that could potentially be looking for lots of different counters. Whats the best way to allow the Elasticsearch community to add to the windows.yml in regards to the counters? It'd be nice if we started a yaml and people just slowly added to it so if someone was looking for ntds, processor, server, etc they'd have a completed list to chose from. Obviously willing to add what I have in ours to start.

mazoutte · December 12, 2019, 4:25pm

Hi Ryan,

Your post gave me the idea to put my hands back on this metricbeat subject.
In the past I wasn't able to have 1 single document with all the metrics for NTDS.
With the 7.x metricbeat agent version, I was able to make it working as I wanted.

So here is the 'complete' config file 'windows.yml' with all the NTDS counters found on a 2012r2 DCs and metricbeat agent 7.4.2.

I use "instance_name" NTDS to have all counters in 1 document.
You can add counters for CPU/Memory/Network ... on the same instance name or use another instance name if you prefer to have seperate documents for different category of counters.

I put some examples with some DNS and System Counters, with this config file , you will have 3 documents per pull per metricbeat agent.

Note : 1 counter is not pulled, and doesn't seem to be working on the server "DRA Threads Getting NC Changes Holding Semaphore".
It's NOT related to metricbeat, but to the OS itself (this behavior happens on other counters like DNS).

gist.github.com

https://gist.github.com/mazoutte/875b3d4cbdae0c1a416b75ee2f3208df

windows.yml

- module: windows
  metricsets: ["perfmon"]
  period: 5s
  perfmon.ignore_non_existent_counters: true
  perfmon.group_measurements_by_instance: true
  perfmon.counters:
      # NTDS
    - instance_label: "instance.name"
      instance_name: "NTDS"
      measurement_label: "ntds.ab.anr.sec"

This file has been truncated. show original

Ryan_Downey · December 12, 2019, 5:45pm

@mazoutte Thanks for following up on this! I'm sure this took a while to put together and I appreciate the extra effort. This is looking pretty good on our end with two hosts reporting the NTDS data and many more servers to come after that. We were able to replicate the Splunk dashboards that the customer would like fairly easily which was a bonus. Each one needed some small tweaks but nothing too major.

system · January 9, 2020, 5:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.