Metricbeat 7.3.0 add_host_metadata

Elastic Stack Beats
,
1

I am trying to implement a way to get host meta data pushed from my clients.

I have followed this link, however I do not see any fields get added to the event.

Here is my current metricbeat.yml configuration file:

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  reload.period: 30s

processors:
- add_host_metadata:
    netinfo.enabled: true
    cache.ttl: -1
    geo:
      name: nyc-dc1-rack1
      location: 40.7128, -74.0060
      continent_name: North America
      country_iso_code: US
      region_name: New York
      region_iso_code: NY
      city_name: New York

#
# The following example enriches each event with process metadata using
# process IDs included in the event.
#
#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["server:5044"]

Here is the event in Kibana:

{
  "_index": "cltb-metricbeat-7.3.2-2019.12.03",
  "_type": "_doc",
  "_id": "CLhNzG4B1KXevdxWHMwB",
  "_version": 1,
  "_score": null,
  "_source": {
    "model": "HP EliteBook 840 G5",
    "@timestamp": "2019-12-03T15:07:14.639Z",
    "agent": {
      "version": "7.3.2",
      "type": "metricbeat",
      "hostname": "host",
      "id": "0a9abfdb-402a-46ae-bc97-2ad18134d3ff",
      "ephemeral_id": "b509fe23-4983-49cc-a0eb-06eb1ff25daf"
    },
    "metricset": {
      "name": "process"
    },
    "event": {
      "duration": 51043100,
      "dataset": "system.process",
      "module": "system"
    },
    "systemtype": "x64-based PC",
    "user": {
      "name": "NT AUTHORITY\\NETWORK SERVICE"
    },
    "os_caption": "Microsoft Windows 10 Enterprise",
    "@version": "1",
    "tags": [
      "beats_input_raw_event"
    ],
    "sentry_earlyadopter": "Early Adopters - Ring 0 - IT Client",
    "os_version": "10.0.17763",
    "service": {
      "type": "system"
    },
    "system": {
      "process": {
        "cpu": {
          "start_time": "2019-12-03T13:09:08.379Z",
          "total": {
            "pct": 0.0907,
            "value": 163765,
            "norm": {
              "pct": 0.0113
            }
          }
        },
        "state": "running",
        "memory": {
          "rss": {
            "pct": 0.0005,
            "bytes": 8204288
          },
          "size": 3375104,
          "share": 0
        },
        "cmdline": "C:\\WINDOWS\\system32\\svchost.exe -k NetworkService -p -s Dnscache"
      }
    },
    "os_buildnumber": "17763",
    "ecs": {
      "version": "1.0.1"
    },
    "process": {
      "args": [
        "C:\\WINDOWS\\system32\\svchost.exe",
        "-k",
        "NetworkService",
        "-p",
        "-s",
        "Dnscache"
      ],
      "pgid": 0,
      "ppid": 948,
      "pid": 1844,
      "name": "svchost.exe"
    }
  },
  "fields": {
    "system.process.cpu.start_time": [
      "2019-12-03T13:09:08.379Z"
    ],
    "@timestamp": [
      "2019-12-03T15:07:14.639Z"
    ]
  },
  "highlight": {
    "agent.hostname": [
      "@kibana-highlighted-field@host@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1575385634639
  ]
}

Is there something I have to do in logstash to get these fields to get populated with the relevant host data? I tried removing my enrichment logic in logstash and it appears to be showing up now:

{
  "_index": "cltb-metricbeat-7.3.2-2019.12.03",
  "_type": "_doc",
  "_id": "1sVhzG4By8QVVSUY8QaY",
  "_version": 1,
  "_score": null,
  "_source": {
    "event": {
      "module": "windows",
      "dataset": "windows.service",
      "duration": 202883300
    },
    "@timestamp": "2019-12-03T15:29:59.929Z",
    "host": {
      "architecture": "x86_64",
      "ip": [
        "fe80::d55c:7b6:e4a9:75ce",
        "10.3.87.127",
        "fe80::2827:c72f:c34:a5fb",
        "169.254.165.251",
        "fe80::55bc:c197:969c:d531",
        "169.254.213.49",
        "fe80::75:adc8:a95b:a3f8",
        "169.254.163.248",
        "fe80::5072:8bd2:f703:4252",
        "169.254.66.82"
      ],
      "geo": {
        "region_iso_code": "NY",
        "name": "nyc-dc1-rack1",
        "region_name": "New York",
        "country_iso_code": "US",
        "continent_name": "North America",
        "location": "40.7128, -74.0060",
        "city_name": "New York"
      },
      "id": "14f4ba81-438a-4e9e-9aa6-dd9ef59a0221",
      "name": "host",
      "hostname": "host",
      "mac": [
        "f4:39:09:d8:95:d2",
        "3c:6a:a7:8e:9b:36",
        "3e:6a:a7:8e:9b:35",
        "3c:6a:a7:8e:9b:35",
        "3c:6a:a7:8e:9b:39"
      ],
      "os": {
        "name": "Windows 10 Enterprise",
        "build": "17763.864",
        "kernel": "10.0.17763.864 (WinBuild.160101.0800)",
        "platform": "windows",
        "version": "10.0",
        "family": "windows"
      }
    },
    "windows": {
      "service": {
        "start_name": "LocalSystem",
        "start_type": "Automatic",
        "id": "mEgkPHPp8j",
        "state": "Running",
        "name": "AdobeARMservice",
        "uptime": {
          "ms": 8449984
        },
        "path_name": "\"C:\\Program Files (x86)\\Common Files\\Adobe\\ARM\\1.0\\armsvc.exe\"",
        "display_name": "Adobe Acrobat Update Service",
        "pid": 5260
      }
    },
    "metricset": {
      "name": "service"
    },
    "ecs": {
      "version": "1.0.1"
    },
    "@version": "1",
    "agent": {
      "ephemeral_id": "a5441a1f-a09e-43bf-b080-44f54238d9b4",
      "id": "0a9abfdb-402a-46ae-bc97-2ad18134d3ff",
      "hostname": "host",
      "type": "metricbeat",
      "version": "7.3.2"
    },
    "service": {
      "type": "windows"
    },
    "tags": [
      "beats_input_raw_event"
    ]
  },
  "fields": {
    "@timestamp": [
      "2019-12-03T15:29:59.929Z"
    ]
  },
  "highlight": {
    "agent.hostname": [
      "@kibana-highlighted-field@host@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1575386999929
  ]
}

Is there anyway to just send active network adapter? Or will this require some logstash parsing?

Thanks,
Raged

2

Hey @raged,

I'm not quite sure what you changed in the "enrichment logic in logstash" to make it work. Can you please post before and after fix, so I can try to reproduce it on my end.

Thank you!

3

Here it is before:

input {
	beats {
		port => 5044
		client_inactivity_timeout => 300
	}
}

filter {
  mutate {
    add_field => { "ip_addr" => "%{[@metadata][ip_address]}" }
  }
}

filter {
  jdbc_static {
    loaders => [ 
      {
        id => "remote-clients"
        query => "select distinct v_R_System.Name0 As 'machinename', v_Collection.Name As collectionname,v_FullCollectionMembership.CollectionID As 'collectionid' from v_FullCollectionMembership JOIN v_R_System on v_FullCollectionMembership.ResourceID = v_R_System.ResourceID JOIN v_Collection on v_FullCollectionMembership.CollectionID = v_Collection.CollectionID where v_FullCollectionMembership.CollectionID IN ('SEP00207','SEP0040E','SEP00208','SEP0040F','SEP00209','SEP0041C')"
        local_table => "clients"
      },
      {
        id => "remote-client-details"
        query => "SELECT cs.name0 as 'name',cs.model0 as 'model',cs.systemtype0 as 'systemtype',cs.resourceid as 'resourceid',os.caption0 as 'oscaption',os.BuildNumber0 as 'osbuildnumber',os.version0 as 'osversion' from [CM_SEP].[dbo].[v_GS_COMPUTER_SYSTEM] cs INNER JOIN [CM_SEP].[dbo].[v_GS_OPERATING_SYSTEM] OS on cs.ResourceId = os.resourceid where cs.name0 IS NOT NULL"
        local_table => "details"
      }      
    ]
    local_db_objects => [ 
      {
        name => "clients"
        index_columns => ["machinename"]
        columns => [
          ["machinename", "varchar(30)"],
		  		["collectionname", "varchar(100)"]
        ]
      },
      {
        name => "details"
        index_columns => ["name"]
        columns => [
          ["name", "varchar(255)"],
          ["model", "varchar(255)"],
          ["systemtype", "varchar(255)"],
          ["resourceid", "varchar(255)"],
          ["oscaption", "varchar(255)"],
          ["osbuildnumber", "varchar(255)"],
          ["osversion", "varchar(255)"]
        ]
      }      
    ]
    local_lookups => [ 
      {
        id => "local-clients"
        query => "select machinename,collectionname from clients WHERE machinename = :beat_name"
        parameters => { beat_name => "[host][name]" }
        target => "client"
	default_hash => {
	  "sentry_earlyadopter" => "NO"
	}
      },
      {
        id => "local-details"
        query => "select name, model, systemtype, resourceid, oscaption, osbuildnumber, osversion from details WHERE name = :beat_name"
        parameters => { beat_name => "[host][name]" }
        target => "details" 
      }      
    ]   

		add_field => { "[sentry_earlyadopter]" => "%{[client][0][collectionname]}" }
    add_field => { "[model]" => "%{[details][0][model]}" }
    add_field => { "[systemtype]" => "%{[details][0][systemtype]}" }
    add_field => { "[os_caption]" => "%{[details][0][oscaption]}" }
    add_field => { "[os_buildnumber]" => "%{[details][0][osbuildnumber]}" }
    add_field => { "[os_version]" => "%{[details][0][osversion]}" }   
    remove_field => "[details]"  
    remove_field => "[client]" 
    jdbc_user => "user"
    jdbc_password => "pass"
    jdbc_driver_class => "Java::net.sourceforge.jtds.jdbc.Driver"
    jdbc_connection_string => "jdbc:jtds:sqlserver://server"
    jdbc_driver_library => "/usr/share/logstash/jtds-1.3.1-dist/jtds-1.3.1.jar"
    loader_schedule => "35 * * * *"
  }
}

filter {
  if [sentry_earlyadopter] =~ /.*collectionname.*/ {
    mutate {
      update => { "sentry_earlyadopter" => "NO"}
    }
  }

  if [event_id] == 41 {
    ruby { code => 'event.set("bugcheckcode_hex", event.get("[event_data][BugcheckCode]").to_i.to_s(16))' }
  }

if [event_data][ProcessId] =~ /.*0x.*/ {
    ruby { code => 'event.set("[event_data][ProcessId]", event.get("[event_data][ProcessId]").to_i(16))' }  
}

  mutate {
    update => { "bugcheckcode_hex" => "0x%{[bugcheckcode_hex]}" }
    remove_field => ["host"]
  }

if [@metadata][beat] =~ /.*filebeat.*/ {
    grok {
      match => { "message" => '\<\!\[LOG\[[+\+ ]+(?<Message>.*)?\]LOG\]\!\>\<time=\"(?<Time>.+)(?<TZAdjust>[+|-])(?<TZOffset>\d{2,3})\"\s+date=\"(?<Date>.+)?\"\s+component=\"(?<Component>.+)?\"\s+context="(?<Context>.*)?\"\s+type=\"(?<Type>\d)?\"\s+thread=\"(?<TID>\d+)?\"\s+file=\"(?<Reference>.+)?\"\>' }
      match => { "message" => '%{TIMESTAMP_ISO8601:TimeStamp} %{WORD:Action} %{WORD:Prot} %{IP:SrcIP} %{IP:DstIP} %{INT:SrcPort} %{INT:DstPort} %{INT:Size} %{GREEDYDATA:Flags} %{WORD:Path}' }
    }
    mutate {
      remove_field => ["message"]
    }
}

}

output {
  if "_grokparsefailure" not in [tags] {
	elasticsearch {
		hosts => ["host:9200"]
		truststore => "/etc/logstash/certs/cacerts"
		truststore_password => "changeit"
		index => "cltb-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
		user => user
		password => pass
	}
  }
}

Here it is afterward:

input {
	beats {
		port => 5044
		client_inactivity_timeout => 300
	}
}

output {
  if "_grokparsefailure" not in [tags] {
	elasticsearch {
		hosts => ["host1:9200"]
		truststore => "/etc/logstash/certs/cacerts"
		truststore_password => "changeit"
		index => "cltb-%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
		user => user
		password => pass
	}
  }
}

I have a feeling this line was removing it:

  mutate {
    update => { "bugcheckcode_hex" => "0x%{[bugcheckcode_hex]}" }
    remove_field => ["host"]
  }

EDIT, yes it was. Removing that remove_field resolves it. Wow.

Well, off to parsing the valid IP.

Thanks for reading.

Raged

1 Like
4

Glad you found a solution!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.