Metricbeat 7.9.2 service doesn't start on windows 2019, but fine on linux

Elastic Stack Beats
1

Attempting a new setup of Metricbeat on a fresh windows 2019 server, and getting error 1067 ("The process terminated unsuccessfully) from services.msc. Powershell startup is essentially the same result.

Setup:

  • The files, etc were put in place via extraction from the zip file (not the msi, as i understand that's still in beta).
  • The service installer run (via install-service-metricbeat.ps1) was successful in terms of not throwing errors; the service showed up afterwards in "stopped" mode.
  • The service is currently configured under the "Local System" account, writing to an elastic cloud instance.
  • The metricbeat.yml and system.yml files i'm using are the same as those i used on a centos 7 server earlier in the day, with the exception of the "load" metric, as I understand that's unsupported. The centos instance of metricbeat had no problems starting up or staying alive as a systemd service.

Debugging steps i've taken:

  • if you pull the startup command from the service def and execute it in a cmd window w/ administrative privileges, it runs in this "live" mode, as such:

C:\Program Files\metricbeat>"C:\Program Files\metricbeat\metricbeat.exe" --environment=windows_service -c "C:\Program Files\metricbeat\metricbeat.yml" --path.home "C:\Program Files\metricbeat" --path.data "C:\ProgramData\metricbeat" --path.logs "C:\ProgramData\metricbeat\logs" -E logging.files.redirect_stderr=true

This runs successfully for as long as i'm willing to let it go in the window, and i can see output on the other side in kibana.

  • i've attempted changing the compatibility settings of metricbeat.exe to implicitly try to run as administrator, but it still makes no difference on the service startup.

  • i've attempted to install metricbeat 7.3, as the perf index i'm writing to is of that original vintage, but it doesn't change the final result.

  • all "metricbeat test " options return 'OK' values.

  • EDIT/UPDATE: i've also tried both the service-launched and the cmd-launched paths with the log level of "debug"; no ERROR values are created in the metricbeat log file.

I'm really at a loss, so I thought i'd ask here. Are there known compat issues with windows 2019? Is there something i'm missing? I did see this:

image
image482×632 15.8 KB

but after unchecking the block, it still didn't work as a service. Again, it starts fine when run directly, so it seems like there's a disconnect in terms of what the windows service infrastructure is expecting vs what the executable is providing.

For reference, the only module i have enabled right now is "system", and here are the uncommented portions of the relevant files:

system.yml:

    - module: system
      period: 30s
      metricsets:
        - cpu
        #- load
        - memory
        - network
        - diskio
        - uptime
        - filesystem
      cpu.metrics:
        - percentages

metricbeat.yml:

    cloud.id: "my cloud auth id"
    cloud.auth: "my cloud username and password"

    setup.template.name: "perf-metricbeat-7.3.1"
    setup.template.pattern: "perf-metricbeat-7.3.1-*"

    setup.ilm.enabled: true
    setup.ilm.rollover_alias: "perf-metricbeat-7.3.1"
    setup.ilm.policy_name: "metrics"
    setup.ilm.check_exists: true


    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    # ---------------------------- Elasticsearch Output ----------------------------
    output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["https://longrandomawsguidstring.us-east-1.aws.found.io:443"]

Any help would be appreciated

2

Hi,

what do you see in your cmd when you just type metricbeat.exe ?

3

If i just run metricbeat.exe, it appears to run in the background; no console output appears, but it appears to be submitting values upstream.

If i throw a -e flag in, it does the same, but visibly...here's the output from both:

    C:\Program Files\metricbeat>metricbeat.exe
    (implied ctrl-c after about two minutes)

    C:\Program Files\metricbeat>metricbeat.exe -e
    2020-10-22T23:04:14.276-0400    INFO    instance/beat.go:640    Home path: [C:\Program Files\metricbeat] Config path: [C:\Program Files\metricbeat] Data path: [C:\Program Files\metricbeat\data] Logs path: [C:\Program Files\metricbeat\logs]
    2020-10-22T23:04:14.287-0400    INFO    instance/beat.go:648    Beat ID: b38acc5f-da24-421d-847d-b52d34b541e9
    2020-10-22T23:04:14.289-0400    INFO    [beat]  instance/beat.go:976    Beat info       {"system_info": {"beat": {"path": {"config": "C:\\Program Files\\metricbeat", "data": "C:\\Program Files\\metricbeat\\data", "home": "C:\\Program Files\\metricbeat", "logs": "C:\\Program Files\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "b38acc5f-da24-421d-847d-b52d34b541e9"}}}
    2020-10-22T23:04:14.289-0400    INFO    [beat]  instance/beat.go:985    Build info      {"system_info": {"build": {"commit": "2ab907f5ccecf9fd82fe37105082e89fd871f684", "libbeat": "7.9.2", "time": "2020-09-22T23:25:16.000Z", "version": "7.9.2"}}}
    2020-10-22T23:04:14.290-0400    INFO    [beat]  instance/beat.go:988    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.14.7"}}}
    2020-10-22T23:04:14.300-0400    INFO    [beat]  instance/beat.go:992    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-10-21T16:55:07.69-04:00","name":"FRUNIDEPWEB01","ip":["172.28.32.20/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.1457 (WinBuild.160101.0800)","mac":["00:50:56:a3:09:dd"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2019 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.1457"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"f6d317ff-566a-47c4-bd51-4d46c04db001"}}}
    2020-10-22T23:04:14.301-0400    INFO    [beat]  instance/beat.go:1021   Process info    {"system_info": {"process": {"cwd": "C:\\Program Files\\metricbeat", "exe": "C:\\Program Files\\metricbeat\\metricbeat.exe", "name": "metricbeat.exe", "pid": 5516, "ppid": 8108, "start_time": "2020-10-22T23:04:13.555-0400"}}}
    2020-10-22T23:04:14.302-0400    INFO    instance/beat.go:299    Setup Beat: metricbeat; Version: 7.9.2
    2020-10-22T23:04:14.302-0400    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'perf-metricbeat-7.3.1' as ILM is enabled.
    2020-10-22T23:04:14.303-0400    INFO    eslegclient/connection.go:99    elasticsearch url: https://longrandomawsguidstring.us-east-1.aws.found.io:443
    2020-10-22T23:04:14.304-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: FRUNIDEPWEB01
    2020-10-22T23:04:14.357-0400    INFO    instance/beat.go:450    metricbeat start running.
    2020-10-22T23:04:14.357-0400    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
    2020-10-22T23:04:14.361-0400    INFO    helper/privileges_windows.go:79 Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":8,"User":{"SID":"S-1-5-21-2642810355-4001929803-2508295686-14775","Account":"jmacnett","Domain":"EXPOEXCHANGE","Type":1},"ProcessPrivs":{"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled":false},"SeCreateSymbolicLinkPrivilege":{"enabled":false},"SeDebugPrivilege":{"enabled":false},"SeDelegateSessionUserImpersonatePrivilege":{"enabled":false},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled":false},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled":false},"SeLoadDriverPrivilege":{"enabled":false},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled":false},"SeRemoteShutdownPrivilege":{"enabled":false},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled":false},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTimeZonePrivilege":{"enabled":false},"SeUndockPrivilege":{"enabled":false}}}
    2020-10-22T23:04:14.365-0400    INFO    helper/privileges_windows.go:111        SeDebugPrivilege is now enabled. SeDebugPrivilege=(Enabled)
    2020-10-22T23:04:14.365-0400    INFO    cfgfile/reload.go:164   Config reloader started
    2020-10-22T23:04:14.368-0400    INFO    cfgfile/reload.go:224   Loading of config files completed.
    2020-10-22T23:04:15.384-0400    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to backoff(elasticsearch(https://longrandomawsguidstring.us-east-1.aws.found.io:443))
    2020-10-22T23:04:15.384-0400    INFO    [publisher]     pipeline/retry.go:219   retryer: send unwait signal to consumer
    2020-10-22T23:04:15.387-0400    INFO    [publisher]     pipeline/retry.go:223     done
    2020-10-22T23:04:15.437-0400    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.0
    2020-10-22T23:04:15.440-0400    INFO    [license]       licenser/es_callback.go:51      Elasticsearch license: Platinum
    2020-10-22T23:04:15.443-0400    INFO    [esclientleg]   eslegclient/connection.go:314   Attempting to connect to Elasticsearch version 7.9.0
    2020-10-22T23:04:15.447-0400    INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
    2020-10-22T23:04:15.450-0400    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
    2020-10-22T23:04:15.451-0400    INFO    [index-management]      idxmgmt/std.go:274      ILM policy successfully loaded.
    2020-10-22T23:04:15.452-0400    INFO    [index-management]      idxmgmt/std.go:407      Set setup.template.name to '{perf-metricbeat-7.3.1 {now/d}-000001}' as ILM is enabled.
    2020-10-22T23:04:15.453-0400    INFO    [index-management]      idxmgmt/std.go:412      Set setup.template.pattern to 'perf-metricbeat-7.3.1-*' as ILM is enabled.
    2020-10-22T23:04:15.453-0400    INFO    [index-management]      idxmgmt/std.go:446      Set settings.index.lifecycle.rollover_alias in template to {perf-metricbeat-7.3.1 {now/d}-000001} as ILM is enabled.
    2020-10-22T23:04:15.454-0400    INFO    [index-management]      idxmgmt/std.go:450      Set settings.index.lifecycle.name in template to {metrics {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
    2020-10-22T23:04:15.456-0400    INFO    template/load.go:89     Template perf-metricbeat-7.3.1 already exists and will not be overwritten.
    2020-10-22T23:04:15.456-0400    INFO    [index-management]      idxmgmt/std.go:298      Loaded index template.
    2020-10-22T23:04:15.456-0400    INFO    [index-management]      idxmgmt/std.go:309      Write alias successfully generated.
    2020-10-22T23:04:15.479-0400    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to backoff(elasticsearch(https://longrandomawsguidstring.us-east-1.aws.found.io:443)) established
    2020-10-22T23:04:44.361-0400    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":296,"time":{"ms":296}},"total":{"ticks":389,"time":{"ms":389},"value":389},"user":{"ticks":93,"time":{"ms":93}}},"handles":{"open":394},"info":{"ephemeral_id":"4922d0bf-1e07-4957-b8bd-4f5c5fa87351","uptime":{"ms":30714}},"memstats":{"gc_next":16109696,"memory_alloc":9198584,"memory_total":29498312,"rss":57413632},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1,"scans":1},"output":{"events":{"acked":7,"batches":1,"total":7},"type":"elasticsearch"},"pipeline":{"clients":6,"events":{"active":0,"published":7,"retry":7,"total":7},"queue":{"acked":7}}},"metricbeat":{"system":{"cpu":{"events":1,"success":1},"diskio":{"events":1,"success":1},"filesystem":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":2,"success":2},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":8}}}}}
    2020-10-22T23:05:14.359-0400    INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312,"time":{"ms":16}},"total":{"ticks":421,"time":{"ms":32},"value":421},"user":{"ticks":109,"time":{"ms":16}}},"handles":{"open":394},"info":{"ephemeral_id":"4922d0bf-1e07-4957-b8bd-4f5c5fa87351","uptime":{"ms":60713}},"memstats":{"gc_next":16109696,"memory_alloc":9915680,"memory_total":30215408,"rss":118784},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":7,"batches":1,"total":7}},"pipeline":{"clients":6,"events":{"active":0,"published":7,"total":7},"queue":{"acked":7}}},"metricbeat":{"system":{"cpu":{"events":1,"success":1},"diskio":{"events":1,"success":1},"filesystem":{"events":1,"success":1},"memory":{"events":1,"success":1},"network":{"events":2,"success":2},"uptime":{"events":1,"success":1}}}}}}
4

Yesterday I was setting up metricbeat on windows 2019 standard. My friend was doing the system administration part I just configured the beat. He gave the system user right to the beat. I am not sure if that's correct or necessary but without it, beat did not want to run as a service.

5

Am willing to try anything at this point, heh.

I'm curious what procedure he followed; as best I can tell, the implicit System group has perms to both, and I've even changed the ownership of both relevant directories to SYSTEM.

image
image1026×1280 73.2 KB

Any chance you could ask him what he did? I would normally offer beer for this, but that's difficult on a forum, so the best I can offer is goodwill...either way, would appreciate it.

6

I asked him and he just said that he had run the install-service-metricbeat.ps1 script that is inside metricbeat folder as admin. I know that in this script logs are saved in c:\ProgramData I remember I was changing the paths in this script to look like this. Can you try this out as pass logs here?

# Delete and stop the service if it already exists.
if (Get-Service metricbeat -ErrorAction SilentlyContinue) {
  $service = Get-WmiObject -Class Win32_Service -Filter "name='metricbeat'"
  $service.StopService()
  Start-Sleep -s 1
  $service.delete()
}

$workdir = Split-Path $MyInvocation.MyCommand.Path

# Create the new service.
New-Service -name metricbeat `
  -displayName Metricbeat `
  -binaryPathName "`"$workdir\metricbeat.exe`" --environment=windows_service -c `"$workdir\metricbeat.yml`" --path.home `"$workdir`" --path.data `"$workdir\data`" --path.logs `"$workdir\logs`" -E logging.files.redirect_stderr=true"

# Attempt to set the service to delayed start using sc config.
Try {
  Start-Process -FilePath sc.exe -ArgumentList 'config metricbeat start= delayed-auto'
}
Catch { Write-Host -f red "An error occurred setting the service to delayed start." }
7

And to correct what I have said before. The metricbeat service did not wanted to work because my keystore was in different location that service was expecting it to be(that was because I added keystore without command that says where it should be saved so it was saved in workdir path and system was expecting in the ProgramData because of the first run of the script that was not changed yet.)

8

Was hopeful for that one, but no dice...output from the metricbeat log file:

    2020-10-23T12:36:18.287-0400	INFO	instance/beat.go:640	Home path: [C:\Program Files\metricbeat] Config path: [C:\Program Files\metricbeat] Data path: [C:\Program Files\metricbeat\data] Logs path: [C:\Program Files\metricbeat\logs]
    2020-10-23T12:36:18.309-0400	INFO	instance/beat.go:648	Beat ID: 025b50a1-9ad0-4021-81f8-1e9a5fe79e67
    2020-10-23T12:36:18.310-0400	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\metricbeat", "data": "C:\\Program Files\\metricbeat\\data", "home": "C:\\Program Files\\metricbeat", "logs": "C:\\Program Files\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "025b50a1-9ad0-4021-81f8-1e9a5fe79e67"}}}
    2020-10-23T12:36:18.310-0400	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "2ab907f5ccecf9fd82fe37105082e89fd871f684", "libbeat": "7.9.2", "time": "2020-09-22T23:25:16.000Z", "version": "7.9.2"}}}
    2020-10-23T12:36:18.311-0400	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.14.7"}}}
    2020-10-23T12:36:18.320-0400	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-10-23T12:10:24.03-04:00","name":"FRUNIDEPWEB01","ip":["172.28.32.20/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.1457 (WinBuild.160101.0800)","mac":["00:50:56:a3:09:dd"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2019 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.1457"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"f6d317ff-566a-47c4-bd51-4d46c04db001"}}}
    2020-10-23T12:36:18.321-0400	INFO	[beat]	instance/beat.go:1021	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "C:\\Program Files\\metricbeat\\metricbeat.exe", "name": "metricbeat.exe", "pid": 1068, "ppid": 740, "start_time": "2020-10-23T12:36:17.471-0400"}}}
    2020-10-23T12:36:18.321-0400	INFO	instance/beat.go:299	Setup Beat: metricbeat; Version: 7.9.2
    2020-10-23T12:36:18.321-0400	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'perf-metricbeat-7.3.1' as ILM is enabled.
    2020-10-23T12:36:18.321-0400	INFO	eslegclient/connection.go:99	elasticsearch url: https://longrandomawsguidstring.us-east-1.aws.found.io:443
    2020-10-23T12:36:18.322-0400	INFO	[publisher]	pipeline/module.go:113	Beat name: FRUNIDEPWEB01
    2020-10-23T12:36:19.028-0400	INFO	instance/beat.go:450	metricbeat start running.
    2020-10-23T12:36:19.028-0400	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
    2020-10-23T12:36:19.043-0400	INFO	helper/privileges_windows.go:79	Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":8,"User":{"SID":"S-1-5-18","Account":"SYSTEM","Domain":"NT AUTHORITY","Type":1},"ProcessPrivs":{"SeAssignPrimaryTokenPrivilege":{"enabled":false},"SeAuditPrivilege":{"enabled_by_default":true,"enabled":true},"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePermanentPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateSymbolicLinkPrivilege":{"enabled_by_default":true,"enabled":true},"SeDebugPrivilege":{"enabled_by_default":true,"enabled":true},"SeDelegateSessionUserImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled_by_default":true,"enabled":true},"SeLoadDriverPrivilege":{"enabled":false},"SeLockMemoryPrivilege":{"enabled_by_default":true,"enabled":true},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled_by_default":true,"enabled":true},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled_by_default":true,"enabled":true},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTcbPrivilege":{"enabled_by_default":true,"enabled":true},"SeTimeZonePrivilege":{"enabled_by_default":true,"enabled":true},"SeUndockPrivilege":{"enabled":false}}}
    2020-10-23T12:36:19.043-0400	INFO	helper/privileges_windows.go:87	SeDebugPrivilege is enabled. SeDebugPrivilege=(Default, Enabled)
    2020-10-23T12:36:19.044-0400	INFO	cfgfile/reload.go:164	Config reloader started
    2020-10-23T12:36:19.046-0400	INFO	cfgfile/reload.go:224	Loading of config files completed.

FWIW: I'm not currently using a keystore file; i have the creds in the metricbeat.yml in an attempt to eliminate moving parts...seems like this must be something else.

Thanks man...I'll keep looking.

9

Did you enable the system module?

10

Yeah, it's enabled...reverified to make sure i wasn't going insane:

image
image986×400 18.6 KB

11

try this options in yml file and start the service

setup.ilm.enabled: false
setup.template.enabled: false

Do you have something in log after this error ?

12

Further update: I discovered PSTools, which allow you to run cmd under the auspices of nt authority\system, so i tried that w/ the cmd window for funsies. I can now unequivocably say it does NOT look look a permissions issue:

image
image1789×1320 268 KB

Seems like it must be something to do with the service registration, but i'm not sure what...this verifies that the account the service is running under (as local system) is able to run it, at any rate.

13

Nothing useful that i could see. Tried your disable commands, it got a line shorter than previous calls:

    2020-10-23T12:52:46.939-0400	INFO	instance/beat.go:640	Home path: [C:\Program Files\metricbeat] Config path: [C:\Program Files\metricbeat] Data path: [C:\Program Files\metricbeat\data] Logs path: [C:\Program Files\metricbeat\logs]
    2020-10-23T12:52:46.954-0400	INFO	instance/beat.go:648	Beat ID: 025b50a1-9ad0-4021-81f8-1e9a5fe79e67
    2020-10-23T12:52:46.956-0400	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\metricbeat", "data": "C:\\Program Files\\metricbeat\\data", "home": "C:\\Program Files\\metricbeat", "logs": "C:\\Program Files\\metricbeat\\logs"}, "type": "metricbeat", "uuid": "025b50a1-9ad0-4021-81f8-1e9a5fe79e67"}}}
    2020-10-23T12:52:46.956-0400	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "2ab907f5ccecf9fd82fe37105082e89fd871f684", "libbeat": "7.9.2", "time": "2020-09-22T23:25:16.000Z", "version": "7.9.2"}}}
    2020-10-23T12:52:46.956-0400	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.14.7"}}}
    2020-10-23T12:52:46.966-0400	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-10-23T12:10:23.98-04:00","name":"FRUNIDEPWEB01","ip":["172.28.32.20/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17763.1457 (WinBuild.160101.0800)","mac":["00:50:56:a3:09:dd"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2019 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.1457"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"f6d317ff-566a-47c4-bd51-4d46c04db001"}}}
    2020-10-23T12:52:46.966-0400	INFO	[beat]	instance/beat.go:1021	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "C:\\Program Files\\metricbeat\\metricbeat.exe", "name": "metricbeat.exe", "pid": 6348, "ppid": 740, "start_time": "2020-10-23T12:52:46.214-0400"}}}
    2020-10-23T12:52:46.966-0400	INFO	instance/beat.go:299	Setup Beat: metricbeat; Version: 7.9.2
    2020-10-23T12:52:46.967-0400	INFO	eslegclient/connection.go:99	elasticsearch url: https://longrandomawsguidstring.us-east-1.aws.found.io:443
    2020-10-23T12:52:46.967-0400	INFO	[publisher]	pipeline/module.go:113	Beat name: FRUNIDEPWEB01
    2020-10-23T12:52:47.017-0400	INFO	instance/beat.go:450	metricbeat start running.
    2020-10-23T12:52:47.017-0400	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
    2020-10-23T12:52:47.021-0400	INFO	helper/privileges_windows.go:79	Metricbeat process and system info: {"OSVersion":{"Major":6,"Minor":2,"Build":9200},"Arch":"amd64","NumCPU":8,"User":{"SID":"S-1-5-18","Account":"SYSTEM","Domain":"NT AUTHORITY","Type":1},"ProcessPrivs":{"SeAssignPrimaryTokenPrivilege":{"enabled":false},"SeAuditPrivilege":{"enabled_by_default":true,"enabled":true},"SeBackupPrivilege":{"enabled":false},"SeChangeNotifyPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateGlobalPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePagefilePrivilege":{"enabled_by_default":true,"enabled":true},"SeCreatePermanentPrivilege":{"enabled_by_default":true,"enabled":true},"SeCreateSymbolicLinkPrivilege":{"enabled_by_default":true,"enabled":true},"SeDebugPrivilege":{"enabled_by_default":true,"enabled":true},"SeDelegateSessionUserImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeImpersonatePrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseBasePriorityPrivilege":{"enabled_by_default":true,"enabled":true},"SeIncreaseQuotaPrivilege":{"enabled":false},"SeIncreaseWorkingSetPrivilege":{"enabled_by_default":true,"enabled":true},"SeLoadDriverPrivilege":{"enabled":false},"SeLockMemoryPrivilege":{"enabled_by_default":true,"enabled":true},"SeManageVolumePrivilege":{"enabled":false},"SeProfileSingleProcessPrivilege":{"enabled_by_default":true,"enabled":true},"SeRestorePrivilege":{"enabled":false},"SeSecurityPrivilege":{"enabled":false},"SeShutdownPrivilege":{"enabled":false},"SeSystemEnvironmentPrivilege":{"enabled":false},"SeSystemProfilePrivilege":{"enabled_by_default":true,"enabled":true},"SeSystemtimePrivilege":{"enabled":false},"SeTakeOwnershipPrivilege":{"enabled":false},"SeTcbPrivilege":{"enabled_by_default":true,"enabled":true},"SeTimeZonePrivilege":{"enabled_by_default":true,"enabled":true},"SeUndockPrivilege":{"enabled":false}}}
    2020-10-23T12:52:47.022-0400	INFO	helper/privileges_windows.go:87	SeDebugPrivilege is enabled. SeDebugPrivilege=(Default, Enabled)
    2020-10-23T12:52:47.022-0400	INFO	cfgfile/reload.go:164	Config reloader started
    2020-10-23T12:52:47.023-0400	INFO	cfgfile/reload.go:224	Loading of config files completed.
14

What is the error when you start service from services window?

15

Standard generic error 1067, "terminated unexpectedly"

image
image1022×639 160 KB

16

More updates: went back and tried a garden-variety install w/ my system.yml and metricbeat.yml files from way up at the top on a Win 2012R2 box, and it worked almost immediately. Sigh.

So either this particular Win 2019 box is hosed for some reason, or Win 2019 in general is hosed (although @Adriann's experience seems to indicate otherwise). I'll try another 2019 box in a few.

1 Like
17

sigh

Two more 2019 boxes later, I'm forced to conclude it's something to do with the original box....setup hasn't been an issue on the now-3-other-boxes I've tried. If i do manage to figure out what's wrong with the original one, i'll post it here.

18

And our culprit rears its ugly, ugly head.

For reasons I won't go into here, some of our our servers are using different antivirus products; the problem one, and others showing the same symptom, were all running Carbon Black's Cloud sensor, whereas the cooperative servers were running Cylance.

So, in tonight's battle of AV products, Cyclance wins and Carbon Black loses, along with myself, because I burnt a day and a half on this. The lesson: pick your AV products wisely....we've had a lot of troubler with CB, so I guess i should have suspected this earlier.

Thanks guys.

2 Likes
19

Thanks for the pointer to Carbon Black Cloud Sensor. I was banging my head with the same issue for 2 days and yes, we have the same Carbon Black Cloud Sensor running on the machine.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.