Error when startingmetricbeat with v7.5.3

vee · March 20, 2020, 9:14pm

I see this error when trying to start metricbeat on ppc64le system:

2020-03-20T15:47:22.639-0500 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://elasticdev:9200)): Connection marked as failed because the onConnect callback failed: resource 'metricbeat-7.5.3' exists, but it is not an alias

Here's the command to start:

metricbeat -E path.config=/tmp/metricbeat -e -c /tmp/metricbeat/metricbeat.yml

Here is the metricbeat.yml:

metricbeat.config.modules:
  path.config: /tmp/metricbeat/current_metricbeat

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

metricbeat.modules:
- module: system
  metricsets: [cpu]
  cpu.metrics: [percentages,normalized_percentages]

- module: system
  metricsets: [process]
  processes: ['.*']
  process.cgroups.enabled: false
  process.cmdline.cache.enabled: true
  process.include_top_n:
    by_cpu: 20      # include top 20  processes by CPU
    by_memory: 20   # include top 20  processes by memory

- module: system
  period: 10m
  metricsets:
    - filesystem
    - fsstat
  filesystem.ignore_types: [nfs, smbfs, autofs, rpc_pipefs, rootfs]
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(dev|run|sys|fs|host|lib|tmpfs|boot|proc)($|/)'

- module: system
  period: 10s
  metricsets:
    - load
    - memory
    - network
    - process_summary
    - diskio

- module: system
  period: 15m
  metricsets:
    - uptime

- module: system
  metricsets: [core]
  core.metrics: [percentages]

setup.template.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

output.elasticsearch:
 hosts: ["http://elastic1.ulinedm.com:9200" , "http://elastic2.ulinedm.com:9200"]
 username: elastic
 password: xyz
logging.level: info
logging.to_files: true
logging.files:
 path: /logs/metricbeat
 name: metricbeat.log
 keepfiles: 7

shaunak · March 23, 2020, 7:59pm

Could you make the following Elasticsearch API calls and post their results here please?

GET _cat/indices/metricbeat-*?v

GET _cat/aliases/metricbeat-*?v

GET _template/metricbeat-*

Thanks,

Shaunak

vee · March 24, 2020, 2:30pm

Here you go:

GET _cat/indices/metricbeat-*?v

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   metricbeat-6.2.2-2020.03.17        07MAB82ZS16I6l-7fMMaDQ   1   1    2309874            0      1.4gb        725.6mb
green  open   metricbeat-6.2.2-2020.03.18        5r3OXkVXTAKMIcd9G5qnaw   1   1    2306493            0      1.4gb        717.9mb
green  open   metricbeat-6.6.2-2020.03.20        HAf4lqZbQVy3DbkQRstsxQ   1   1      13455            0      6.9mb          3.3mb
green  open   metricbeat-6.2.2-2020.03.19        fUKX2DTCT0SaRhfqvnCCwg   1   1    2315937            0      1.4gb        720.4mb
green  open   metricbeat-6.2.2-2020.03.23        5yyvKMFaSYOjj1Qv8NBXFg   1   1    1528948            0      1.2gb        690.7mb
green  open   metricbeat-7.5.2-2020.03.23-000001 BO5m035VQpG0Av1v-KxpVg   1   1          0            0       566b           283b
green  open   metricbeat-7.5.3                   tsyNEweSR9eWsnYOj8QoiA   1   1          0            0       566b           283b
green  open   metricbeat-6.2.2-2020.03.20        BqfgBDgeTumVB1t04s0eaQ   1   1    2333112            0      1.4gb        734.3mb
green  open   metricbeat-6.2.2-2020.03.21        Grn-9hNVTwmJNqsWc0WKWw   1   1    2314736            0      1.4gb          725mb
green  open   metricbeat-6.2.2-2020.03.22        k5XV0AbLT4GU5vp3wgGw_w   1   1    2318557            0      1.4gb        725.1mb

GET _cat/aliases/metricbeat-*?v

alias            index                              filter routing.index routing.search is_write_index
metricbeat-7.5.2 metricbeat-7.5.2-2020.03.23-000001 -      -             -              true

GET _template/metricbeat-*

{
  "metricbeat-7.5.3" : {...}
  "metricbeat-6.8.3" : {...}
  "metricbeat-6.8.0" : {...}
  "metricbeat-7.5.1" : {...}
  "metricbeat-6.6.0" : {...}
  "metricbeat-6.8.6" : {...}
  "metricbeat-6.6.2" : {...}
}

shaunak · March 24, 2020, 4:11pm

Thanks. So the problem is that somehow there is a metricbeat-7.5.3 index, whereas Metricbeat is expecting this to be an alias.

At this point, I'd suggest the following steps:

Stop all instances of Metricbeat 7.5.3.
Delete the metricbeat-7.5.3 index as it doesn't have any documents in it:
```
DELETE metricbeat-7.5.3
```
Delete the Metricbeat 7.5.3 index template.
```
DELETE _template/metricbeat-7.5.3
```
Restart the stopped Metricbeat instances. This should recreate the template, the index alias, and the correct index.

Shaunak

vee · March 24, 2020, 6:17pm

Tried to delete the metricbeat and it's template v7.5.3 - and when restarted, I still see the same error.

GET _cat/indices/metricbeat-7.5.2
GET _cat/indices/metricbeat-7.5.3

GET _cat/aliases/metricbeat-*?v

GET _template/metricbeat-7.5.2
GET _template/metricbeat-7.5.3

# GET _cat/indices/metricbeat-7.5.2
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [metricbeat-7.5.2]",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : "metricbeat-7.5.2",
        "index" : "metricbeat-7.5.2"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [metricbeat-7.5.2]",
    "index_uuid" : "_na_",
    "resource.type" : "index_or_alias",
    "resource.id" : "metricbeat-7.5.2",
    "index" : "metricbeat-7.5.2"
  },
  "status" : 404
}


# GET _cat/indices/metricbeat-7.5.3
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [metricbeat-7.5.3]",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : "metricbeat-7.5.3",
        "index" : "metricbeat-7.5.3"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [metricbeat-7.5.3]",
    "index_uuid" : "_na_",
    "resource.type" : "index_or_alias",
    "resource.id" : "metricbeat-7.5.3",
    "index" : "metricbeat-7.5.3"
  },
  "status" : 404
}


# GET _cat/aliases/metricbeat-*?v
alias index filter routing.index routing.search is_write_index


# GET _template/metricbeat-7.5.2
{ }


# GET _template/metricbeat-7.5.3
{ }

vee · March 24, 2020, 6:23pm

Error message at startup:

[root@host05 current_metricbeat]# ./metricbeat -E path.config=/opt/apps/metricbeat -e -c /opt/apps/metricbeat/metricbeat.yml
2020-03-24T13:19:44.784-0500    INFO    instance/beat.go:610    Home path: [/opt/apps/metricbeat/current_metricbeat] Config path: [/opt/apps/metricbeat] Data path: [/opt/apps/metricbeat/current_metricbeat/data] Logs path: [/opt/apps/metricbeat/current_metricbeat/logs]
2020-03-24T13:19:44.784-0500    INFO    instance/beat.go:618    Beat ID: 8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6
2020-03-24T13:19:44.785-0500    INFO    [seccomp]       seccomp/seccomp.go:101  Syscall filter could not be installed because the kernel does not support seccomp
2020-03-24T13:19:44.785-0500    INFO    [beat]  instance/beat.go:941    Beat info       {"system_info": {"beat": {"path": {"config": "/opt/apps/metricbeat", "data": "/opt/apps/metricbeat/current_metricbeat/data", "home": "/opt/apps/metricbeat/current_metricbeat", "logs": "/opt/apps/metricbeat/current_metricbeat/logs"}, "type": "metricbeat", "uuid": "8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6"}}}
2020-03-24T13:19:44.785-0500    INFO    [beat]  instance/beat.go:950    Build info      {"system_info": {"build": {"commit": "8b46ba0adc954b7d27fe7ecc05ec0fddb6879a69", "libbeat": "7.5.3", "time": "2020-03-17T14:28:07.000Z", "version": "7.5.3"}}}
2020-03-24T13:19:44.785-0500    INFO    [beat]  instance/beat.go:953    Go runtime info {"system_info": {"go": {"os":"linux","arch":"ppc64le","max_procs":32,"version":"go1.13.8"}}}
2020-03-24T13:19:44.788-0500    INFO    [beat]  instance/beat.go:957    Host info       {"system_info": {"host": {"architecture":"ppc64le","boot_time":"2020-01-09T19:39:45-06:00","containerized":false,"name":"host05.xyz.com","ip":["127.0.0.1/8","10.50.38.180/24"],"kernel_version":"3.10.0-1062.9.1.el7.ppc64le","mac":["fa:f4:02:aa:2f:20"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.7 (Maipo)","major":7,"minor":7,"patch":0,"codename":"Maipo"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"f918f59022314edea2d4e2da716f2f55"}}}
2020-03-24T13:19:44.789-0500    INFO    [beat]  instance/beat.go:986    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/opt/apps/metricbeat/metricbeat-7.5.2-linux-ppc64le", "exe": "/opt/apps/metricbeat/metricbeat-7.5.2-linux-ppc64le/metricbeat", "name": "metricbeat", "pid": 10081, "ppid": 1705, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-03-24T13:19:44.330-0500"}}}
2020-03-24T13:19:44.789-0500    INFO    instance/beat.go:297    Setup Beat: metricbeat; Version: 7.5.3
2020-03-24T13:19:44.789-0500    INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'metricbeat-7.5.3' as ILM is enabled.
2020-03-24T13:19:44.789-0500    INFO    elasticsearch/client.go:171     Elasticsearch url: http://elastic09.xyz.com:9200
2020-03-24T13:19:44.789-0500    INFO    elasticsearch/client.go:171     Elasticsearch url: http://elastic10.xyz.com:9200
2020-03-24T13:19:44.790-0500    INFO    [publisher]     pipeline/module.go:97   Beat name: host05.xyz.com
2020-03-24T13:19:44.790-0500    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: nfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T13:19:44.790-0500    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %snfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T13:19:44.791-0500    INFO    instance/beat.go:429    metricbeat start running.
2020-03-24T13:19:44.791-0500    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-03-24T13:19:44.791-0500    INFO    cfgfile/reload.go:171   Config reloader started
2020-03-24T13:19:46.227-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elastic10.xyz.com:9200))
2020-03-24T13:19:46.227-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elastic09.xyz.com:9200))
2020-03-24T13:19:46.230-0500    INFO    elasticsearch/client.go:753     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T13:19:46.230-0500    INFO    elasticsearch/client.go:753     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T13:19:46.231-0500    INFO    [index-management]      idxmgmt/std.go:256      Auto ILM enable success.
2020-03-24T13:19:46.233-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T13:19:46.233-0500    INFO    [index-management]      idxmgmt/std.go:269      ILM policy successfully loaded.
2020-03-24T13:19:46.356-0500    INFO    [index-management]      idxmgmt/std.go:304      Write alias successfully generated.
2020-03-24T13:19:46.356-0500    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elastic10.xyz.com:9200)) established
2020-03-24T13:19:46.356-0500    INFO    [index-management]      idxmgmt/std.go:256      Auto ILM enable success.
2020-03-24T13:19:46.358-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T13:19:46.358-0500    INFO    [index-management]      idxmgmt/std.go:269      ILM policy successfully loaded.
2020-03-24T13:19:46.361-0500    INFO    [index-management]      idxmgmt/std.go:304      Write alias successfully generated.
2020-03-24T13:19:46.361-0500    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elastic09.xyz.com:9200)) established
2020-03-24T13:19:46.522-0500    WARN    elasticsearch/client.go:511     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b1b054cb20f3, ext:630135898, loc:(*time.Location)(0x14bc1ae0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"2a01b503-de58-4687-8eb2-1fe97148b3aa", "hostname":"host05.xyz.com", "id":"8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6", "type":"metricbeat", "version":"7.5.3"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"system.diskio", "duration":6480866, "module":"system"}, "host":common.MapStr{"name":"host05.xyz.com"}, "metricset":common.MapStr{"name":"diskio", "period":10000}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"diskio":common.MapStr{"io":common.MapStr{"time":0x1e0}, "iostat":common.MapStr{"await":0, "busy":0, "queue":common.MapStr{"avg_size":0}, "read":common.MapStr{"await":0, "per_sec":common.MapStr{"bytes":0}, "request":common.MapStr{"merges_per_sec":0, "per_sec":0}}, "request":common.MapStr{"avg_size":0}, "service_time":0, "write":common.MapStr{"await":0, "per_sec":common.MapStr{"bytes":0}, "request":common.MapStr{"merges_per_sec":0, "per_sec":0}}}, "name":"dm-13", "read":common.MapStr{"bytes":0x1c90000, "count":0xd3, "time":0x1e0}, "write":common.MapStr{"bytes":0x0, "count":0x0, "time":0x0}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}
2020-03-24T13:19:46.523-0500    WARN    elasticsearch/client.go:511     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b1b054cb20f3, ext:630135898, loc:(*time.Location)(0x14bc1ae0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"2a01b503-de58-4687-8eb2-1fe97148b3aa", "hostname":"host05.xyz.com", "id":"8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6", "type":"metricbeat", "version":"7.5.3"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"system.diskio", "duration":6554349, "module":"system"}, "host":common.MapStr{"name":"host05.xyz.com"}, "metricset":common.MapStr{"name":"diskio", "period":10000}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"diskio":common.MapStr{"io":common.MapStr{"time":0x28fbae}, "iostat":common.MapStr{"await":0, "busy":0, "queue":common.MapStr{"avg_size":0}, "read":common.MapStr{"await":0, "per_sec":common.MapStr{"bytes":0}, "request":common.MapStr{"merges_per_sec":0, "per_sec":0}}, "request":common.MapStr{"avg_size":0}, "service_time":0, "write":common.MapStr{"await":0, "per_sec":common.MapStr{"bytes":0}, "request":common.MapStr{"merges_per_sec":0, "per_sec":0}}}, "name":"dm-20", "read":common.MapStr{"bytes":0x51519600, "count":0x66d7, "time":0x3604c}, "write":common.MapStr{"bytes":0x9f3ac4200, "count":0x22ddf6, "time":0x150fbda}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}
2020-03-24T13:19:46.523-0500    WARN    elasticsearch/client.go:511     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b1b054cb20f3, ext:630135898, loc:(*time.Location)(0x14bc1ae0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"2a01b503-de58-4687-8eb2-1fe97148b3aa", "hostname":"host05.xyz.com", "id":"8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6", "type":"metricbeat", "version":"7.5.3"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"system.diskio", "duration":6566004, "module":"system"}, "host":common.MapStr{"name":"host05.xyz.com"}, "metricset":common.MapStr{"name":"diskio", "period":10000}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"diskio":common.MapStr{"io":common.MapStr{"time":0x5a032}, "iostat":common.MapStr{"await":0, "busy":0, "queue":common.MapStr{"avg_size":0}, "read":common.MapStr{"await":0, "per_sec":common.MapStr{"bytes":0}, "request":common.MapStr{"merges_per_sec":0, "per_sec":0}}, "request":common.MapStr{"avg_size":0}, "service_time":0, "write":common.MapStr{"await":0, "per_sec":common.MapStr{"bytes":0}, "request":common.MapStr{"merges_per_sec":0, "per_sec":0}}}, "name":"dm-22", "read":common.MapStr{"bytes":0xa30c9a00, "count":0x1f06, "time":0x2a3d2}, "write":common.MapStr{"bytes":0x127410bc00, "count":0x8c6e6, "time":0x1049f38}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}

shaunak · March 24, 2020, 6:42pm

That's strange. Thanks for posting the logs. These messages in your log look odd to me:

2020-03-24T13:19:46.233-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T13:19:46.358-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false

Could you post the entire contents of your metricbeat.yml please (after redacting any sensitive information)?

Thanks,

Shaunak

shaunak · March 24, 2020, 6:44pm

Never mind, please ignore my last post. I see that you posted your metricbeat.yml above already!

Could you try commenting out this line from your metricbeat.yml?

setup.template.enabled: false

And then restart Metricbeat and check things again?

Thanks,

Shaunak

vee · March 24, 2020, 7:23pm

Still same error after commenting it out.

GET _cat/indices/metricbeat-7.5.3
GET _template/metricbeat-7.5.3
GET _cat/aliases/metricbeat-*?v

# GET _cat/indices/metricbeat-7.5.3
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [metricbeat-7.5.3]",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : "metricbeat-7.5.3",
        "index" : "metricbeat-7.5.3"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [metricbeat-7.5.3]",
    "index_uuid" : "_na_",
    "resource.type" : "index_or_alias",
    "resource.id" : "metricbeat-7.5.3",
    "index" : "metricbeat-7.5.3"
  },
  "status" : 404
}


# GET _template/metricbeat-7.5.3
{ }


# GET _cat/aliases/metricbeat-*?v
alias index filter routing.index routing.search is_write_index

Metricbeat.yml:

[user@host metricbeat]# cat /opt/apps/metricbeat/metricbeat.yml
metricbeat.config.modules:
  path.config: /opt/apps/metricbeat/current_metricbeat

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

metricbeat.modules:
- module: system
  metricsets: [cpu]
  cpu.metrics: [percentages,normalized_percentages]

- module: system
  metricsets: [process]
  processes: ['.*']
  process.cgroups.enabled: false
  process.cmdline.cache.enabled: true
  process.include_top_n:
    by_cpu: 20      # include top 20  processes by CPU
    by_memory: 20   # include top 20  processes by memory

- module: system
  period: 10m
  metricsets:
    - filesystem
    - fsstat
  filesystem.ignore_types: [nfs, smbfs, autofs, rpc_pipefs, rootfs]
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(dev|run|sys|fs|host|lib|tmpfs|boot|proc)($|/)'

- module: system
  period: 10s
  metricsets:
    - load
    - memory
    - network
    - process_summary
    - diskio

- module: system
  period: 15m
  metricsets:
    - uptime

- module: system
  metricsets: [core]
  core.metrics: [percentages]

#setup.template.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

output.elasticsearch:
 hosts: ["http://elasticdev09.xyz.com:9200" , "http://elastic10.xyz.com:9200"]
 username: elastic
 password: elastic
logging.level: info
logging.to_files: true
logging.files:
 path: /opt/logs/metricbeat
 name: metricbeat.log
 keepfiles: 7

vee · March 24, 2020, 7:24pm

Startup error:

[user@host metricbeat]# /opt/apps/metricbeat/current_metricbeat/metricbeat -E path.config=/opt/apps/metricbeat -e -c /opt/apps/metricbeat/metricbeat.yml
2020-03-24T14:20:49.436-0500    INFO    instance/beat.go:610    Home path: [/opt/apps/metricbeat/current_metricbeat] Config path: [/opt/apps/metricbeat] Data path: [/opt/apps/metricbeat/current_metricbeat/data] Logs path: [/opt/apps/metricbeat/current_metricbeat/logs]
2020-03-24T14:20:49.437-0500    INFO    instance/beat.go:618    Beat ID: 8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6
2020-03-24T14:20:49.437-0500    INFO    [seccomp]       seccomp/seccomp.go:101  Syscall filter could not be installed because the kernel does not support seccomp
2020-03-24T14:20:49.437-0500    INFO    [beat]  instance/beat.go:941    Beat info       {"system_info": {"beat": {"path": {"config": "/opt/apps/metricbeat", "data": "/opt/apps/metricbeat/current_metricbeat/data", "home": "/opt/apps/metricbeat/current_metricbeat", "logs": "/opt/apps/metricbeat/current_metricbeat/logs"}, "type": "metricbeat", "uuid": "8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6"}}}
2020-03-24T14:20:49.437-0500    INFO    [beat]  instance/beat.go:950    Build info      {"system_info": {"build": {"commit": "8b46ba0adc954b7d27fe7ecc05ec0fddb6879a69", "libbeat": "7.5.3", "time": "2020-03-17T14:28:07.000Z", "version": "7.5.3"}}}
2020-03-24T14:20:49.437-0500    INFO    [beat]  instance/beat.go:953    Go runtime info {"system_info": {"go": {"os":"linux","arch":"ppc64le","max_procs":32,"version":"go1.13.8"}}}
2020-03-24T14:20:49.440-0500    INFO    [beat]  instance/beat.go:957    Host info       {"system_info": {"host": {"architecture":"ppc64le","boot_time":"2020-01-09T19:39:45-06:00","containerized":false,"name":"dc2-svzdev05.xyz.com","ip":["127.0.0.1/8","10.50.38.180/24"],"kernel_version":"3.10.0-1062.9.1.el7.ppc64le","mac":["fa:f4:02:aa:2f:20"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.7 (Maipo)","major":7,"minor":7,"patch":0,"codename":"Maipo"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"f918f59022314edea2d4e2da716f2f55"}}}
2020-03-24T14:20:49.441-0500    INFO    [beat]  instance/beat.go:986    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/opt/apps/metricbeat", "exe": "/opt/apps/metricbeat/metricbeat-7.5.2-linux-ppc64le/metricbeat", "name": "metricbeat", "pid": 31216, "ppid": 1705, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2020-03-24T14:20:48.990-0500"}}}
2020-03-24T14:20:49.441-0500    INFO    instance/beat.go:297    Setup Beat: metricbeat; Version: 7.5.3
2020-03-24T14:20:49.441-0500    INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'metricbeat-7.5.3' as ILM is enabled.
2020-03-24T14:20:49.441-0500    INFO    elasticsearch/client.go:171     Elasticsearch url: http://elastic09.xyz.com:9200
2020-03-24T14:20:49.441-0500    INFO    elasticsearch/client.go:171     Elasticsearch url: http://elastic10.xyz.com:9200
2020-03-24T14:20:49.441-0500    INFO    [publisher]     pipeline/module.go:97   Beat name: dc2-svzdev05.xyz.com
2020-03-24T14:20:49.443-0500    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: nfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T14:20:49.444-0500    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %snfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T14:20:49.444-0500    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-03-24T14:20:49.444-0500    INFO    instance/beat.go:429    metricbeat start running.
2020-03-24T14:20:49.444-0500    INFO    cfgfile/reload.go:171   Config reloader started
2020-03-24T14:20:50.915-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elastic10.xyz.com:9200))
2020-03-24T14:20:50.915-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elastic09.xyz.com:9200))
2020-03-24T14:20:50.918-0500    INFO    elasticsearch/client.go:753     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T14:20:50.918-0500    INFO    elasticsearch/client.go:753     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T14:20:50.920-0500    INFO    [index-management]      idxmgmt/std.go:256      Auto ILM enable success.
2020-03-24T14:20:50.921-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T14:20:50.921-0500    INFO    [index-management]      idxmgmt/std.go:269      ILM policy successfully loaded.
2020-03-24T14:20:50.921-0500    INFO    [index-management]      idxmgmt/std.go:408      Set setup.template.name to '{metricbeat-7.5.3 {now/d}-000001}' as ILM is enabled.
2020-03-24T14:20:50.921-0500    INFO    [index-management]      idxmgmt/std.go:413      Set setup.template.pattern to 'metricbeat-7.5.3-*' as ILM is enabled.
2020-03-24T14:20:50.922-0500    INFO    [index-management]      idxmgmt/std.go:447      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.5.3 {now/d}-000001} as ILM is enabled.
2020-03-24T14:20:50.922-0500    INFO    [index-management]      idxmgmt/std.go:451      Set settings.index.lifecycle.name in template to {metricbeat-7.5.3 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-24T14:20:51.228-0500    INFO    template/load.go:109    Try loading template metricbeat-7.5.3 to Elasticsearch
2020-03-24T14:20:51.479-0500    INFO    template/load.go:101    template with name 'metricbeat-7.5.3' loaded.
2020-03-24T14:20:51.479-0500    INFO    [index-management]      idxmgmt/std.go:293      Loaded index template.
2020-03-24T14:20:51.509-0500    INFO    [index-management]      idxmgmt/std.go:302      Write alias exists already
2020-03-24T14:20:51.509-0500    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elastic10.xyz.com:9200)) established
2020-03-24T14:20:51.509-0500    INFO    [index-management]      idxmgmt/std.go:256      Auto ILM enable success.
2020-03-24T14:20:51.511-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T14:20:51.511-0500    INFO    [index-management]      idxmgmt/std.go:269      ILM policy successfully loaded.
2020-03-24T14:20:51.511-0500    INFO    [index-management]      idxmgmt/std.go:408      Set setup.template.name to '{metricbeat-7.5.3 {now/d}-000001}' as ILM is enabled.
2020-03-24T14:20:51.511-0500    INFO    [index-management]      idxmgmt/std.go:413      Set setup.template.pattern to 'metricbeat-7.5.3-*' as ILM is enabled.
2020-03-24T14:20:51.511-0500    INFO    [index-management]      idxmgmt/std.go:447      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.5.3 {now/d}-000001} as ILM is enabled.
2020-03-24T14:20:51.511-0500    INFO    [index-management]      idxmgmt/std.go:451      Set settings.index.lifecycle.name in template to {metricbeat-7.5.3 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-24T14:20:51.559-0500    INFO    template/load.go:89     Template metricbeat-7.5.3 already exists and will not be overwritten.
2020-03-24T14:20:51.559-0500    INFO    [index-management]      idxmgmt/std.go:293      Loaded index template.
2020-03-24T14:20:51.660-0500    WARN    elasticsearch/client.go:511     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b5447687d48b, ext:538934666, loc:(*time.Location)(0x14bc1ae0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"fafc56a8-a104-428d-9f89-a048ecb53121", "hostname":"dc2-svzdev05.xyz.com", "id":"8fa1f294-3ad0-4a1e-abe9-7fc9cf6be3b6", "type":"metricbeat", "version":"7.5.3"}, "ecs":common.MapStr{"version":"1.1.0"}, "event":common.MapStr{"dataset":"system.cpu", "duration":124062, "module":"system"}, "host":common.MapStr{"name":"dc2-svzdev05.xyz.com"}, "metricset":common.MapStr{"name":"cpu", "period":10000}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"cpu":common.MapStr{"cores":32, "idle":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "iowait":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "irq":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "nice":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "softirq":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "steal":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "system":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "total":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}, "user":common.MapStr{"norm":common.MapStr{"pct":0}, "pct":0}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}
2020-03-24T14:20:52.532-0500    INFO    [monitoring]    log/log.go:154  Uptime: 3.152743749s
2020-03-24T14:20:52.532-0500    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2020-03-24T14:20:52.532-0500    INFO    instance/beat.go:435    metricbeat stopped.

shaunak · March 24, 2020, 7:43pm

I dug around the code where this log message is being generated from:

2020-03-24T14:20:51.511-0500    INFO    [index-management.ilm]  ilm/std.go:138  do not generate ilm policy: exists=true, overwrite=false

Long story short, it looks like we're running into this bug: Change `setup` behavior to respect `ilm.overwrite` setting · Issue #14735 · elastic/beats · GitHub. The bug was fixed in https://github.com/elastic/beats/pull/14741 and the fix was released in Metricbeat 7.6.0. Would it be possible for you to try out that version?

Shaunak

vee · March 24, 2020, 8:29pm

Downloaded v7.6.0, tried with both setup template enabled & commented out in metricbeat.yml, but still see errors:

[user@host metricbeat]$ sudo current_metricbeat/metricbeat -E path.config=/opt/apps/metricbeat -e -c /opt/apps/metricbeat/metricbeat.yml
2020-03-24T15:26:51.927-0500    INFO    instance/beat.go:622    Home path: [/opt/apps/metricbeat/current_metricbeat] Config path: [/opt/apps/metricbeat] Data path: [/opt/apps/metricbeat/current_metricbeat/data] Logs path: [/opt/apps/metricbeat/current_metricbeat/logs]
2020-03-24T15:26:51.928-0500    INFO    instance/beat.go:630    Beat ID: 5ee67c4f-da63-43c8-89f8-3d4bf049f871
2020-03-24T15:26:51.928-0500    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-03-24T15:26:51.929-0500    INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "/opt/apps/metricbeat", "data": "/opt/apps/metricbeat/current_metricbeat/data", "home": "/opt/apps/metricbeat/current_metricbeat", "logs": "/opt/apps/metricbeat/current_metricbeat/logs"}, "type": "metricbeat", "uuid": "5ee67c4f-da63-43c8-89f8-3d4bf049f871"}}}
2020-03-24T15:26:51.929-0500    INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:10:10.000Z", "version": "7.6.0"}}}
2020-03-24T15:26:51.929-0500    INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.7"}}}
2020-03-24T15:26:51.933-0500    INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-08T08:03:45-06:00","containerized":false,"name":"host.xyzdm.com","ip":["127.0.0.1/8","10.50.80.67/20"],"kernel_version":"3.10.0-1062.9.1.el7.x86_64","mac":["00:50:56:b2:6d:9e"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.7 (Maipo)","major":7,"minor":7,"patch":0,"codename":"Maipo"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"6f72e7e35f214eab9e167ef87d5bb515"}}}
2020-03-24T15:26:51.933-0500    INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/opt/apps/metricbeat", "exe": "/opt/apps/metricbeat/metricbeat-7.6.0-linux-x86_64/metricbeat", "name": "metricbeat", "pid": 10039, "ppid": 10035, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-03-24T15:26:50.930-0500"}}}
2020-03-24T15:26:51.934-0500    INFO    instance/beat.go:298    Setup Beat: metricbeat; Version: 7.6.0
2020-03-24T15:26:51.934-0500    INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'metricbeat-7.6.0' as ILM is enabled.
2020-03-24T15:26:51.934-0500    INFO    elasticsearch/client.go:174     Elasticsearch url: http://elasticdev09.xyzdm.com:9200
2020-03-24T15:26:51.934-0500    INFO    elasticsearch/client.go:174     Elasticsearch url: http://elasticdev10.xyzdm.com:9200
2020-03-24T15:26:51.935-0500    INFO    [publisher]     pipeline/module.go:110  Beat name: host.xyzdm.com
2020-03-24T15:26:51.947-0500    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: nfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T15:26:51.947-0500    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %snfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T15:26:51.947-0500    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-03-24T15:26:51.948-0500    INFO    instance/beat.go:439    metricbeat start running.
2020-03-24T15:26:51.948-0500    INFO    cfgfile/reload.go:171   Config reloader started
2020-03-24T15:26:53.696-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticdev10.xyzdm.com:9200))
2020-03-24T15:26:53.697-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticdev09.xyzdm.com:9200))
2020-03-24T15:26:53.700-0500    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T15:26:53.700-0500    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T15:26:53.702-0500    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Platinum
2020-03-24T15:26:53.704-0500    INFO    [index-management]      idxmgmt/std.go:258      Auto ILM enable success.
2020-03-24T15:26:53.705-0500    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T15:26:53.705-0500    INFO    [index-management]      idxmgmt/std.go:271      ILM policy successfully loaded.
2020-03-24T15:26:53.705-0500    INFO    [index-management]      idxmgmt/std.go:410      Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
2020-03-24T15:26:53.705-0500    INFO    [index-management]      idxmgmt/std.go:415      Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
2020-03-24T15:26:53.705-0500    INFO    [index-management]      idxmgmt/std.go:449      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.
2020-03-24T15:26:53.705-0500    INFO    [index-management]      idxmgmt/std.go:453      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-24T15:26:53.992-0500    INFO    template/load.go:109    Try loading template metricbeat-7.6.0 to Elasticsearch
2020-03-24T15:26:54.256-0500    INFO    template/load.go:101    template with name 'metricbeat-7.6.0' loaded.
2020-03-24T15:26:54.256-0500    INFO    [index-management]      idxmgmt/std.go:295      Loaded index template.
2020-03-24T15:26:54.261-0500    INFO    [index-management]      idxmgmt/std.go:306      Write alias successfully generated.
2020-03-24T15:26:54.261-0500    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elasticdev09.xyzdm.com:9200)) established
2020-03-24T15:26:54.265-0500    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Platinum
2020-03-24T15:26:54.265-0500    INFO    [index-management]      idxmgmt/std.go:258      Auto ILM enable success.
2020-03-24T15:26:54.267-0500    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T15:26:54.267-0500    INFO    [index-management]      idxmgmt/std.go:271      ILM policy successfully loaded.
2020-03-24T15:26:54.267-0500    INFO    [index-management]      idxmgmt/std.go:410      Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
2020-03-24T15:26:54.267-0500    INFO    [index-management]      idxmgmt/std.go:415      Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
2020-03-24T15:26:54.267-0500    INFO    [index-management]      idxmgmt/std.go:449      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.
2020-03-24T15:26:54.267-0500    INFO    [index-management]      idxmgmt/std.go:453      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-24T15:26:54.303-0500    INFO    template/load.go:89     Template metricbeat-7.6.0 already exists and will not be overwritten.
2020-03-24T15:26:54.303-0500    INFO    [index-management]      idxmgmt/std.go:295      Loaded index template.
2020-03-24T15:26:54.313-0500    INFO    [index-management]      idxmgmt/std.go:306      Write alias successfully generated.
2020-03-24T15:26:54.323-0500    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elasticdev10.xyzdm.com:9200)) established
2020-03-24T15:26:54.414-0500    WARN    elasticsearch/client.go:517     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b923295ad536, ext:897189266, loc:(*time.Location)(0x7eb2ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"d618cc58-5236-41d7-833e-0070dde46584","hostname":"host.xyzdm.com","id":"5ee67c4f-da63-43c8-89f8-3d4bf049f871","type":"metricbeat","version":"7.6.0"},"ecs":{"version":"1.4.0"},"event":{"dataset":"system.memory","duration":2218074,"module":"system"},"host":{"name":"host.xyzdm.com"},"metricset":{"name":"memory","period":10000},"service":{"type":"system"},"system":{"memory":{"actual":{"free":3005251584,"used":{"bytes":33731878912,"pct":0.918200}},"free":330768384,"hugepages":{"default_size":2097152,"free":0,"reserved":0,"surplus":0,"swap":{"out":{"fallback":0,"pages":0}},"total":0,"used":{"bytes":0,"pct":0.000000}},"page_stats":{"pgfree":{"pages":6524561947},"pgscan_direct":{"pages":0},"pgscan_kswapd":{"pages":0},"pgsteal_direct":{"pages":0},"pgsteal_kswapd":{"pages":0}},"swap":{"free":0,"in":{"pages":95520},"out":{"pages":415587},"readahead":{"cached":26083,"pages":39989},"total":1048571904,"used":{"bytes":1048571904,"pct":1.000000}},"total":36737130496,"used":{"bytes":36406362112,"pct":0.991000}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}

vee · March 24, 2020, 8:33pm

[user@host metricbeat]$ sudo current_metricbeat/metricbeat -E path.config=/opt/apps/metricbeat -e -c /opt/apps/metricbeat/metricbeat.yml
2020-03-24T15:31:07.754-0500    INFO    instance/beat.go:622    Home path: [/opt/apps/metricbeat/current_metricbeat] Config path: [/opt/apps/metricbeat] Data path: [/opt/apps/metricbeat/current_metricbeat/data] Logs path: [/opt/apps/metricbeat/current_metricbeat/logs]
2020-03-24T15:31:07.754-0500    INFO    instance/beat.go:630    Beat ID: 5ee67c4f-da63-43c8-89f8-3d4bf049f871
2020-03-24T15:31:07.755-0500    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2020-03-24T15:31:07.755-0500    INFO    [beat]  instance/beat.go:958    Beat info       {"system_info": {"beat": {"path": {"config": "/opt/apps/metricbeat", "data": "/opt/apps/metricbeat/current_metricbeat/data", "home": "/opt/apps/metricbeat/current_metricbeat", "logs": "/opt/apps/metricbeat/current_metricbeat/logs"}, "type": "metricbeat", "uuid": "5ee67c4f-da63-43c8-89f8-3d4bf049f871"}}}
2020-03-24T15:31:07.755-0500    INFO    [beat]  instance/beat.go:967    Build info      {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:10:10.000Z", "version": "7.6.0"}}}
2020-03-24T15:31:07.755-0500    INFO    [beat]  instance/beat.go:970    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.13.7"}}}
2020-03-24T15:31:07.759-0500    INFO    [beat]  instance/beat.go:974    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-01-08T08:03:45-06:00","containerized":false,"name":"host.xyzdm.com","ip":["127.0.0.1/8","10.50.80.67/20"],"kernel_version":"3.10.0-1062.9.1.el7.x86_64","mac":["00:50:56:b2:6d:9e"],"os":{"family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux Server","version":"7.7 (Maipo)","major":7,"minor":7,"patch":0,"codename":"Maipo"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"6f72e7e35f214eab9e167ef87d5bb515"}}}
2020-03-24T15:31:07.760-0500    INFO    [beat]  instance/beat.go:1003   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/opt/apps/metricbeat", "exe": "/opt/apps/metricbeat/metricbeat-7.6.0-linux-x86_64/metricbeat", "name": "metricbeat", "pid": 17173, "ppid": 17168, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-03-24T15:31:06.820-0500"}}}
2020-03-24T15:31:07.760-0500    INFO    instance/beat.go:298    Setup Beat: metricbeat; Version: 7.6.0
2020-03-24T15:31:07.760-0500    INFO    [index-management]      idxmgmt/std.go:182      Set output.elasticsearch.index to 'metricbeat-7.6.0' as ILM is enabled.
2020-03-24T15:31:07.761-0500    INFO    elasticsearch/client.go:174     Elasticsearch url: http://elasticdev09.xyzdm.com:9200
2020-03-24T15:31:07.761-0500    INFO    elasticsearch/client.go:174     Elasticsearch url: http://elasticdev10.xyzdm.com:9200
2020-03-24T15:31:07.761-0500    INFO    [publisher]     pipeline/module.go:110  Beat name: host.xyzdm.com
2020-03-24T15:31:07.773-0500    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: nfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T15:31:07.773-0500    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %snfs, smbfs, autofs, rpc_pipefs, rootfs
2020-03-24T15:31:07.774-0500    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-03-24T15:31:07.774-0500    INFO    instance/beat.go:439    metricbeat start running.
2020-03-24T15:31:07.775-0500    INFO    cfgfile/reload.go:171   Config reloader started
2020-03-24T15:31:11.307-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticdev10.xyzdm.com:9200))
2020-03-24T15:31:11.308-0500    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://elasticdev09.xyzdm.com:9200))
2020-03-24T15:31:11.312-0500    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T15:31:11.312-0500    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.0
2020-03-24T15:31:11.314-0500    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Platinum
2020-03-24T15:31:11.315-0500    INFO    [index-management]      idxmgmt/std.go:258      Auto ILM enable success.
2020-03-24T15:31:11.316-0500    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T15:31:11.316-0500    INFO    [index-management]      idxmgmt/std.go:271      ILM policy successfully loaded.
2020-03-24T15:31:11.316-0500    INFO    [index-management]      idxmgmt/std.go:410      Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
2020-03-24T15:31:11.316-0500    INFO    [index-management]      idxmgmt/std.go:415      Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
2020-03-24T15:31:11.316-0500    INFO    [index-management]      idxmgmt/std.go:449      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.
2020-03-24T15:31:11.316-0500    INFO    [index-management]      idxmgmt/std.go:453      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-24T15:31:11.717-0500    INFO    template/load.go:109    Try loading template metricbeat-7.6.0 to Elasticsearch
2020-03-24T15:31:11.958-0500    INFO    template/load.go:101    template with name 'metricbeat-7.6.0' loaded.
2020-03-24T15:31:11.958-0500    INFO    [index-management]      idxmgmt/std.go:295      Loaded index template.
2020-03-24T15:31:11.992-0500    INFO    [index-management]      idxmgmt/std.go:304      Write alias exists already
2020-03-24T15:31:11.992-0500    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://elasticdev10.xyzdm.com:9200)) established
2020-03-24T15:31:11.995-0500    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Platinum
2020-03-24T15:31:11.995-0500    INFO    [index-management]      idxmgmt/std.go:258      Auto ILM enable success.
2020-03-24T15:31:11.997-0500    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-24T15:31:11.997-0500    INFO    [index-management]      idxmgmt/std.go:271      ILM policy successfully loaded.
2020-03-24T15:31:11.997-0500    INFO    [index-management]      idxmgmt/std.go:410      Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
2020-03-24T15:31:11.997-0500    INFO    [index-management]      idxmgmt/std.go:415      Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
2020-03-24T15:31:11.997-0500    INFO    [index-management]      idxmgmt/std.go:449      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.
2020-03-24T15:31:11.997-0500    INFO    [index-management]      idxmgmt/std.go:453      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-24T15:31:12.026-0500    INFO    template/load.go:89     Template metricbeat-7.6.0 already exists and will not be overwritten.
2020-03-24T15:31:12.026-0500    INFO    [index-management]      idxmgmt/std.go:295      Loaded index template.
2020-03-24T15:31:12.590-0500    WARN    elasticsearch/client.go:517     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b9639216674a, ext:2620664027, loc:(*time.Location)(0x7eb2ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"42ec7695-d938-410c-8c1f-280909c8e1b2","hostname":"host.xyzdm.com","id":"5ee67c4f-da63-43c8-89f8-3d4bf049f871","type":"metricbeat","version":"7.6.0"},"ecs":{"version":"1.4.0"},"event":{"dataset":"system.uptime","duration":15299,"module":"system"},"host":{"name":"host.xyzdm.com"},"metricset":{"name":"uptime","period":900000},"service":{"type":"system"},"system":{"uptime":{"duration":{"ms":6589645000}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}
2020-03-24T15:31:12.591-0500    WARN    elasticsearch/client.go:517     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf96b9639bff9240, ext:2786939853, loc:(*time.Location)(0x7eb2ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"42ec7695-d938-410c-8c1f-280909c8e1b2","hostname":"host.xyzdm.com","id":"5ee67c4f-da63-43c8-89f8-3d4bf049f871","type":"metricbeat","version":"7.6.0"},"ecs":{"version":"1.4.0"},"event":{"dataset":"system.process","duration":360255873,"module":"system"},"host":{"name":"host.xyzdm.com"},"metricset":{"name":"process","period":10000},"process":{"args":["/usr/lib/systemd/systemd","--switched-root","--system","--deserialize","22"],"executable":"/usr/lib/systemd/systemd","name":"systemd","pgid":1,"pid":1,"ppid":0,"working_directory":"/"},"service":{"type":"system"},"system":{"process":{"cmdline":"/usr/lib/systemd/systemd --switched-root --system --deserialize 22","cpu":{"start_time":"2020-01-08T14:03:45.000Z","total":{"norm":{"pct":0.000000},"pct":0.000000,"value":0.000000}},"fd":{"limit":{"hard":1048576,"soft":1048576},"open":72},"memory":{"rss":{"bytes":5685248,"pct":0.000200},"share":3137536,"size":207294464},"state":"sleeping"}},"user":{"name":"root"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}

shaunak · March 24, 2020, 9:36pm

Hmmm, could you check one more thing in Elasticsearch please?

GET _ilm/policy

Is there one (or more) returned for Metricbeat?

Shaunak

vee · March 25, 2020, 1:46pm

Yes, I see multiple returned for metricbeat:

GET _ilm/policy

{
  "metricbeat" : {
    "version" : 1,
    "modified_date" : "2020-03-24T20:22:37.095Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "metricbeat-7.5.1" : {
    "version" : 1,
    "modified_date" : "2020-02-11T15:49:10.496Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "metricbeat-7.5.2" : {
    "version" : 1,
    "modified_date" : "2020-03-23T14:30:26.956Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "metricbeat-7.5.3" : {
    "version" : 8,
    "modified_date" : "2020-03-20T20:47:28.180Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        }
      }
    }
  },
  "slm-history-ilm-policy" : {
    "version" : 1,
    "modified_date" : "2019-12-12T21:44:42.205Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "delete" : {
          "min_age" : "90d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  },
  "watch-history-ilm-policy" : {
    "version" : 1,
    "modified_date" : "2019-12-12T21:44:42.410Z",
    "policy" : {
      "phases" : {
        "delete" : {
          "min_age" : "7d",
          "actions" : {
            "delete" : { }
          }
        }
      }
    }
  }
}

shaunak · March 25, 2020, 2:14pm

Thanks, that looks OK to me. The upgrade might've been a red herring.

Looking at the error in your logs (pre- and post-upgrade):

Cannot write to a field alias [event.module]

I'm now wondering if there's an old Metricbeat template that's resulting in a mapping conflict, similar to the one described in Filebeat errors after update to version 7 - #11. Can you look through your Metricbeat templates (output of GET _template/metricbeat-*) to see if there's one that matches the index pattern metricbeat-*?

vee · March 25, 2020, 3:27pm

I checked that and removed any metricbeat templates newer than v7. However, I still see this error even though the template does not exist when I tried to start metricbeat:

Is there some sort of cache that should be cleared to flush out any existing templates?

2020-03-25T09:39:59.003-0500    INFO    template/load.go:89     Template metricbeat-7.6.0 already exists and will not be overwritten.
2020-03-25T09:39:59.004-0500    INFO    [index-management]      idxmgmt/std.go:295      Loaded index template.
2020-03-25T09:40:00.448-0500    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://elasticdev09.xyzdm.com:9200)): Connection marked as failed because the onConnect callback failed: resource 'metricbeat-7.6.0' exists, but it is not an alias
2020-03-25T09:40:00.448-0500    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://elasticdev09.xyzdm.com:9200)) with 1 reconnect attempt(s)
2020-03-25T09:40:00.449-0500    INFO    elasticsearch/client.go:757     Attempting to connect to Elasticsearch version 7.5.0
2020-03-25T09:40:00.458-0500    INFO    [license]       licenser/es_callback.go:50      Elasticsearch license: Platinum
2020-03-25T09:40:00.458-0500    INFO    [index-management]      idxmgmt/std.go:258      Auto ILM enable success.
2020-03-25T09:40:00.459-0500    INFO    [index-management.ilm]  ilm/std.go:139  do not generate ilm policy: exists=true, overwrite=false
2020-03-25T09:40:00.459-0500    INFO    [index-management]      idxmgmt/std.go:271      ILM policy successfully loaded.
2020-03-25T09:40:00.460-0500    INFO    [index-management]      idxmgmt/std.go:410      Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
2020-03-25T09:40:00.460-0500    INFO    [index-management]      idxmgmt/std.go:415      Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
2020-03-25T09:40:00.460-0500    INFO    [index-management]      idxmgmt/std.go:449      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.

shaunak · March 25, 2020, 3:41pm

No, there is no cache.

I see that now the resource 'metricbeat-7.6.0' exists, but it is not an alias error has come back.

In addition to deleting any Metricbeat templates with index_patterns as metricbeat-* (i.e. a template that would very broadly match all indices starting with metricbeat-), could you also follow the steps in Error when startingmetricbeat with v7.5.3, but for 7.6.0 this time?

Shaunak

vee · March 25, 2020, 7:17pm

Sure, I did run the same commands to delete the template and any indices earlier.

DELETE metricbeat-7.6.0
DELETE _template/metricbeat-7.6.0

GET _template/metricbeat-7*
GET _cat/indices/metricbeat-7*
GET _cat/aliases/metricbeat-7*?v

# GET _template/metricbeat-7*
{ }


# GET _cat/indices/metricbeat-7*


# GET _cat/aliases/metricbeat-7*?v
alias index filter routing.index routing.search is_write_index

./metricbeat version
metricbeat version 7.6.0 (amd64), libbeat 7.6.0 [6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c built 2020-02-05 23:10:10 +0000 UTC]

current_metricbeat/metricbeat -E path.config=/opt/apps/metricbeat -e -c /opt/apps/metricbeat/metricbeat.yml

2020-03-25T14:14:34.043-0500    INFO    [index-management]      idxmgmt/std.go:453      Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-25T14:14:34.078-0500    INFO    template/load.go:89     Template metricbeat-7.6.0 already exists and will not be overwritten.
2020-03-25T14:14:34.078-0500    INFO    [index-management]      idxmgmt/std.go:295      Loaded index template.
2020-03-25T14:14:34.182-0500    WARN    elasticsearch/client.go:517     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf97094615fde099, ext:1000853224, loc:(*time.Location)(0x7eb2ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"bbeaeaa5-aa70-4424-9344-3ae56d58efff","hostname":"host.xyz.com","id":"5ee67c4f-da63-43c8-89f8-3d4bf049f871","type":"metricbeat","version":"7.6.0"},"ecs":{"version":"1.4.0"},"event":{"dataset":"system.uptime","duration":16125,"module":"system"},"host":{"name":"host.xyz.com"},"metricset":{"name":"uptime","period":900000},"service":{"type":"system"},"system":{"uptime":{"duration":{"ms":6671447000}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Cannot write to a field alias [event.module]."}}

vee · March 25, 2020, 7:23pm

Attaching config file just incase something obvious is missing once:
(note: setup.template.enabled is commented out)

/opt/apps/metricbeat/metricbeat.yml:

metricbeat.config.modules:
  path.config: /opt/apps/metricbeat/current_metricbeat

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

metricbeat.modules:
- module: system
  metricsets: [cpu]
  cpu.metrics: [percentages,normalized_percentages]

- module: system
  metricsets: [process]
  processes: ['.*']
  process.cgroups.enabled: false
  process.cmdline.cache.enabled: true
  process.include_top_n:
    by_cpu: 20      # include top 20  processes by CPU
    by_memory: 20   # include top 20  processes by memory

- module: system
  period: 10m
  metricsets:
    - filesystem
    - fsstat
  filesystem.ignore_types: [nfs, smbfs, autofs, rpc_pipefs, rootfs]
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(dev|run|sys|fs|host|lib|tmpfs|boot|proc)($|/)'

- module: system
  period: 10s
  metricsets:
    - load
    - memory
    - network
    - process_summary
    - diskio

- module: system
  period: 15m
  metricsets:
    - uptime

- module: system
  metricsets: [core]
  core.metrics: [percentages]

#setup.template.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

output.elasticsearch:
 hosts: ["http://host09.host.com:9200" , "http://host10.host.com:9200"]
 username: elastic
 password: elastic
logging.level: info
logging.to_files: true
logging.files:
 path: /opt/logs/metricbeat
 name: metricbeat.log
 keepfiles: 7