MySQL module: Configuration

Md_Saidur_Rahman · April 22, 2020, 3:20pm

I've configured mysql moduld for MariaDB but didn't get any log in kibana dashboard. Configuration and Dashboards attached bellow:

MySQL module configuration:

module: mysql
#metricsets:
- status
- galera_status
  period: 10s
  hosts: ["tcp(10.88.220.200:3306)/"]
  username: testuser
  password: testpass

DB User: testuser
Password: testpass

Module Status:

MySQL Metrics Status:

Is there anything missing or misconfiguration for MySQL module?

Ref: https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-mysql.html

shaunak · April 22, 2020, 9:13pm

Hi @Md_Saidur_Rahman,

Let's examine the documents in your Metricbeat indices. Could you please post the results of the following query?

POST metricbeat-*/_search?filter_path=hits.hits._source.mysql,hits.total
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "event.module": "mysql"
        }
      }
    }
  },
  "collapse": {
    "field": "event.dataset"
  }
}

Thanks,

Shaunak

Md_Saidur_Rahman · April 23, 2020, 4:58am

Hi, Mr. @shaunak,
Thank you for your response. I've attached the output of your mentioned query.

Best Regard,
Md. Saidur Rahman

shaunak · April 23, 2020, 2:21pm

That's quite strange. It seems to indicate that there are 10000 documents for the mysql module but none of them have the mysql field in them!

Now I suspect there might be some error happening so we need to take a deeper look. Could you please post the results of the following query? It's the same query as before, just without the filter_path so we can see the entire documents this time.

POST metricbeat-*/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "event.module": "mysql"
        }
      }
    }
  },
  "collapse": {
    "field": "event.dataset"
  }
}

Md_Saidur_Rahman · April 23, 2020, 4:29pm

Getting access denied for user root but currently, I'm using another user "testuser" in mysql module configuration. I don't know why it stored old information in the module. How can I remove the old cache?

Query Result:
"Error 1045: Access denied for user 'root'@'test.kdb.com' (using password: YES)"

Best Regards,
Md. Saidur Rahman

shaunak · April 23, 2020, 4:58pm

There is no cache AFAIK. Let's try one more query, just to make sure we are looking at the latest errors and not some old ones:

POST metricbeat-*/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": {
          "event.module": "mysql"
        }
      }
    }
  },
  "collapse": {
    "field": "event.dataset"
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}

Shaunak

Md_Saidur_Rahman · April 23, 2020, 5:21pm

Query's output:

{
"took" : 99,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : null,
"hits" : [
{
"_index" : "metricbeat-7.2.1-2020.04.16-000001",
"_type" : "_doc",
"_id" : "6O0JqHEBhhYgzDqa8HzS",
"_score" : null,
"_source" : {
"@timestamp" : "2020-04-23T17:16:59.356Z",
"service" : {
"address" : "10.88.220.200:3306",
"type" : "mysql"
},
"error" : {
"message" : "Error 1045: Access denied for user 'root'@'test.kdb.com' (using password: YES)"
},
"agent" : {
"id" : "fff474f2-e90c-4841-b01b-ec792d7b66ad",
"version" : "7.2.1",
"type" : "metricbeat",
"ephemeral_id" : "9eee171a-734d-4db9-b0c1-49ceca993c7a",
"hostname" : "test.kdb.com"
},
...

BR,
Md. Saidur Rahman

shaunak · April 23, 2020, 6:52pm

Okay, thanks. So Metricbeat is still trying to use the root user to connect to MariaDB for some reason.

Could you please post your complete metricbeat.yml configuration here, please? Please be sure to redact any sensitive information in it before posting.

Similarly, could you please post your complete mysql module configuration here as well?

Thanks,

Shaunak

Md_Saidur_Rahman · April 23, 2020, 7:50pm

Well, I've attached metricbeat.yml and **mysql module configuration file here.

#metricbeat.yml: in metricbeat.yml file I've only changed kibana and elasticsearch information.

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  host: 10.88.250.111
  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.88.250.111:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "pass"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

#mysql.yml:

- module: mysql
  metricsets: ["status"]
  hosts: ["tcp(10.88.250.118:3306)/"]
  username: testuser
  password: testpass

I've followed below mentioned link:
https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-mysql.html
Is there anything needed for configuring mysql module? FYI, I'm getting others log like uptime using heartbeat, System module using metricbeat.

Thanks in advance.

BR,
Md. Saidur Rahman

shaunak · April 23, 2020, 10:39pm

Hmmm, all of that looks right to me.

I just noticed another thing from your earlier post. The service.address in the document is 10.88.220.200:3306, which is not the same as the one you have configured in your modules.d/mysql.yml file.

Can you restart Metricbeat with the -d"*" CLI flag so it produces debug-level logs? And then please post the logs from the first 10 seconds or so over here. I want to see if there are any log messages about connecting to MySQL and if they give us some clue.

Thanks,

Shaunak

Md_Saidur_Rahman · April 24, 2020, 9:33pm

Yes, you are right. I've configured multiple MySQL modules on different servers. That's why here you getting different IP. I've attached metricbeat debug log here:

2020-04-25T03:30:29.335+0600    INFO    instance/beat.go:606    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-04-25T03:30:29.335+0600    DEBUG   [beat]  instance/beat.go:658    Beat metadata path: /var/lib/metricbeat/meta.json
2020-04-25T03:30:29.335+0600    INFO    instance/beat.go:614    Beat ID: 8fb83fcc-7a27-4d16-abbe-8d2997882b38
2020-04-25T03:30:29.338+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:164    add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-04-25T03:30:29.445+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for qcloud after 107.001219ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/placement/zone: dial tcp: lookup metadata.tencentyun.com on 8.8.8.8:53: no such host, metadata={}]
2020-04-25T03:30:32.339+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for digitalocean after 3.000276083s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get http://169.254.169.254/metadata/v1.json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), metadata={}]
2020-04-25T03:30:32.339+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for gcp after 3.000468081s. result=[provider:gcp, error=failed requesting gcp metadata: Get http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2020-04-25T03:30:32.339+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for openstack after 3.000581019s. result=[provider:openstack, error=failed requesting openstack metadata: Get http://169.254.169.254/2009-04-04/meta-data/hostname: dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2020-04-25T03:30:32.339+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:203    add_cloud_metadata: timed-out waiting for all responses
2020-04-25T03:30:32.339+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:167    add_cloud_metadata: fetchMetadata ran for 3.000719406s
2020-04-25T03:30:32.339+0600    INFO    add_cloud_metadata/add_cloud_metadata.go:347    add_cloud_metadata: hosting provider type not detected.
2020-04-25T03:30:32.339+0600    DEBUG   [processors]    processors/processor.go:93      Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null
2020-04-25T03:30:32.339+0600    DEBUG   [seccomp]       seccomp/seccomp.go:109  Loading syscall filter  {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-04-25T03:30:32.340+0600    INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
2020-04-25T03:30:32.340+0600    INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"}}}
2020-04-25T03:30:32.340+0600    INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "dd3f47f0fb299aa5de9c5c1468faacc1b9b3c27f", "libbeat": "7.2.1", "time": "2019-07-24T17:12:03.000Z", "version": "7.2.1"}}}
2020-04-25T03:30:32.340+0600    INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2020-04-25T03:30:32.342+0600    INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-03-13T02:09:57+06:00","containerized":false,"name":"ltdb.kpp.com","ip":["127.0.0.1/8","::1/128","10.88.250.118/24","fe80::d10a:cda0:6840:fb99/64","192.168.122.1/24"],"kernel_version":"4.1.12-124.33.4.el7uek.x86_64","mac":["00:0c:29:5e:60:17","52:54:00:7c:8f:ec","52:54:00:7c:8f:ec"],"os":{"family":"","platform":"ol","name":"Oracle Linux Server","version":"7.8","major":7,"minor":8,"patch":0},"timezone":"+06","timezone_offset_sec":21600,"id":"8e10311cf2a147a2ab615c47becc2358"}}}
2020-04-25T03:30:32.343+0600    INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 4444, "ppid": 2063, "seccomp": {"mode":"filter"}, "start_time": "2020-04-25T03:30:28.760+0600"}}}
2020-04-25T03:30:32.344+0600    INFO    instance/beat.go:292    Setup Beat: metricbeat; Version: 7.2.1
2020-04-25T03:30:32.344+0600    DEBUG   [beat]  instance/beat.go:318    Initializing output plugins
2020-04-25T03:30:32.344+0600    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'metricbeat-7.2.1' as ILM is enabled.
2020-04-25T03:30:32.344+0600    INFO    elasticsearch/client.go:166     Elasticsearch url: http://10.88.250.111:9200
2020-04-25T03:30:32.345+0600    DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2020-04-25T03:30:32.345+0600    INFO    [publisher]     pipeline/module.go:97   Beat name: ltdb.kpp.com
2020-04-25T03:30:32.345+0600    DEBUG   [modules]       beater/metricbeat.go:103        Register [ModuleFactory:[aws, docker, elasticsearch, kibana, mongodb, mssql, mysql, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, aws/cloudwatch, aws/ec2, aws/s3_daily_storage, aws/s3_request, aws/sqs, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/raid, system/socket, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server]]
2020-04-25T03:30:32.346+0600    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-04-25T03:30:32.346+0600    INFO    instance/beat.go:421    metricbeat start running.
2020-04-25T03:30:32.346+0600    DEBUG   [cfgfile]       cfgfile/reload.go:134   Checking module configs from: /etc/metricbeat/modules.d/*.yml
2020-04-25T03:30:32.347+0600    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: /etc/metricbeat/modules.d/mysql.yml
2020-04-25T03:30:32.349+0600    DEBUG   [cfgfile]       cfgfile/reload.go:148   Number of module configs found: 1
2020-04-25T03:30:32.349+0600    INFO    cfgfile/reload.go:172   Config reloader started
2020-04-25T03:30:32.349+0600    DEBUG   [cfgfile]       cfgfile/reload.go:198   Scan for new config files
2020-04-25T03:30:32.350+0600    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: /etc/metricbeat/modules.d/mysql.yml
2020-04-25T03:30:32.350+0600    DEBUG   [cfgfile]       cfgfile/reload.go:217   Number of module configs found: 1
2020-04-25T03:30:32.350+0600    DEBUG   [reload]        cfgfile/list.go:62      Starting reload procedure, current runners: 0
2020-04-25T03:30:32.350+0600    DEBUG   [reload]        cfgfile/list.go:80      Start list: 1, Stop list: 0
2020-04-25T03:30:32.351+0600    DEBUG   [reload]        cfgfile/list.go:101     Starting runner: mysql [metricsets=1]
2020-04-25T03:30:32.351+0600    DEBUG   [module]        module/wrapper.go:117   Starting Wrapper[name=mysql, len(metricSetWrappers)=1]
2020-04-25T03:30:32.351+0600    INFO    cfgfile/reload.go:227   Loading of config files completed.
2020-04-25T03:30:32.351+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=mysql, name=status, host=10.88.250.118:3306]
2020-04-25T03:30:32.361+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-24T21:30:32.351Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "event": {
    "dataset": "mysql.status",
    "module": "mysql",
    "duration": 8312479
  },
  "metricset": {
    "name": "status"
  },
  "ecs": {
    "version": "1.0.0"
  },

BR,
Saidur

shaunak · April 24, 2020, 11:17pm

That's only about 3 seconds worth of logs. Is that all of it? If not, could you post more please? As I said in my last comment, the first 10 seconds or so should be good enough.

Md_Saidur_Rahman · April 25, 2020, 6:47am

If you need more log I can provide. Most-probably this is the reason.

Getting warn log: "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}"

2020-04-25T12:17:27.898+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac118b62, ext:3129245425, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"mysql.status", "duration":13895842, "module":"mysql"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"status"}, "mysql":common.MapStr{"status":common.MapStr{"aborted":common.MapStr{"clients":4455, "connects":40436}, "binlog":common.MapStr{"cache":common.MapStr{"disk_use":654, "use":5608}}, "bytes":common.MapStr{"received":273635898, "sent":609549224}, "command":common.MapStr{"delete":71, "insert":304352, "select":335184, "update":1394}, "connections":75527, "created":common.MapStr{"tmp":common.MapStr{"disk_tables":42392, "files":190, "tables":268100}}, "delayed":common.MapStr{"errors":0, "insert_threads":0, "writes":0}, "flush_commands":1, "max_used_connections":311, "open":common.MapStr{"files":6, "streams":0, "tables":64}, "opened_tables":52359, "threads":common.MapStr{"cached":21, "connected":139, "created":874, "running":6}}}, "service":common.MapStr{"address":"10.88.250.118:3306", "type":"mysql"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

Md_Saidur_Rahman · April 25, 2020, 10:37am

Debug log-1

2020-04-25T12:17:23.716+0600    INFO    instance/beat.go:606    Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-04-25T12:17:23.717+0600    DEBUG   [beat]  instance/beat.go:658    Beat metadata path: /var/lib/metricbeat/meta.json
2020-04-25T12:17:23.717+0600    INFO    instance/beat.go:614    Beat ID: 8fb83fcc-7a27-4d16-abbe-8d2997882b38
2020-04-25T12:17:23.721+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:164    add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-04-25T12:17:23.828+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:196    add_cloud_metadata: received disposition for qcloud after 107.085645ms. result=[provider:qcloud, error=failed requesting qcloud metadata: Get http://metadata.tencentyun.com/meta-data/instance-id: dial tcp: lookup metadata.tencentyun.com on 8.8.8.8:53: no such host, metadata={}]
2020-04-25T12:17:26.722+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:203    add_cloud_metadata: timed-out waiting for all responses
2020-04-25T12:17:26.722+0600    DEBUG   [filters]       add_cloud_metadata/add_cloud_metadata.go:167    add_cloud_metadata: fetchMetadata ran for 3.000538389s
2020-04-25T12:17:26.722+0600    INFO    add_cloud_metadata/add_cloud_metadata.go:347    add_cloud_metadata: hosting provider type not detected.
2020-04-25T12:17:26.722+0600    DEBUG   [processors]    processors/processor.go:93      Generated new processors: add_host_metadata=[netinfo.enabled=[false], cache.ttl=[5m0s]], add_cloud_metadata=null
2020-04-25T12:17:26.722+0600    DEBUG   [seccomp]       seccomp/seccomp.go:109  Loading syscall filter  {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchown","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-04-25T12:17:26.722+0600    INFO    [seccomp]       seccomp/seccomp.go:116  Syscall filter successfully installed
2020-04-25T12:17:26.723+0600    INFO    [beat]  instance/beat.go:902    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"}}}
2020-04-25T12:17:26.723+0600    INFO    [beat]  instance/beat.go:911    Build info      {"system_info": {"build": {"commit": "dd3f47f0fb299aa5de9c5c1468faacc1b9b3c27f", "libbeat": "7.2.1", "time": "2019-07-24T17:12:03.000Z", "version": "7.2.1"}}}
2020-04-25T12:17:26.723+0600    INFO    [beat]  instance/beat.go:914    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2020-04-25T12:17:26.727+0600    INFO    [beat]  instance/beat.go:918    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-03-13T02:09:57+06:00","containerized":false,"name":"ltdb.kpp.com","ip":["127.0.0.1/8","::1/128","10.88.250.118/24","fe80::d10a:cda0:6840:fb99/64","192.168.122.1/24"],"kernel_version":"4.1.12-124.33.4.el7uek.x86_64","mac":["00:0c:29:5e:60:17","52:54:00:7c:8f:ec","52:54:00:7c:8f:ec"],"os":{"family":"","platform":"ol","name":"Oracle Linux Server","version":"7.8","major":7,"minor":8,"patch":0},"timezone":"+06","timezone_offset_sec":21600,"id":"8e10311cf2a147a2ab615c47becc2358"}}}
2020-04-25T12:17:26.728+0600    INFO    [beat]  instance/beat.go:947    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 2319, "ppid": 2137, "seccomp": {"mode":"filter"}, "start_time": "2020-04-25T12:17:23.130+0600"}}}
2020-04-25T12:17:26.728+0600    INFO    instance/beat.go:292    Setup Beat: metricbeat; Version: 7.2.1
2020-04-25T12:17:26.728+0600    DEBUG   [beat]  instance/beat.go:318    Initializing output plugins
2020-04-25T12:17:26.728+0600    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'metricbeat-7.2.1' as ILM is enabled.
2020-04-25T12:17:26.729+0600    INFO    elasticsearch/client.go:166     Elasticsearch url: http://10.88.250.111:9200
2020-04-25T12:17:26.729+0600    DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2020-04-25T12:17:26.729+0600    INFO    [publisher]     pipeline/module.go:97   Beat name: ltdb.kpp.com
2020-04-25T12:17:26.730+0600    DEBUG   [modules]       beater/metricbeat.go:103        Register [ModuleFactory:[aws, docker, elasticsearch, kibana, mongodb, mssql, mysql, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, aws/cloudwatch, aws/ec2, aws/s3_daily_storage, aws/s3_request, aws/sqs, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/state_container, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_statefulset, kubernetes/system, kubernetes/volume, kvm/dommemstat, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/process_summary, system/raid, system/socket, system/socket_summary, system/uptime, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server]]
2020-04-25T12:17:26.730+0600    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2020-04-25T12:17:26.730+0600    INFO    instance/beat.go:421    metricbeat start running.
2020-04-25T12:17:26.730+0600    DEBUG   [cfgfile]       cfgfile/reload.go:134   Checking module configs from: /etc/metricbeat/modules.d/*.yml
2020-04-25T12:17:26.731+0600    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: /etc/metricbeat/modules.d/mysql.yml
2020-04-25T12:17:26.731+0600    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: /etc/metricbeat/modules.d/system.yml
2020-04-25T12:17:26.732+0600    DEBUG   [cfgfile]       cfgfile/reload.go:148   Number of module configs found: 4
2020-04-25T12:17:26.734+0600    DEBUG   [system.process]        process/process.go:84   process cgroup data collection is enabled, using hostfs=''
2020-04-25T12:17:26.736+0600    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, pipefs, devpts, hugetlbfs, autofs, pstore, mqueue, selinuxfs, binfmt_misc, rpc_pipefs, nfs, nfs4
2020-04-25T12:17:26.736+0600    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, pipefs, devpts, hugetlbfs, autofs, pstore, mqueue, selinuxfs, binfmt_misc, rpc_pipefs, nfs, nfs4
2020-04-25T12:17:26.737+0600    INFO    cfgfile/reload.go:172   Config reloader started
2020-04-25T12:17:26.737+0600    DEBUG   [cfgfile]       cfgfile/reload.go:198   Scan for new config files
2020-04-25T12:17:26.737+0600    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: /etc/metricbeat/modules.d/mysql.yml
2020-04-25T12:17:26.737+0600    DEBUG   [cfgfile]       cfgfile/cfgfile.go:175  Load config from file: /etc/metricbeat/modules.d/system.yml
2020-04-25T12:17:26.738+0600    DEBUG   [cfgfile]       cfgfile/reload.go:217   Number of module configs found: 4
2020-04-25T12:17:26.738+0600    DEBUG   [reload]        cfgfile/list.go:62      Starting reload procedure, current runners: 0
2020-04-25T12:17:26.738+0600    DEBUG   [reload]        cfgfile/list.go:80      Start list: 4, Stop list: 0
2020-04-25T12:17:26.739+0600    DEBUG   [reload]        cfgfile/list.go:101     Starting runner: mysql [metricsets=1]
2020-04-25T12:17:26.739+0600    DEBUG   [module]        module/wrapper.go:117   Starting Wrapper[name=mysql, len(metricSetWrappers)=1]
2020-04-25T12:17:26.739+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=mysql, name=status, host=10.88.250.118:3306]
2020-04-25T12:17:26.739+0600    DEBUG   [system.process]        process/process.go:84   process cgroup data collection is enabled, using hostfs=''
2020-04-25T12:17:26.740+0600    DEBUG   [reload]        cfgfile/list.go:101     Starting runner: system [metricsets=7]
2020-04-25T12:17:26.740+0600    DEBUG   [module]        module/wrapper.go:117   Starting Wrapper[name=system, len(metricSetWrappers)=7]
2020-04-25T12:17:26.740+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=process, host=]
2020-04-25T12:17:26.740+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=memory, host=]

Md_Saidur_Rahman · April 25, 2020, 10:40am

@shaunak
Debug log-2:

2020-04-25T12:17:26.740+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=cpu, host=]
2020-04-25T12:17:26.740+0600    DEBUG   [conditions]    conditions/conditions.go:98     New condition regexp: map[]
2020-04-25T12:17:26.740+0600    DEBUG   [processors]    processors/processor.go:93      Generated new processors: drop_event, condition=regexp: map[]
2020-04-25T12:17:26.741+0600    INFO    filesystem/filesystem.go:57     Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, pipefs, devpts, hugetlbfs, autofs, pstore, mqueue, selinuxfs, binfmt_misc, rpc_pipefs, nfs, nfs4
2020-04-25T12:17:26.741+0600    INFO    [system.fsstat] fsstat/fsstat.go:56     Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, pipefs, devpts, hugetlbfs, autofs, pstore, mqueue, selinuxfs, binfmt_misc, rpc_pipefs, nfs, nfs4
2020-04-25T12:17:26.741+0600    DEBUG   [reload]        cfgfile/list.go:101     Starting runner: system [metricsets=2]
2020-04-25T12:17:26.741+0600    DEBUG   [module]        module/wrapper.go:117   Starting Wrapper[name=system, len(metricSetWrappers)=2]
2020-04-25T12:17:26.741+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=socket_summary, host=]
2020-04-25T12:17:26.742+0600    DEBUG   [reload]        cfgfile/list.go:101     Starting runner: system [metricsets=1]
2020-04-25T12:17:26.742+0600    DEBUG   [module]        module/wrapper.go:117   Starting Wrapper[name=system, len(metricSetWrappers)=1]
2020-04-25T12:17:26.742+0600    INFO    cfgfile/reload.go:227   Loading of config files completed.
2020-04-25T12:17:26.743+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "name": "ltdb.kpp.com"
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "event": {
    "module": "system",
    "duration": 1057023,
    "dataset": "system.cpu"
  },
  "metricset": {
    "name": "cpu"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "cpu": {
      "total": {
        "pct": 0
      },
      "cores": 4,
      "user": {
        "pct": 0
      },
      "iowait": {
        "pct": 0
      },
      "irq": {
        "pct": 0
      },
      "nice": {
        "pct": 0
      },
      "softirq": {
        "pct": 0
      },
      "steal": {
        "pct": 0
      },
      "idle": {
        "pct": 0
      },
      "system": {
        "pct": 0
      }
    }
  }
}
2020-04-25T12:17:26.743+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=filesystem, host=]
2020-04-25T12:17:26.744+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=fsstat, host=]
2020-04-25T12:17:26.745+0600    DEBUG   [system.fsstat] fsstat/fsstat.go:87     filesystem: %s total=%d, used=%d, free=%d/home44127686656 3897868288 40229818368
2020-04-25T12:17:26.745+0600    DEBUG   [system.fsstat] fsstat/fsstat.go:87     filesystem: %s total=%d, used=%d, free=%d/107343773696 69901537280 37442236416
2020-04-25T12:17:26.745+0600    DEBUG   [system.fsstat] fsstat/fsstat.go:87     filesystem: %s total=%d, used=%d, free=%d/boot1063256064 413298688 649957376
2020-04-25T12:17:26.746+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.743Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com"
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "system": {
    "filesystem": {
      "used": {
        "pct": 0.3887,
        "bytes": 413298688
      },
      "free": 649957376,
      "device_name": "/dev/sda1",
      "mount_point": "/boot",
      "free_files": 1048205,
      "type": "xfs",
      "total": 1063256064,
      "available": 649957376,
      "files": 1048576
    }
  },
  "event": {
    "duration": 1300581,
    "dataset": "system.filesystem",
    "module": "system"
  },
  "metricset": {
    "name": "filesystem"
  }
}
2020-04-25T12:17:26.746+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.743Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "event": {
    "dataset": "system.filesystem",
    "module": "system",
    "duration": 1334334
  },
  "service": {
    "type": "system"
  },
  "system": {
    "filesystem": {
      "available": 40229818368,
      "total": 44127686656,
      "files": 43114496,
      "mount_point": "/home",
      "used": {
        "bytes": 3897868288,
        "pct": 0.0883
      },
      "device_name": "/dev/mapper/ol-home",
      "free": 40229818368,
      "type": "xfs",
      "free_files": 43112954
    }
  },
  "metricset": {
    "name": "filesystem"
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false
  }
}
2020-04-25T12:17:26.746+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.744Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "metricset": {
    "name": "fsstat"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "fsstat": {
      "total_files": 149016576,
      "total_size": {
        "free": 78322012160,
        "used": 74212704256,
        "total": 152534716416
      },
      "count": 3
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358"
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "event": {
    "dataset": "system.fsstat",
    "module": "system",
    "duration": 1005971
  }
}
2020-04-25T12:17:26.746+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=uptime, host=]
2020-04-25T12:17:26.747+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.746Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "uptime": {
      "duration": {
        "ms": 3751650000
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "name": "ltdb.kpp.com",
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    }
  },
  "agent": {
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat"
  },
  "event": {
    "dataset": "system.uptime",
    "module": "system",
    "duration": 13252
  },
  "metricset": {
    "name": "uptime"
  }
}

Md_Saidur_Rahman · April 25, 2020, 10:42am

Debug log-3:

2020-04-25T12:17:26.748+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "memory": {
      "hugepages": {
        "used": {
          "bytes": 0,
          "pct": 0
        },
        "free": 0,
        "reserved": 0,
        "surplus": 0,
        "default_size": 2097152,
        "total": 0
      },
      "total": 8100167680,
      "used": {
        "pct": 0.9856,
        "bytes": 7983575040
      },
      "free": 116592640,
      "actual": {
        "used": {
          "pct": 0.5672,
          "bytes": 4594659328
        },
        "free": 3505508352
      },
      "swap": {
        "total": 8455712768,
        "used": {
          "bytes": 398835712,
          "pct": 0.0472
        },
        "free": 8056877056
      }
    }
  },
  "event": {
    "dataset": "system.memory",
    "module": "system",
    "duration": 2899813
  },
  "metricset": {
    "name": "memory"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "os": {
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64"
  },
  "agent": {
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat"
  },
  "ecs": {
    "version": "1.0.0"
  }
}
2020-04-25T12:17:26.748+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=network, host=]
2020-04-25T12:17:26.748+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.743Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "system": {
    "filesystem": {
      "free_files": 104518678,
      "free": 37442236416,
      "available": 37442236416,
      "files": 104853504,
      "type": "xfs",
      "device_name": "/dev/mapper/ol-root",
      "total": 107343773696,
      "mount_point": "/",
      "used": {
        "pct": 0.6512,
        "bytes": 69901537280
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com"
  },
  "agent": {
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com"
  },
  "event": {
    "module": "system",
    "duration": 3893315,
    "dataset": "system.filesystem"
  },
  "metricset": {
    "name": "filesystem"
  },
  "service": {
    "type": "system"
  }
}
2020-04-25T12:17:26.748+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=process_summary, host=]
2020-04-25T12:17:26.749+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.748Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "host": {
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "name": "ltdb.kpp.com",
    "containerized": false
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "system": {
    "network": {
      "out": {
        "errors": 0,
        "dropped": 0,
        "packets": 0,
        "bytes": 0
      },
      "name": "virbr0-nic",
      "in": {
        "dropped": 0,
        "bytes": 0,
        "packets": 0,
        "errors": 0
      }
    }
  },
  "event": {
    "module": "system",
    "duration": 400543,
    "dataset": "system.network"
  },
  "metricset": {
    "name": "network"
  },
  "service": {
    "type": "system"
  },
  "ecs": {
    "version": "1.0.0"
  }
}
2020-04-25T12:17:26.749+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.748Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "system": {
    "network": {
      "out": {
        "errors": 0,
        "dropped": 0,
        "packets": 0,
        "bytes": 0
      },
      "name": "virbr0",
      "in": {
        "bytes": 0,
        "packets": 0,
        "errors": 0,
        "dropped": 0
      }
    }
  },
  "event": {
    "module": "system",
    "duration": 424631,
    "dataset": "system.network"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "metricset": {
    "name": "network"
  },
  "service": {
    "type": "system"
  }
}
2020-04-25T12:17:26.749+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.748Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "network": {
      "name": "ens160",
      "in": {
        "errors": 0,
        "dropped": 615,
        "bytes": 18580342905,
        "packets": 104221588
      },
      "out": {
        "packets": 80860650,
        "bytes": 30579072882,
        "errors": 0,
        "dropped": 0
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "architecture": "x86_64",
    "name": "ltdb.kpp.com",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com"
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "metricset": {
    "name": "network"
  },
  "event": {
    "module": "system",
    "duration": 438808,
    "dataset": "system.network"
  }
}
2020-04-25T12:17:26.750+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.748Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "name": "ltdb.kpp.com",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    }
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "network": {
      "name": "lo",
      "in": {
        "packets": 1777995,
        "errors": 0,
        "dropped": 0,
        "bytes": 288227670
      },
      "out": {
        "packets": 1777995,
        "bytes": 288227670,
        "errors": 0,
        "dropped": 0
      }
    }
  },
  "event": {
    "dataset": "system.network",
    "module": "system",
    "duration": 1433876
  },
  "metricset": {
    "name": "network"
  }
}
2020-04-25T12:17:26.740+0600    DEBUG   [module]        module/wrapper.go:179   Starting metricSetWrapper[module=system, name=load, host=]
2020-04-25T12:17:26.750+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.750Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "host": {
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "name": "ltdb.kpp.com",
    "containerized": false,
    "hostname": "ltdb.kpp.com"
  },
  "agent": {
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat"
  },
  "event": {
    "dataset": "system.load",
    "module": "system",
    "duration": 154816
  },
  "metricset": {
    "name": "load"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "load": {
      "15": 0.17,
      "norm": {
        "1": 0.04,
        "5": 0.055,
        "15": 0.0425
      },
      "cores": 4,
      "1": 0.16,
      "5": 0.22
    }
  },
  "ecs": {
    "version": "1.0.0"
  }
}
2020-04-25T12:17:26.753+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.739Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "address": "10.88.250.118:3306",
    "type": "mysql"
  },
  "mysql": {
    "status": {
      "command": {
        "select": 335184,
        "update": 1394,
        "delete": 71,
        "insert": 304352
      },
      "created": {
        "tmp": {
          "files": 190,
          "tables": 268100,
          "disk_tables": 42392
        }
      },
      "binlog": {
        "cache": {
          "disk_use": 654,
          "use": 5608
        }
      },
      "delayed": {
        "errors": 0,
        "insert_threads": 0,
        "writes": 0
      },
      "max_used_connections": 311,
      "open": {
        "files": 6,
        "streams": 0,
        "tables": 64
      },
      "opened_tables": 52359,
      "threads": {
        "created": 874,
        "connected": 139,
        "running": 6,
        "cached": 21
      },
      "bytes": {
        "received": 273635898,
        "sent": 609549224
      },
      "connections": 75527,
      "flush_commands": 1,
      "aborted": {
        "clients": 4455,
        "connects": 40436
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "containerized": false,
    "name": "ltdb.kpp.com",
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358"
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "event": {
    "module": "mysql",
    "duration": 13895842,
    "dataset": "mysql.status"
  },
  "metricset": {
    "name": "status"
  }
}

Md_Saidur_Rahman · April 25, 2020, 10:45am

Debug log:4

2020-04-25T12:17:26.881+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.748Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "event": {
    "dataset": "system.process.summary",
    "module": "system",
    "duration": 132440882
  },
  "metricset": {
    "name": "process_summary"
  },
  "service": {
    "type": "system"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "name": "ltdb.kpp.com",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358"
  },
  "agent": {
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat"
  },
  "system": {
    "process": {
      "summary": {
        "stopped": 0,
        "zombie": 0,
        "unknown": 0,
        "dead": 0,
        "total": 333,
        "sleeping": 331,
        "running": 2,
        "idle": 0
      }
    }
  }
}
2020-04-25T12:17:26.933+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.742Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "os": {
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64"
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "event": {
    "duration": 191403016,
    "dataset": "system.socket.summary",
    "module": "system"
  },
  "metricset": {
    "name": "socket_summary"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "socket": {
      "summary": {
        "udp": {
          "all": {
            "count": 7
          },
          "memory": 12288
        },
        "all": {
          "listening": 17,
          "count": 240
        },
        "tcp": {
          "all": {
            "orphan": 0,
            "count": 233,
            "listening": 17,
            "established": 210,
            "close_wait": 0,
            "time_wait": 6
          },
          "memory": 0
        }
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  }
}
2020-04-25T12:17:27.107+0600    DEBUG   [processes]     process/process.go:434  Filtered top processes down to 10 processes
2020-04-25T12:17:27.110+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "process": {
      "state": "sleeping",
      "fd": {
        "limit": {
          "soft": 1048576,
          "hard": 1048576
        },
        "open": 130
      },
      "cmdline": "/usr/lib/systemd/systemd --system --deserialize 25",
      "cpu": {
        "start_time": "2020-03-12T20:09:57.000Z",
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        }
      },
      "memory": {
        "size": 198934528,
        "rss": {
          "bytes": 8101888,
          "pct": 0.001
        },
        "share": 4968448
      }
    }
  },
  "user": {
    "name": "root"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "os": {
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": ""
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64"
  },
  "process": {
    "pid": 1,
    "ppid": 0,
    "pgid": 1,
    "working_directory": "/",
    "executable": "/usr/lib/systemd/systemd",
    "args": [
      "/usr/lib/systemd/systemd",
      "--system",
      "--deserialize",
      "25"
    ],
    "name": "systemd"
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 369635668
  },
  "metricset": {
    "name": "process"
  },
  "agent": {
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361"
  }
}
2020-04-25T12:17:27.111+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "process": {
      "memory": {
        "size": 0,
        "rss": {
          "bytes": 0,
          "pct": 0
        },
        "share": 0
      },
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-03-12T20:09:57.000Z"
      },
      "fd": {
        "open": 0,
        "limit": {
          "soft": 1024,
          "hard": 4096
        }
      },
      "state": "sleeping"
    }
  },
  "process": {
    "ppid": 0,
    "pgid": 0,
    "name": "kthreadd",
    "pid": 2
  },
  "user": {
    "name": "root"
  },
  "event": {
    "duration": 369690549,
    "dataset": "system.process",
    "module": "system"
  },
  "metricset": {
    "name": "process"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64"
  },
  "agent": {
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat"
  }
}
2020-04-25T12:17:27.111+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "user": {
    "name": "root"
  },
  "metricset": {
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "host": {
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "name": "ltdb.kpp.com"
  },
  "process": {
    "name": "ksoftirqd/0",
    "pid": 3,
    "ppid": 2,
    "pgid": 0
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 369703597
  },
  "system": {
    "process": {
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-03-12T20:09:57.000Z"
      },
      "fd": {
        "open": 0,
        "limit": {
          "soft": 1024,
          "hard": 4096
        }
      },
      "state": "sleeping",
      "memory": {
        "size": 0,
        "rss": {
          "bytes": 0,
          "pct": 0
        },
        "share": 0
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  }
}
2020-04-25T12:17:27.111+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "metricset": {
    "name": "process"
  },
  "process": {
    "pid": 5,
    "ppid": 2,
    "pgid": 0,
    "name": "kworker/0:0H"
  },
  "user": {
    "name": "root"
  },
  "system": {
    "process": {
      "cpu": {
        "total": {
          "pct": 0,
          "norm": {
            "pct": 0
          },
          "value": 0
        },
        "start_time": "2020-03-12T20:09:57.000Z"
      },
      "state": "sleeping",
      "memory": {
        "size": 0,
        "rss": {
          "bytes": 0,
          "pct": 0
        },
        "share": 0
      },
      "fd": {
        "open": 0,
        "limit": {
          "hard": 4096,
          "soft": 1024
        }
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "name": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358"
  },
  "agent": {
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361"
  },
  "event": {
    "duration": 370354567,
    "dataset": "system.process",
    "module": "system"
  },
  "service": {
    "type": "system"
  }
}

Md_Saidur_Rahman · April 25, 2020, 11:01am

2020-04-25T12:17:27.111+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "metricset": {
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "process": {
    "name": "rcu_sched",
    "pid": 7,
    "ppid": 2,
    "pgid": 0
  },
  "event": {
    "module": "system",
    "duration": 370673092,
    "dataset": "system.process"
  },
  "agent": {
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38"
  },
  "host": {
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": ""
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "name": "ltdb.kpp.com"
  },
  "user": {
    "name": "root"
  },
  "system": {
    "process": {
      "state": "running",
      "memory": {
        "size": 0,
        "rss": {
          "bytes": 0,
          "pct": 0
        },
        "share": 0
      },
      "cpu": {
        "total": {
          "norm": {
            "pct": 0
          },
          "value": 0,
          "pct": 0
        },
        "start_time": "2020-03-12T20:09:57.000Z"
      },
      "fd": {
        "limit": {
          "soft": 1024,
          "hard": 4096
        },
        "open": 0
      }
    }
  }
}
2020-04-25T12:17:27.112+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "user": {
    "name": "oracle"
  },
  "event": {
    "duration": 370923902,
    "dataset": "system.process",
    "module": "system"
  },
  "agent": {
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "metricset": {
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "process": {
      "fd": {
        "open": 16,
        "limit": {
          "soft": 65536,
          "hard": 65536
        }
      },
      "state": "sleeping",
      "memory": {
        "size": 2895458304,
        "rss": {
          "bytes": 1653510144,
          "pct": 0.2041
        },
        "share": 1645064192
      },
      "cmdline": "ora_dbw0_ltdb",
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-03-13T04:51:28.000Z"
      }
    }
  },
  "process": {
    "working_directory": "/u01/app/oracle/product/12.2.0/dbhome_1/dbs",
    "executable": "/u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle",
    "args": [
      "ora_dbw0_ltdb"
    ],
    "name": "ora_dbw0_ltdb",
    "pid": 9253,
    "ppid": 1,
    "pgid": 9253
  },
  "host": {
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "name": "ltdb.kpp.com",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server"
    }
  }
}
2020-04-25T12:17:27.112+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "agent": {
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com"
  },
  "metricset": {
    "name": "process"
  },
  "system": {
    "process": {
      "memory": {
        "rss": {
          "bytes": 1457176576,
          "pct": 0.1799
        },
        "share": 1456869376,
        "size": 2879393792
      },
      "cmdline": "ora_mman_ltdb",
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-03-13T04:51:28.000Z"
      },
      "fd": {
        "open": 9,
        "limit": {
          "soft": 65536,
          "hard": 65536
        }
      },
      "state": "sleeping"
    }
  },
  "process": {
    "pid": 9229,
    "ppid": 1,
    "pgid": 9229,
    "working_directory": "/u01/app/oracle/product/12.2.0/dbhome_1/dbs",
    "executable": "/u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle",
    "args": [
      "ora_mman_ltdb"
    ],
    "name": "ora_mman_ltdb"
  },
  "user": {
    "name": "oracle"
  },
  "event": {
    "module": "system",
    "duration": 371196458,
    "dataset": "system.process"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "service": {
    "type": "system"
  },
  "host": {
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "name": "ltdb.kpp.com",
    "containerized": false,
    "hostname": "ltdb.kpp.com"
  }
}
2020-04-25T12:17:27.112+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 371524230
  },
  "metricset": {
    "name": "process"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "host": {
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    },
    "name": "ltdb.kpp.com"
  },
  "process": {
    "ppid": 1,
    "pgid": 9348,
    "working_directory": "/u01/app/oracle/product/12.2.0/dbhome_1/dbs",
    "executable": "/u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle",
    "args": [
      "ora_cjq0_ltdb"
    ],
    "name": "ora_cjq0_ltdb",
    "pid": 9348
  },
  "user": {
    "name": "oracle"
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "process": {
      "memory": {
        "share": 1202155520,
        "size": 2884866048,
        "rss": {
          "bytes": 1206439936,
          "pct": 0.1489
        }
      },
      "fd": {
        "open": 12,
        "limit": {
          "soft": 65536,
          "hard": 65536
        }
      },
      "cmdline": "ora_cjq0_ltdb",
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-03-13T04:51:34.000Z"
      },
      "state": "sleeping"
    }
  }
}
2020-04-25T12:17:27.113+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "metricset": {
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "agent": {
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1",
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64",
      "platform": "ol",
      "version": "7.8",
      "family": ""
    }
  },
  "user": {
    "name": "mysql"
  },
  "system": {
    "process": {
      "state": "sleeping",
      "cmdline": "/usr/sbin/mysqld",
      "memory": {
        "size": 2880688128,
        "rss": {
          "bytes": 904499200,
          "pct": 0.1117
        },
        "share": 12156928
      },
      "cpu": {
        "start_time": "2020-04-20T14:00:16.000Z",
        "total": {
          "pct": 0,
          "norm": {
            "pct": 0
          },
          "value": 0
        }
      },
      "fd": {
        "open": 392,
        "limit": {
          "soft": 16364,
          "hard": 16364
        }
      }
    }
  },
  "process": {
    "ppid": 1,
    "pgid": 32113,
    "working_directory": "/var/lib/mysql",
    "executable": "/usr/sbin/mysqld",
    "args": [
      "/usr/sbin/mysqld"
    ],
    "name": "mysqld",
    "pid": 32113
  },
  "ecs": {
    "version": "1.0.0"
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 371779609
  }
}
2020-04-25T12:17:27.113+0600    DEBUG   [processors]    processing/processors.go:183    Publish event: {
  "@timestamp": "2020-04-25T06:17:26.740Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "7.2.1"
  },
  "event": {
    "dataset": "system.process",
    "module": "system",
    "duration": 372032222
  },
  "process": {
    "name": "ora_smon_ltdb",
    "pid": 9267,
    "ppid": 1,
    "pgid": 9267,
    "working_directory": "/u01/app/oracle/product/12.2.0/dbhome_1/dbs",
    "executable": "/u01/app/oracle/product/12.2.0/dbhome_1/bin/oracle",
    "args": [
      "ora_smon_ltdb"
    ]
  },
  "user": {
    "name": "oracle"
  },
  "host": {
    "name": "ltdb.kpp.com",
    "id": "8e10311cf2a147a2ab615c47becc2358",
    "containerized": false,
    "hostname": "ltdb.kpp.com",
    "architecture": "x86_64",
    "os": {
      "platform": "ol",
      "version": "7.8",
      "family": "",
      "name": "Oracle Linux Server",
      "kernel": "4.1.12-124.33.4.el7uek.x86_64"
    }
  },
  "agent": {
    "type": "metricbeat",
    "ephemeral_id": "924c5954-0b12-4247-aa55-d82f04d83361",
    "hostname": "ltdb.kpp.com",
    "id": "8fb83fcc-7a27-4d16-abbe-8d2997882b38",
    "version": "7.2.1"
  },
  "metricset": {
    "name": "process"
  },
  "service": {
    "type": "system"
  },
  "system": {
    "process": {
      "cmdline": "ora_smon_ltdb",
      "cpu": {
        "total": {
          "value": 0,
          "pct": 0,
          "norm": {
            "pct": 0
          }
        },
        "start_time": "2020-03-13T04:51:28.000Z"
      },
      "fd": {
        "open": 13,
        "limit": {
          "soft": 65536,
          "hard": 65536
        }
      },
      "state": "sleeping",
      "memory": {
        "rss": {
          "pct": 0.0794,
          "bytes": 643481600
        },
        "share": 640905216,
        "size": 2882424832
      }
    }
  },
  "ecs": {
    "version": "1.0.0"
  }
}

Md_Saidur_Rahman · April 25, 2020, 11:14am

Debug log-6:

2020-04-25T12:17:27.743+0600    INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://10.88.250.111:9200))
2020-04-25T12:17:27.743+0600    DEBUG   [elasticsearch] elasticsearch/client.go:711     ES Ping(url=http://10.88.250.111:9200)
2020-04-25T12:17:27.748+0600    DEBUG   [elasticsearch] elasticsearch/client.go:734     Ping status code: 200
2020-04-25T12:17:27.748+0600    INFO    elasticsearch/client.go:735     Attempting to connect to Elasticsearch version 7.2.1
2020-04-25T12:17:27.748+0600    DEBUG   [elasticsearch] elasticsearch/client.go:753     GET http://10.88.250.111:9200/_xpack?human=false  <nil>
2020-04-25T12:17:27.809+0600    DEBUG   [elasticsearch] elasticsearch/client.go:753     GET http://10.88.250.111:9200/_xpack  <nil>
2020-04-25T12:17:27.870+0600    INFO    [index-management]      idxmgmt/std.go:252      Auto ILM enable success.
2020-04-25T12:17:27.870+0600    DEBUG   [elasticsearch] elasticsearch/client.go:753     GET http://10.88.250.111:9200/_ilm/policy/metricbeat-7.2.1  <nil>
2020-04-25T12:17:27.872+0600    INFO    [index-management.ilm]  ilm/std.go:134  do not generate ilm policy: exists=true, overwrite=false
2020-04-25T12:17:27.872+0600    INFO    [index-management]      idxmgmt/std.go:265      ILM policy successfully loaded.
2020-04-25T12:17:27.872+0600    INFO    [index-management]      idxmgmt/std.go:394      Set setup.template.name to '{metricbeat-7.2.1 {now/d}-000001}' as ILM is enabled.
2020-04-25T12:17:27.872+0600    INFO    [index-management]      idxmgmt/std.go:399      Set setup.template.pattern to 'metricbeat-7.2.1-*' as ILM is enabled.
2020-04-25T12:17:27.872+0600    INFO    [index-management]      idxmgmt/std.go:433      Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.2.1 {now/d}-000001} as ILM is enabled.
2020-04-25T12:17:27.872+0600    INFO    [index-management]      idxmgmt/std.go:437      Set settings.index.lifecycle.name in template to {metricbeat-7.2.1 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-04-25T12:17:27.872+0600    DEBUG   [elasticsearch] elasticsearch/client.go:753     HEAD http://10.88.250.111:9200/_template/metricbeat-7.2.1  <nil>
2020-04-25T12:17:27.885+0600    INFO    template/load.go:88     Template metricbeat-7.2.1 already exists and will not be overwritten.
2020-04-25T12:17:27.885+0600    INFO    [index-management]      idxmgmt/std.go:289      Loaded index template.
2020-04-25T12:17:27.885+0600    DEBUG   [elasticsearch] elasticsearch/client.go:753     HEAD http://10.88.250.111:9200/_alias/metricbeat-7.2.1  <nil>
2020-04-25T12:17:27.886+0600    INFO    [index-management]      idxmgmt/std.go:300      Write alias successfully generated.
2020-04-25T12:17:27.886+0600    DEBUG   [elasticsearch] elasticsearch/client.go:753     GET http://10.88.250.111:9200/  <nil>
2020-04-25T12:17:27.888+0600    INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://10.88.250.111:9200)) established
2020-04-25T12:17:27.892+0600    DEBUG   [elasticsearch] elasticsearch/client.go:339     PublishEvents: 25 events have been published to elasticsearch in 4.292253ms.
2020-04-25T12:17:27.893+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac27d396, ext:3130705768, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.cpu", "duration":1057023, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"cpu"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"cpu":common.MapStr{"cores":4, "idle":common.MapStr{"pct":0}, "iowait":common.MapStr{"pct":0}, "irq":common.MapStr{"pct":0}, "nice":common.MapStr{"pct":0}, "softirq":common.MapStr{"pct":0}, "steal":common.MapStr{"pct":0}, "system":common.MapStr{"pct":0}, "total":common.MapStr{"pct":0}, "user":common.MapStr{"pct":0}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.893+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac4f1c6f, ext:3133280247, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.filesystem", "duration":1300581, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"filesystem"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"filesystem":common.MapStr{"available":0x26bd9000, "device_name":"/dev/sda1", "files":0x100000, "free":0x26bd9000, "free_files":0xffe8d, "mount_point":"/boot", "total":0x3f600000, "type":"xfs", "used":common.MapStr{"bytes":0x18a27000, "pct":0.3887}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.894+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac4f1c6f, ext:3133280247, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.filesystem", "duration":1334334, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"filesystem"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"filesystem":common.MapStr{"available":0x95de25000, "device_name":"/dev/mapper/ol-home", "files":0x291e000, "free":0x95de25000, "free_files":0x291d9fa, "mount_point":"/home", "total":0xa46371000, "type":"xfs", "used":common.MapStr{"bytes":0xe854c000, "pct":0.0883}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.895+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac65c5e6, ext:3134765472, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.fsstat", "duration":1005971, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"fsstat"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"fsstat":common.MapStr{"count":3, "total_files":0x8e1d000, "total_size":common.MapStr{"free":0x123c5b1000, "total":0x2383c71000, "used":0x11476c0000}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.895+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac83a6a2, ext:3136723508, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.uptime", "duration":13252, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"uptime"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"uptime":common.MapStr{"duration":common.MapStr{"ms":3751650000}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.896+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac274640, ext:3130669482, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.memory", "duration":2899813, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"memory"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"memory":common.MapStr{"actual":common.MapStr{"free":0xd0f1d000, "used":common.MapStr{"bytes":0x111dcf000, "pct":0.5672}}, "free":0x6f31000, "hugepages":common.MapStr{"default_size":0x200000, "free":0x0, "reserved":0x0, "surplus":0x0, "total":0x0, "used":common.MapStr{"bytes":0x0, "pct":0}}, "swap":common.MapStr{"free":0x1e03a3000, "total":0x1f7fff000, "used":common.MapStr{"bytes":0x17c5c000, "pct":0.0472}}, "total":0x1e2cec000, "used":common.MapStr{"bytes":0x1dbdbb000, "pct":0.9856}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.896+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac4f1c6f, ext:3133280247, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.filesystem", "duration":3893315, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}},

Md_Saidur_Rahman · April 25, 2020, 11:15am

"metricset":common.MapStr{"name":"filesystem"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"filesystem":common.MapStr{"available":0x8b7bb3000, "device_name":"/dev/mapper/ol-root", "files":0x63ff000, "free":0x8b7bb3000, "free_files":0x63ad416, "mount_point":"/", "total":0x18fe300000, "type":"xfs", "used":common.MapStr{"bytes":0x104674d000, "pct":0.6512}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.896+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac9cba4d, ext:3138367023, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), 
Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.network", "duration":400543, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"network"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"network":common.MapStr{"in":common.MapStr{"bytes":0x0, "dropped":0x0, "errors":0x0, "packets":0x0}, "name":"virbr0-nic", "out":common.MapStr{"bytes":0x0, "dropped":0x0, "errors":0x0, "packets":0x0}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.898+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac9cba4d, ext:3138367023, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.network", "duration":424631, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"network"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"network":common.MapStr{"in":common.MapStr{"bytes":0x0, "dropped":0x0, "errors":0x0, "packets":0x0}, "name":"virbr0", "out":common.MapStr{"bytes":0x0, "dropped":0x0, "errors":0x0, "packets":0x0}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.898+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac9cba4d, ext:3138367023, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.network", "duration":438808, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"network"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"network":common.MapStr{"in":common.MapStr{"bytes":0x453798879, "dropped":0x267, "errors":0x0, "packets":0x6364b94}, "name":"ens160", "out":common.MapStr{"bytes":0x71ea79f72, "dropped":0x0, "errors":0x0, "packets":0x4d1d5ea}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.898+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac9cba4d, ext:3138367023, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.network", "duration":1433876, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"network"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"network":common.MapStr{"in":common.MapStr{"bytes":0x112e0156, "dropped":0x0, "errors":0x0, "packets":0x1b214b}, "name":"lo", "out":common.MapStr{"bytes":0x112e0156, "dropped":0x0, "errors":0x0, "packets":0x1b214b}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.898+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dacba2def, ext:3140297116, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"system.load", "duration":154816, "module":"system"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"load"}, "service":common.MapStr{"type":"system"}, "system":common.MapStr{"load":common.MapStr{"1":0.16, "15":0.17, "5":0.22, "cores":4, "norm":common.MapStr{"1":0.04, "15":0.0425, "5":0.055}}}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=403): {"type":"cluster_block_exception","reason":"index [metricbeat-7.2.1-2020.04.16-000001] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}
2020-04-25T12:17:27.898+0600    WARN    elasticsearch/client.go:527     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfa1135dac118b62, ext:3129245425, loc:(*time.Location)(0x51442a0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"924c5954-0b12-4247-aa55-d82f04d83361", "hostname":"ltdb.kpp.com", "id":"8fb83fcc-7a27-4d16-abbe-8d2997882b38", "type":"metricbeat", "version":"7.2.1"}, "ecs":common.MapStr{"version":"1.0.0"}, "event":common.MapStr{"dataset":"mysql.status", "duration":13895842, "module":"mysql"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ltdb.kpp.com", "id":"8e10311cf2a147a2ab615c47becc2358", "name":"ltdb.kpp.com", "os":common.MapStr{"family":"", "kernel":"4.1.12-124.33.4.el7uek.x86_64", "name":"Oracle Linux Server", "platform":"ol", "version":"7.8"}}, "metricset":common.MapStr{"name":"status"},