Metricbeat Error on field not found with Apache

Hi guys
I recently installed metricbeat and while checking the logs I found the error below, is this a bug?
As an additional info, I found also that only happens when I enable the apache module, for example on another server which is not configured I do not see the error.

Thanks,
Regards

Metric Beat Version:
metricbeat-5.0.0-1.x86_64






2016-11-01T09:56:29-05:00 ERR Error on field 'total_accesses': Key `Total Accesses` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'server_uptime': Key `ServerUptimeSeconds` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'uptime': Key `Uptime` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'user': Key `CPUUser` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'system': Key `CPUSystem` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'children_user': Key `CPUChildrenUser` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'children_system': Key `CPUChildrenSystem` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'total': Key `ConnsTotal` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'writing': Key `ConnsAsyncWriting` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'keep_alive': Key `ConnsAsyncKeepAlive` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'closing': Key `ConnsAsyncClosing` not found

2016-11-01T09:56:29-05:00 ERR Error on field '1': Key `Load1` not found

2016-11-01T09:56:29-05:00 ERR Error on field '5': Key `Load5` not found

2016-11-01T09:56:29-05:00 ERR Error on field '15': Key `Load15` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'total_kbytes': Key `Total kBytes` not found

I also notice some strange indices names, is this correct, phppath. login.php, what's that?
Should I remove topbeat/packetbeat indexes before installing metricbeat template?

    yellow open   servlet                         5   1          0            0       795b           795b
    yellow open   login.php                       5   1          0            0       795b           795b
    yellow open   messagebroker                   5   1          0            0       795b           795b
    yellow open   phppath                         5   1          0            0       795b           795b
    yellow open   lcds-samples                    5   1          0            0       795b           795b
    yellow open   index.php                       5   1          0            0       795b           795b
    yellow open   cgi-bin                         5   1          0            0       795b           795b
    yellow open   formmail.pl                     5   1          0            0       795b           795b
    yellow open   getpassword.php                 5   1          0            0       795b           795b
    yellow open   metricbeat-2016.10.31           5   1    2259523            0    763.2mb        763.2mb
    yellow open   shopsearch.asp                  5   1          0            0       795b           795b
    yellow open   spipe                           5   1          0            0       795b           795b
    yellow open   blazeds                         5   1          0            0       795b           795b
    yellow open   .packetbeat-topology            5   1          0            0       797b           797b
    yellow open   topbeat-2016.10.21              5   1     351011            0     91.4mb         91.4mb
    yellow open   samples                         5   1          0            0       795b           795b
    yellow open   packetbeat-2016.10.26           5   1      20139            0      2.4mb          2.4mb
    yellow open   packetbeat-2016.10.23           5   1       2144            0    528.3kb        528.3kb
    yellow open   metricbeat-2016.11.01           5   1    3501524            0      1.1gb          1.1gb
    yellow open   sawmillcl.exe                   5   1          0            0       795b           795b
    yellow open   packetbeat-2016.10.24           5   1       8090            0      1.1mb          1.1mb
    yellow open   formmail                        5   1          0            0       795b           795b
    yellow open   packetbeat-2016.10.21           5   1        987            0    455.6kb        455.6kb
    yellow open   filebeat-2016.10.23             5   1     343140            0    125.8mb        125.8mb
    yellow open   filebeat-2016.10.26             5   1     283399            0      108mb          108mb
    yellow open   admin                           5   1          0            0       795b           795b
    yellow open   bower_components                5   1          0            0       795b           795b
    yellow open   flex2gateway                    5   1          0            0       795b           795b
    yellow open   wsavae1.html                    5   1          0            0       795b           795b
    yellow open   webui                           5   1          0            0       795b           795b
    yellow open   .kibana                         1   1        163            2      189kb          189kb
    yellow open   perl                            5   1          0            0       795b           795b
    yellow open   samba                           5   1          0            0       795b           795b
    yellow open   scripts                         5   1          0            0       795b           795b
    yellow open   sawmill6cl.exe                  5   1          0            0       795b           795b
    yellow open   comersus_backoffice_login.php   5   1          0            0       795b           795b
    yellow open   gw                              5   1          0            0       795b           795b
    yellow open   lcds                            5   1          0            0       795b           795b
    yellow open   smbshr.pl                       5   1          0            0       795b           795b

    yellow open   kb.cgi                          5   1          0            0       795b           795b

What version of Apache HTTPD are you running? Those errors occur when the specified field is not present on the mod_status webpage (/server-status?auto). A value of 0 is reported for the field.

What does your configuration look like? Topbeat and Packetbeat indices should not conflict in any way with Metricbeat.

Hi Andrew

Well, it seems the apache version is not supported, I was sure it was 2.4 but mine is httpd-2.2.15-47.el6_7.3.x86_64 , I can see other information on the dashboard anyway, but of course some info is missing.

I notice those rare indexes when I loaded the metricbeat template or as soon I started the service, the topbeat dashboard changed also, it seems as if something was overwritten or changed cuz I see numbers where there was a graphic before..

Top Beat System Load I see 0.011 Average system.load.norm.1 and Could not locate that index-pattern-field (id: cpu.user_p) on another dashboards, memory usage changed to numbers as well instead of a graphic ( 6.625GB Average system.memory.used.bytes )

Thank for your time and support
Regards

What's in your configuration file for Metricbeat? Are you using Logstash? If so, share that too.

Hi Andrew..

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - load
    - diskio
    - filesystem
    - fsstat
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']

- module: apache
  metricsets: ["status"]
  enabled: true
  period: 60s

  hosts: ["http://hostname"]

- module: redis
  metricsets: ["info", "keyspace"]
  enabled: true
  period: 60s

  hosts: ["rediserver:6379"]
  timeout: 5s

name: server-name

output.logstash:
  enabled: true
  hosts: ["logstashserver:5044"]

  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat.log
  keepfiles: 7

Logstash configuration, this is working with metricbeat, topbeat, filebeat, but not packetbeat, for packetbeat I configure it to send the data directly to elasticsearch, anyway, I wont use packetbeat anymore in favor or metricbeat.

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}


output {
  elasticsearch {
    hosts => ["server-name"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}


Thank you for your time and support
Regards

Those configs look good. I don't see any issues that would cause those indices.

If all data goes through Logstash, then you could add a conditional to route events to a file where [@metadata][index] doesn't match topbeat/packetbeat/metricbeat. Then you could examine those events to figure out where those indices are coming from.

Ok, thank you!
Regards

This topic was automatically closed after 21 days. New replies are no longer allowed.