Metricbeat Error on field not found with Apache

Elastic Stack Beats
1

Hi guys
I recently installed metricbeat and while checking the logs I found the error below, is this a bug?
As an additional info, I found also that only happens when I enable the apache module, for example on another server which is not configured I do not see the error.

Thanks,
Regards

Metric Beat Version:
metricbeat-5.0.0-1.x86_64






2016-11-01T09:56:29-05:00 ERR Error on field 'total_accesses': Key `Total Accesses` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'server_uptime': Key `ServerUptimeSeconds` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'uptime': Key `Uptime` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'user': Key `CPUUser` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'system': Key `CPUSystem` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'children_user': Key `CPUChildrenUser` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'children_system': Key `CPUChildrenSystem` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'total': Key `ConnsTotal` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'writing': Key `ConnsAsyncWriting` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'keep_alive': Key `ConnsAsyncKeepAlive` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'closing': Key `ConnsAsyncClosing` not found

2016-11-01T09:56:29-05:00 ERR Error on field '1': Key `Load1` not found

2016-11-01T09:56:29-05:00 ERR Error on field '5': Key `Load5` not found

2016-11-01T09:56:29-05:00 ERR Error on field '15': Key `Load15` not found

2016-11-01T09:56:29-05:00 ERR Error on field 'total_kbytes': Key `Total kBytes` not found
2

I also notice some strange indices names, is this correct, phppath. login.php, what's that?
Should I remove topbeat/packetbeat indexes before installing metricbeat template?

    yellow open   servlet                         5   1          0            0       795b           795b
    yellow open   login.php                       5   1          0            0       795b           795b
    yellow open   messagebroker                   5   1          0            0       795b           795b
    yellow open   phppath                         5   1          0            0       795b           795b
    yellow open   lcds-samples                    5   1          0            0       795b           795b
    yellow open   index.php                       5   1          0            0       795b           795b
    yellow open   cgi-bin                         5   1          0            0       795b           795b
    yellow open   formmail.pl                     5   1          0            0       795b           795b
    yellow open   getpassword.php                 5   1          0            0       795b           795b
    yellow open   metricbeat-2016.10.31           5   1    2259523            0    763.2mb        763.2mb
    yellow open   shopsearch.asp                  5   1          0            0       795b           795b
    yellow open   spipe                           5   1          0            0       795b           795b
    yellow open   blazeds                         5   1          0            0       795b           795b
    yellow open   .packetbeat-topology            5   1          0            0       797b           797b
    yellow open   topbeat-2016.10.21              5   1     351011            0     91.4mb         91.4mb
    yellow open   samples                         5   1          0            0       795b           795b
    yellow open   packetbeat-2016.10.26           5   1      20139            0      2.4mb          2.4mb
    yellow open   packetbeat-2016.10.23           5   1       2144            0    528.3kb        528.3kb
    yellow open   metricbeat-2016.11.01           5   1    3501524            0      1.1gb          1.1gb
    yellow open   sawmillcl.exe                   5   1          0            0       795b           795b
    yellow open   packetbeat-2016.10.24           5   1       8090            0      1.1mb          1.1mb
    yellow open   formmail                        5   1          0            0       795b           795b
    yellow open   packetbeat-2016.10.21           5   1        987            0    455.6kb        455.6kb
    yellow open   filebeat-2016.10.23             5   1     343140            0    125.8mb        125.8mb
    yellow open   filebeat-2016.10.26             5   1     283399            0      108mb          108mb
    yellow open   admin                           5   1          0            0       795b           795b
    yellow open   bower_components                5   1          0            0       795b           795b
    yellow open   flex2gateway                    5   1          0            0       795b           795b
    yellow open   wsavae1.html                    5   1          0            0       795b           795b
    yellow open   webui                           5   1          0            0       795b           795b
    yellow open   .kibana                         1   1        163            2      189kb          189kb
    yellow open   perl                            5   1          0            0       795b           795b
    yellow open   samba                           5   1          0            0       795b           795b
    yellow open   scripts                         5   1          0            0       795b           795b
    yellow open   sawmill6cl.exe                  5   1          0            0       795b           795b
    yellow open   comersus_backoffice_login.php   5   1          0            0       795b           795b
    yellow open   gw                              5   1          0            0       795b           795b
    yellow open   lcds                            5   1          0            0       795b           795b
    yellow open   smbshr.pl                       5   1          0            0       795b           795b

    yellow open   kb.cgi                          5   1          0            0       795b           795b
3

What version of Apache HTTPD are you running? Those errors occur when the specified field is not present on the mod_status webpage (/server-status?auto). A value of 0 is reported for the field.

What does your configuration look like? Topbeat and Packetbeat indices should not conflict in any way with Metricbeat.

4

Hi Andrew

Well, it seems the apache version is not supported, I was sure it was 2.4 but mine is httpd-2.2.15-47.el6_7.3.x86_64 , I can see other information on the dashboard anyway, but of course some info is missing.

I notice those rare indexes when I loaded the metricbeat template or as soon I started the service, the topbeat dashboard changed also, it seems as if something was overwritten or changed cuz I see numbers where there was a graphic before..

Top Beat System Load I see 0.011 Average system.load.norm.1 and Could not locate that index-pattern-field (id: cpu.user_p) on another dashboards, memory usage changed to numbers as well instead of a graphic ( 6.625GB Average system.memory.used.bytes )

Thank for your time and support
Regards

5

What's in your configuration file for Metricbeat? Are you using Logstash? If so, share that too.

6

Hi Andrew..

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - load
    - diskio
    - filesystem
    - fsstat
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']

- module: apache
  metricsets: ["status"]
  enabled: true
  period: 60s

  hosts: ["http://hostname"]

- module: redis
  metricsets: ["info", "keyspace"]
  enabled: true
  period: 60s

  hosts: ["rediserver:6379"]
  timeout: 5s

name: server-name

output.logstash:
  enabled: true
  hosts: ["logstashserver:5044"]

  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat.log
  keepfiles: 7

Logstash configuration, this is working with metricbeat, topbeat, filebeat, but not packetbeat, for packetbeat I configure it to send the data directly to elasticsearch, anyway, I wont use packetbeat anymore in favor or metricbeat.

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}


filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}


output {
  elasticsearch {
    hosts => ["server-name"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}


Thank you for your time and support
Regards
7

Those configs look good. I don't see any issues that would cause those indices.

If all data goes through Logstash, then you could add a conditional to route events to a file where [@metadata][index] doesn't match topbeat/packetbeat/metricbeat. Then you could examine those events to figure out where those indices are coming from.

8

Ok, thank you!
Regards

9

This topic was automatically closed after 21 days. New replies are no longer allowed.