Filebeat 6.2.3 index is empty

Elastic Stack Beats
1

Since upgrading to 6.2.3 the filebeat index is empty.

Any idea where to start looking?

Filebeat debug output:

2018-03-21T16:11:28.994-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {
"@timestamp": "2018-03-21T20:11:28.994Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.3"
},
"source": "/usr/local/apache/logs/error_log",
"offset": 222572523,
"message": "2018-03-21 16:11:27.897285 [INFO] [APVH_ocpsa_Suphp56:] PID: 14576, add child process pid: 16067, procinfo: 0x8972610",
"prospector": {
"type": "log"
},
"beat": {
"name": "servername-filebeat",
"hostname": "servername",
"version": "6.2.3"

Discover output:
{
"_index": "filebeat-6.2.3-2018.03.21",
"_type": "doc",
"_id": "GVckSmIB41zG7qDLXuZ8",
"_version": 1,
"_score": null,
"fields": {
"@timestamp": [
"2018-03-21T19:58:39.000Z"
]
},
"sort": [
1521662319000
]
}

2

Check your Filebeat and Elasticsearch logs for errors.

Please share your Filebeat config.

3

No errors in filebeat log.

One recurring error in the elasticsearch log. Is this the proper way to enable logging?

PUT /_cluster/settings
{
  "transient": {
    "logger._root": "debug"
  }
}

Elasticsearch error: [DEBUG][o.e.g.MetaStateService ] [es-01] [OiByffRoRqWXA1pJ7tdPiA] failed to find metadata for existing index location

filebeat.yml

filebeat.prospectors:
- type: log

  paths:
    - /var/log/*.log
    - /usr/local/cpanel/logs/*.log
    - /usr/local/apache/logs/*.log
    - /usr/local/apache/logs/*_log
    - /usr/local/lsws/logs/*.log
    - /usr/local/cpanel/logs/*_log
    - /var/lib/mysql/rssd5382.webaccountserver.err*

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  index.codec: best_compression
  _source.enabled: false

#================================ Outputs =====================================

#-------------------------- Elasticsearch output -------------------------------

output.elasticsearch:
  enabled: true
  hosts: ["https://XXXXXXXXXXXXXXXXX:443/"]
  protocol: "https"
  username: "XXXXXXXXXX"
  password: "XXXXXXXXXXXXXX"
  worker: 4
  timeout: 90
  ssl.enabled: true
  ssl.verification_mode: none
  ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
  ssl.certificate_authorities: ["XXXXXXXXXXXXXXXXXX"]
  ssl.certificate: "XXXXXXXXXXXXXXXXXX"
  ssl.key: "XXXXXXXXXXXXXXXXXX"
4

Your filebeat config looks pretty standard. No issues there.

What does GET _cat/indices/filebeat-6.2.3* return?

And in your Filebeat logs can you post some of the lines containing "monitoring"? These contain metrics that get logged every 30s and will show if events are being written to ES.

That looks right to me.

5

yellow open filebeat-6.2.3-2018.03.22 UqJCmvBARQmTZYN4L_RfgA 3 1 1756515 0 164.4mb 164.4mb
yellow open filebeat-6.2.3-2018.03.21 o1oQLEH6QYaVXTi_TUgm_g 3 1 1785767 0 167.8mb 167.8mb
yellow open filebeat-6.2.3-2018.03.20 OU5a1GkPRz2cYxixrf9qlw 3 1 59354 0 5.1mb 5.1mb

filebeat log output:
2018-03-22T09:57:25.112-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":129400,"time":129400},"total":{"ticks":682240,"time":682243,"value":682240},"user":{"ticks":552840,"time":552843}},"info":{"ephemeral_id":"275a6546-e45a-4f3a-b5c4-56c62efc88f7","uptime":{"ms":63990014}},"memstats":{"gc_next":8065776,"memory_alloc":6928800,"memory_total":47135657920}},"filebeat":{"events":{"active":7,"added":148,"done":141},"harvester":{"open_files":10,"running":10,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":132,"batches":28,"total":132},"read":{"bytes":22393},"write":{"bytes":62758}},"pipeline":{"clients":1,"events":{"active":10,"filtered":9,"published":139,"total":148},"queue":{"acked":132}}},"registrar":{"states":{"current":813,"update":141},"writes":28},"system":{"load":{"1":4.58,"15":4.63,"5":4.12,"norm":{"1":0.1908,"15":0.1929,"5":0.1717}}}}}}
2018-03-22T09:57:55.113-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":129470,"time":129476},"total":{"ticks":682620,"time":682627,"value":682620},"user":{"ticks":553150,"time":553151}},"info":{"ephemeral_id":"275a6546-e45a-4f3a-b5c4-56c62efc88f7","uptime":{"ms":64020015}},"memstats":{"gc_next":7479104,"memory_alloc":5016448,"memory_total":47160140872}},"filebeat":{"events":{"active":-6,"added":97,"done":103},"harvester":{"open_files":10,"running":10}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":103,"batches":28,"total":103},"read":{"bytes":22204},"write":{"bytes":52646}},"pipeline":{"clients":1,"events":{"active":4,"published":97,"total":97},"queue":{"acked":103}}},"registrar":{"states":{"current":813,"update":103},"writes":28},"system":{"load":{"1":4.77,"15":4.64,"5":4.22,"norm":{"1":0.1987,"15":0.1933,"5":0.1758}}}}}}

6

You have data in your indices (docs.count is 1756515, 1785767, and 59354 in the indicies). And Filebeat indicates that it's sending successfully.

        "output": {
          "events": {
            "acked": 132,
            "batches": 28,
            "total": 132
          },

Do you see events when you query GET filebeat-6.2.3-2018.03.22/_search?

7

It appears the message field and others are missing. If I do the same query on a 6.2.2 index they are populated.

Output:

  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "J5bdS2IB41zG7qDLINXB",
    "_score": 1
  },
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "KJbdS2IB41zG7qDLINXB",
    "_score": 1
  },
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "N5bdS2IB41zG7qDLINXi",
    "_score": 1
  },
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "OZbdS2IB41zG7qDLINXi",
    "_score": 1
8

You have disabled _source. I think this is a consequence. See _source field | Elasticsearch Guide [8.11] | Elastic

9

Interesting. We have about 4 servers running filebeat and on 6.2.2 they were all working. 6.2.3 they all stopped. I have to confirm with one other person - but we would have had no reason to modify this.

In looking at our old config this was disabled before as well. Is it enabled out of the box?

10

_source is enabled by default. You could check your index template and mappings used by 6.2.2 to see if _source was enabled. Those are stored in Elasticsearch.

11

I've reinstalled filebeat and still the same issue. I am guessing the problem is on the elastic-side since the filebeat log output shows all the correct fields with message.

Is there a way to tell if elastic is stripping out the fields? Is there a processor or something to look at in elastic?

12

So you re-installed. Did you set _source.enabled: true or remove that option so that the default is used?

If you do make a change to that value then you need to overwrite the installed index template. (docs)

filebeat setup -e --template -E setup.template.overwrite=true

And even after updating the template the change won't take effect until the next daily index is created.

13

I left _source.enabled to #_source.enabled: false so it is commented out as in the default yml.

BAM! That did it.

So the moral of the story when updating filebeat is to run filebeat setup -e --template -E setup.template.overwrite=true after the update?

14

Well, filebeat installs versioned templates (e.g. _template/filebeat-6.2.3) so it's not necessary to forcibly update the template on a version update because it will be installing a whole new template. It's only necessary to force an update if you modify config options affecting the template.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.