Filebeat and packetbeat not sending _source data

Elastic Stack Beats
1

Hi,

Before 3 days, all beats are sending data properly on my elastic instance.
I was noticed new version is available. So, I upgraded all beats including elastic and kiban from 6.2.3 to 6.2.4. (i am using all beats accept winlog.)
After up-gradation, everything working as it is, accept filebeat and packetbeat not sending _source data to elastic search. On kibana's discover tab, only date is showing and next _source is empty. But i can see some hits are coming by upper side graph's visualization.

I checked everything(from last 2days), but don't know what i missed...

Please suggest me, regarding it.

2

@shivang_patel I've taken a look a the changelog between 6.2.3 and 6.2.4 and I don't see any changes related to the mapping, the _source field is managed on the Elasticsearch side, I've tested with FB 6.2.4 with the default template mapping and the _source is included.

{
  "_index": "filebeat-6.2.4-2018.06.11",
  "_type": "doc",
  "_id": "yg9p72MBFf8FPI8lp7i7",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-06-11T15:14:27.922Z",
    "message": "Jun 11 11:14:22 sashimi login[3765]: USER_PROCESS: 3765 ttys005",
    "prospector": {
      "type": "log"
    },
    "beat": {
      "hostname": "sashimi",
      "version": "6.2.4",
      "name": "sashimi"
    },
    "source": "/var/log/system.log",
    "offset": 120742
  },
  "fields": {
    "@timestamp": [
      "2018-06-11T15:14:27.922Z"
    ]
  },
  "sort": [
    1528730067922
  ]
}

Did you change the default mapping?
Can you create a gist with the output of the following call?

http://localhost:9200/filebeat*/_mapping/doc

3

I know we added the ability to disable _source in the generated index templates. But the configuration to disable source is commented out in the config files we distribute with the Beats. If you're letting the Beat generate the index template, then did you uncomment this option?

4

Ya. I already tried this option. but I don't know, why it's not working for me !

5

No. I did not change any default mapping.

I will post "http://localhost:9200/filebeat*/_mapping/doc" output tomorrow at morning.

6

as per your request : https://gist.github.com/shivangpatel/5ce46779daf95678388c0aa56714aac1

7

@pierhugues : as per your suggestion, I observe that output and I can see

        "_source": {
          "enabled": false
        },

is still false ! So, Can you tell me how to make it true ? I already changed in filebeat.yml file.

8

@shivang_patel I think the easiest way to get running is to delete the template mapping and let Filebeat create it again, make sure _source.enabled: false is commented before starting it.

To delete the template:

curl -X DELETE "localhost:9200/_template/filebeat-6.2.4"

The newly indexed data will have the _source.

9

Sorry, but it's not working !
Actually,

GET /_template/filebeat results showing

  "_source": {
              "enabled": true
            }

But,https://myurl.in/elastic/filebeat*/_mapping/doc still showing false !

10

Sorry, let me clarify what I've said previously.

The way template works are to dynamically assign a specific mapping to a matching index_pattern, in your case "filebeat-6.2.4-*". So when Filebeat creates a new indice that match the pattern the mapping will be automatically applied. If you look at the mapping for the created index after your change, the _source.enabled should be set to true.

Previously created indices mapping won't change, the only way to change it is to reindex your data.

But in your case you want to have access to the _source, this field contains the original JSON data, when _source.enabled is set to false we don't persist it, It is gone. The only way to have it is to reindex your data from the original file.

1 Like
11

Thanks it's working :+1:

And this last information is very informative.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.