# Filebeat Hints Using Incorrect Fileset/Dataset

**URL:** <https://discuss.elastic.co/t/filebeat-hints-using-incorrect-fileset-dataset/187909>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 27, 2019, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-hints-using-incorrect-fileset-dataset/187909 "2019-06-27T19:25:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\
**Post date:** [June 27, 2019, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-hints-using-incorrect-fileset-dataset/187909/1 "2019-06-27T19:25:30Z")

</div>

Hi there,

I'm currently testing out upgrading Filebeat from 6.x to 7.2. I've taken our existing 6.x config and changed a couple of fields:

`filebeat.registry_flush` \> `filebeat.registry.flush`

`filebeat.config.prospectors` \> `filebeat.config.input`

```auto
      when.equals:
        fileset.module: "ingress-nginx"

```

to

```auto
      when.equals:
        event.module: "ingress-nginx"

```

The rest of the configuration is the same as it was previously.

We have a custom module, `ingress-nginx` with two configured filesets; `access` & `error`. We use Kubernetes autodiscover + hints annotations to tell Filebeat to use this module for a certain set of pods:

```auto
    co.elastic.logs/fileset.stderr: error
    co.elastic.logs/fileset.stdout: access
    co.elastic.logs/module: ingress-nginx

```

This has been working fine on 6.x however after changing to 7.2 it seems as though Filebeat is somehow using the incorrect the filesets, using the `error` fileset for stdout stream.

Taking a look at the below debug log from filebeat you can see the annotations are set to use the `ingress-nginx` module and use fileset `access` for `stdout` stream and `error` for `stderr` stream.

```auto
2019-06-27T19:16:57.696Z	DEBUG	[processors]	processing/processors.go:183	Publish event: {
  "@timestamp": "2019-06-27T19:16:57.677Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.2.0",
    "pipeline": "filebeat-7.2.0-ingress-nginx-error-pipeline"
  },
  "agent": {
    "version": "7.2.0",
    "type": "filebeat",
    "ephemeral_id": "3c57afc9-923e-41cd-8bb2-87bae6772477",
    "hostname": "filebeat-testing-rqv7k",
    "id": "5c021faf-e67b-4ad2-83e2-63f11650fca8"
  },
  "stream": "stdout",
  "message": "<REDACTED>",
  "fileset": {
    "name": "error"
  },
  "input": {
    "type": "container"
  },
  "event": {
    "module": "ingress-nginx",
    "dataset": "ingress-nginx.error"
  },
  "kubernetes": {
    "namespace": "ingress-nginx",
    "replicaset": {
      "name": "ingress-nginx-internal-controller-5d567b6b5d"
    },
    "labels": {
      "RESTARTED_ON": "1561489759",
      "app": "ingress-nginx-internal",
      "pod-template-hash": "5d567b6b5d",
      "purpose": "ingress-nginx"
    },
    "annotations": {
      "co_elastic_logs/fileset_stdout": "access",
      "co_elastic_logs/module": "ingress-nginx",
      "co_elastic_logs/fileset_stderr": "error"
    },
    "pod": {
      "name": "ingress-nginx-internal-controller-5d567b6b5d-h9dbc",
      "uid": "be036817-990d-11e9-a3ba-42010aa400aa"
    },
    "node": {
      "name": "gke-delivery-platform-core-20190401-6dbff651-xk0b"
    },
    "container": {
      "name": "master"
    }
  },
  "host": {
    "name": "filebeat-testing-rqv7k"
  },
  "log": {
    "offset": 1244891,
    "file": {
      "path": "/var/lib/docker/containers/f9cdb5502f7405473b4ca152f4cdf1698cb17472955171568efd923c19d1ef25/f9cdb5502f7405473b4ca152f4cdf1698cb17472955171568efd923c19d1ef25-json.log"
    }
  },
  "service": {
    "type": "ingress-nginx"
  },
  "ecs": {
    "version": "1.0.0"
  },
  "cloud": {
    "provider": "gcp",
    "instance": {
      "id": "3801039341841480502",
      "name": "gke-delivery-platform-core-20190401-6dbff651-xk0b"
    },
    "machine": {
      "type": "custom-12-16384"
    },
    "availability_zone": "europe-west4-b",
    "project": {
      "id": "<REDACTED>"
    }
  }
}

```

However for some reason `fileset.name` and `event.dataset` are both showing `error`, and the pipeline is ultimately going to the error ingest (`filebeat-7.2.0-ingress-nginx-error-pipeline`).

Our Filebeat config is as below:

```auto
http:
  enabled: true
  host: localhost
  port: 5066

logging.level: debug

filebeat.registry.flush: 2s

# Overwrite pipelines with new config
filebeat.overwrite_pipelines: true

filebeat.config:
  input:
    # Mounted `filebeat-prospectors` configmap:
    path: ${path.config}/prospectors.d/*.yml
    # Reload prospectors configs as they change:
    reload.enabled: false
  modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      include_annotations: ['at/logging','co.elastic.logs/module','co.elastic.logs/fileset.stderr','co.elastic.logs/fileset.stdout']

processors:
- add_cloud_metadata: ~
- drop_event:
    when:
      not:
        equals:
          kubernetes.annotations.at/logging: 'true'
- drop_event:
    when:
      equals:
        kubernetes.container.name: 'istio-proxy'
- drop_event:
    when:
      equals:
        kubernetes.container.name: 'istio-init'
- decode_json_fields:
    fields: ["message"]
    process_array: false
    max_depth: 1
    target: ""
    overwrite_keys: true

output.elasticsearch:
  hosts: ["<REDACTED>"]
  protocol: 'https'
  username: '<REDACTED>'
  password: "${ELASTICSEARCH_PASSWORD}"
  index: "kubernetes-%{+yyyy.MM.dd}"
  bulk_max_size: 250
  indices:
    - index: "ingress-nginx-%{+yyyy.MM.dd}"
      when.equals:
        event.module: "ingress-nginx"

setup.template.enabled: false
setup.template.name: 'custom-beats'
setup.template.pattern: "beats-*"
setup.template.overwrite: true
setup.template.settings:
  index:
    number_of_shards: 2
    number_of_replicas: 1
    codec: best_compression

xpack.monitoring.enabled: false

```

Any pointers would be appreciated. I've looked through the 6 -\> 7 upgrade guide and the breaking changes docs but I've not found anything that stands out

This is trying to push to an ES 6 cluster that I know isn't supported (Just testing atm), however, I believe Elasticsearch isn't involved yet at the point the event is being constructed incorrectly.

You can see in the below they almost seem flipped:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7c35170a5af12d0e1ccf36284caffb458328881.png)

cc/ @exekias -- Sorry to @ you, I just wanted to check if [https://github.com/elastic/beats/pull/12348](https://github.com/elastic/beats/pull/12348) / [https://github.com/elastic/beats/pull/12162](https://github.com/elastic/beats/pull/12162) / [https://github.com/elastic/beats/issues/12346](https://github.com/elastic/beats/issues/12346) could be at all related to this?

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2019, 7:25pm UTC](https://discuss.elastic.co/t/filebeat-hints-using-incorrect-fileset-dataset/187909/2 "2019-07-25T19:25:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.