Force Filebeat to re process a file

[Jackson]

Hello,

I added a new file on the filebeat.yml. It went into elastic but some of the fields were not parsed correctly. This file is only updated once per day at 9AM.

I modified the filebeat.yml to properly parse the file but now filebeat refuses to send the file to elastic a second time.

I started by removing the bad data from my index.

POST filebeat-*/_delete_by_query
{
  "query": {
    "match": {
      "log.file.path":"/var/log/qrsender/qrsender.log"
    }
  }
}

I stopped the filebeat service and I edited the .json file in the registry by fully removing the only line that referenced "/var/log/qrsender/qrsender.log"

After restarting the service, I got nothing in elastic from this file.

Next, I modified "/var/log/qrsender/qrsender.log" by adding a line, saving, and then deleting the line so that the OS shows it was updated.

This didn't work.

I checked the .json file in the registry and there was a new line pointing to "/var/log/qrsender/qrsender.log" with the offset of 20K+.

I changed the offset to 0 and saved the file.

This didn't work.

I ran filebeat setup -e

This didn't work

filebeat test config returns Config OK

I checked the .json file and it again has the offset for "/var/log/qrsender/qrsender.log" at 22K+

I've been working on this for hours and it's quite frustrating. How can I get this file into elastic? I don't want to delete the entire registry as I have millions of documents.

Thanks in advance.

Here is the relevant parts of my filebeat.yml

- type: log
  enabled: true
  paths:
    - /var/log/qrsender/*.log
  json.keys_under_root: true
  json.add_error_key: true

and

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - decode_json_fields:
      fields: ["error"]
      process_array: false
      max_depth: 1
      target: ""
      overwrite_keys: true
      add_error_key: false
  - rename:
      fields:
      - from: "error"
        to: "error.message"
      ignore_missing: true