Remote Filebeat cannot ship logs to elastic ERROR [publisher_pipeline_output]

I have been trying to ship logs from a remote server to my elk. Below is my conf file. I tested my conf file using filebeat -c filebeat.yml test output everything seems fine. Could you please help. Thank you

        elasticsearch: [<elastic_host>:9200](http://<elastic_host>/)...
        parse url... OK
        connection...
        parse host... OK
        dns lookup... OK
        addresses: <elastic_host>
        dial up... OK
        TLS... WARN secure connection disabled
        talk to server... OK
        version: 7.8.1

Below is the error in the filebeat logs

```220-09-16T17:31:28.991-0700	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(http://10.176.140.123:9200)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"}],"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"},"status":400}. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"}],"type":"parse_exception","reason":"processor [set] doesn't support one or more provided configuration parameters [ignore_empty_value]","processor_type":"set"},"status":400} ```

My filebeat config file is:

                filebeat:
                  config:
                    modules:
                      path: /etc/filebeat/modules.d/*.yml
                      reload:
                        enabled: false
                  inputs:
                  - enabled: true
                    paths:
                    - /var/log/st2/st2actionrunner.gelf.log
                    - /var/log/st2/st2sensorcontainer.gelf.log
                    type: log
                logging:
                  files:
                    keepfiles: 7
                    name: filebeat
                    path: /var/log/filebeat
                    permissions: 420
                  level: warning
                  selectors:
                  - '*'
                  to_files: true
                output:
                  elasticsearch:
                    hosts:
                    - <elastic_ip_edited_for_confidentiality>:9200
                    index: ss-logs-%{[agent.version]}-%{+yyyy.MM.dd}
                path:
                  config: /etc/filebeat
                  data: /var/lib/filebeat
                  home: /usr/share/filebeat
                  logs: /var/log/filebeat
                processors:
                - add_host_metadata: null
                - add_cloud_metadata: null
                - add_docker_metadata: null
                - add_kubernetes_metadata: null
                - decode_json_fields:
                    fields:
                    - message
                setup:
                  ilm:
                    enabled: false
                  kibana: null
                  template:
                    fields: fields.yml
                    name: ss-logs
                    pattern: ss-logs-*
                    settings:
                      index:
                        number_of_shards: 1

Welcome to our community! :smiley: FYI it's not the best idea to just post an error and a config file without explaining what you are looking for.

However; Can you connect to your Elasticsearch instance?

Thank you @warkolm. Edited it. Yes,
I can connect to my Elastic instance

You'll need Elasticsearch 7.9 to use this Filebeat 7.9 module. When Filebeat 7.9.2 comes out this requirement to have ES 7.9 will be relaxed. See https://github.com/elastic/beats/pull/20908.

So downgrading to 7.8 should help right?

Yes, using Filebeat 7.8 until the new release comes out should work as those Ingest Node pipelines are not using the ignore_empty_value option in the set processor that was introduced in Elasticsearch 7.9.

Or you could update your Elasticsearch cluster to 7.9.