How to remove field in logstash output

[neal1991 (Neal Caffery)]

I have set up an ELK stack. For the logstash instance, it has two output including Kafka and elasticsearch.

For the output of elasticsearch, I want to keep the field @timestamp. For the output of Kafka, I want to remove the field @timestamp. So I cannot just remove field @timestamp in the filter. I just want it removed for the Kafka output.

I have not found this kind of solution.

[Badger]

Use pipeline-to-pipeline communication with a forked path pattern.

[neal1991 (Neal Caffery)]

It seems that pipeline is enough. Just seperate the config of different outputs.

[neal1991 (Neal Caffery)]

Try to set the pipelines.yml

- pipeline.id: stdout
    queue.type: persisted
    config.string: |
    input {
      beats {
        port => 5044
      }
    }

    output {
      stdout{}
    }

- pipeline.id: kafka
    queue.type: persisted
    config.string: |
    input {
      beats {
        port => 5044
      }
    }
    
    output {
      kafka {
        codec => json
        topic_id => "zeek-http"
      }
    }

Start the logstash: ./bin/logstash --path.settings /Users/dongbing/tool/ELK/logstash-7.4.0/config -t. But it always indicates:

[2020-03-19T13:04:17,589][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/Users/dongbing/tool/ELK/logstash-7.4.0/config/pipelines.yml"}
ERROR: Failed to read pipelines yaml file. Location: /Users/dongbing/tool/ELK/logstash-7.4.0/config/pipelines.ym\