Logstash filter mutate modifies elasticsearch documents from input source

Elastic Stack Logstash
1

When using elasticsearch as input and adding fields in logstash filter, the field is being added to the document in the sourced elasticsearch index.

This is not an expecting behaviour, take into account this configuration:

input {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "*"
        query => '{ "query": { "query_string": { "query": "*" } } }'
        scroll => "5m"
        docinfo => true
    }
}

filter {
    mutate {
        add_field => {
            "is_migration" => true
            "_agent" => "http"
        }
    }
}

output {
    http {
        url => "https://ingest.example.com"
        http_method => "post"
        content_type => "application/json"
    }
}

Now, the source documents from the input elasticsearch are all being modified and every document has a is_migration and _agent field.

How a filter process would modify the input store? It should modify the object in the event queue but there's no way it would modify the document from the input source.

Yes, I'm aware about metadata field but how this wasn't spotted in production environments?

2

Do you have any other files in the config directory that could be interfering with this?

3

Yes, one more config file. The input data for elastic search. For your comments I suspect that they are being merged into a single logic unit.

4

Yes, that is how it works. Would that explain what you are seeing? Does the other file have an elasticsearch output?

5

Yes, the main config has output config for elasticsearch. This was supposed to be another config file for migration of data into another logstash endpoint.

6

Then you need to control the flow using conditionals.

7

Is there a way to be independent of each other in the same logstash instance?

8

Yes, but you need to use the Multi-pipeline feature.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.