# Logstash elasticsearch-filter module return the wrong document

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-filter-module-return-the-wrong-document/305304>\
**Category:** Logstash\
**Created:** [May 20, 2022, 5:05pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-module-return-the-wrong-document/305304 "2022-05-20T17:05:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![CyaTo](https://avatars.discourse-cdn.com/v4/letter/c/e0b2c6/32.png) [@CyaTo](https://discuss.elastic.co/u/CyaTo)\
**Post date:** [May 20, 2022, 5:05pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-module-return-the-wrong-document/305304/1 "2022-05-20T17:05:47Z")

</div>

Hi, I'm trying to run a logstash ingestion pipeline that is able to enrich an event by querying an Elasticsearch node

But so far I can't get it to work 100%, here is what I did:

My logstash pipeline look like that :

```json
input {
    http {
        port => 5140
	    ecs_compatibility => disabled
    }
}

filter {

    mutate {
        add_field => {
            "field" => "%{[headers][field]}"
            "data" => "%{[headers][data]}"
            "size" => "%{[headers][size]}"
        }
    }

    mutate {
        remove_field => ["headers"]
        remove_field => ["message"]
        remove_field => ["host"]
        remove_field => ["@version"]
    }

    elasticsearch {
        hosts => ["localhost:9200"]
        user => ["logstash"]
        index => "test_index"
        password => "${logstash.password}"
        query => "field:%{[field]} and data:%{[data]}"
        sort => "@timestamp:desc"
        fields => {
            "field" => "previousField"
            "data" => "previousData"
            "size" => "previousSize"
        }
        periodic_flush => "true"
   }

   ruby {
        code => '
        if event.get("size") == event.get("previousSize")
            event.set("sizeChangeDetected", "0")
        else
            event.set("sizeChangeDetected", "1")
        end
        '
   }
}

output {
    stdout {}
    elasticsearch {
        hosts => ["localhost:9200"]
        user => ["logstash"]
        index => "test_index"
        password => "${logstash.password}"
        index => "test_index"
    }
}

```

And now my stdout from logstash :

```json
{
                 "event" => {
        "original" => ""
    },
                  "data" => "\\First line in \\document \\with weird char",
          "previousData" => "\\Fourth \\line \\in \\document",
    "sizeChangeDetected" => "1",
                  "size" => "154",
          "previousSize" => "147",
            "@timestamp" => 2022-05-20T16:56:32.092007Z,
         "previousField" => "field_4",
                 "field" => "field_1"
}
{
                 "event" => {
        "original" => ""
    },
                  "data" => "\\Second \\line \\in \\document",
          "previousData" => "\\Fourth \\line \\in \\document",
    "sizeChangeDetected" => "1",
                  "size" => "2457",
          "previousSize" => "147",
            "@timestamp" => 2022-05-20T16:56:32.129535Z,
         "previousField" => "field_4",
                 "field" => "field_2"
}
{
                 "event" => {
        "original" => ""
    },
                  "data" => "\\Third \\line \\in \\document",
          "previousData" => "\\Fourth \\line \\in \\document",
    "sizeChangeDetected" => "1",
                  "size" => "7854",
          "previousSize" => "147",
            "@timestamp" => 2022-05-20T16:56:32.156779Z,
         "previousField" => "field_4",
                 "field" => "field_3"
}
{
                 "event" => {
        "original" => ""
    },
                  "data" => "\\Fourth \\line \\in \\document",
          "previousData" => "\\Fourth \\line \\in \\document",
    "sizeChangeDetected" => "0",
                  "size" => "147",
          "previousSize" => "147",
            "@timestamp" => 2022-05-20T16:56:32.203754Z,
         "previousField" => "field_4",
                 "field" => "field_4"
}

```

Edit, the template behind the indice, but I don't think it cause the problem :

```json
{
    "template":{
        "settings":{
            "number_of_shards": 1,
            "number_of_replicas": 0
        },
        "mappings": {
            "properties": {
                "field": { 
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword", "ignore_above": 256 
                        }
                    }
                },
                "size": { 
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword", "ignore_above": 256 
                        }
                    }
                },
                "data": { 
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword", "ignore_above": 256 
                        }
                    }
                },
                "TimeStamp": { "type": "date" }
            }
        }
    },
    "index_patterns": [
        "test_index*"
    ],
    "composed_of": []
}

```

As you can see, the request goes to Elasticsearch, but the answer is not the right one, I suspect the characters like backslash to be the cause of the problem

Has anyone else had this use case and can point me in the right direction? 🙂 thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 17, 2022, 5:06pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-filter-module-return-the-wrong-document/305304/2 "2022-06-17T17:06:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.