Es multi_match problem

Hi I have this search query (Ford in Detroit)

It is evaluating as (Ford AND in AND Detroit)

When I apply the stop word filter (analyzer) it become (Ford AND AND
Detroit)

When I run this

Here is the JSON which ES is evaluating

{ "filtered" : { "query" : { "constant_score" : { "query" : { "bool" : {
"must" : [ { "multi_match" : { "query" : "in", "fields" : [ "authorName",
"authorTags", "cnt_en", "permalink", "t_en" ], "type" : "boolean",
"operator" : "AND", "use_dis_max" : false } }, { "multi_match" : { "query"
: "Detroit", "fields" : [ "authorName", "authorTags", "cnt_en",
"permalink", "t_en" ], "type" : "boolean", "operator" : "AND",
"use_dis_max" : false } }, { "multi_match" : { "query" : "Ford", "fields" :
[ "authorName", "authorTags", "cnt_en", "permalink", "t_en" ], "type" :
"boolean", "operator" : "AND", "use_dis_max" : false } } ] } }, "boost" :
1.0 } }, "filter" : { "and" : { "filters" : [ { "or" : { "filters" : [ {
"term" : { "authorCountry" : "United States", "_cache" : false } } ] } }, {
"or" : { "filters" : [ { "term" : { "authorLanguage" : 12, "_cache" : false
} } ] } }, { "numeric_range" : { "publishedDate" : { "from" :
1355616000000, "to" : 1358467199999, "include_lower" : true,
"include_upper" : true } } } ] } } } }

Is there a way so the ES analyzer remove the multi_match when it find the
stop word and in that case my search term will become (Ford AND Detroit)

or any idea how can i remove this

Thanks

--

multi_match is analyzed so you don't need to wrap each term inside the
boolean query. Why don't you use a single multi_match, set the query to
"Ford in Detroit" and operator to AND? By doing this, your "in" will be
removed due to stopwords and your query will turn into "Ford AND Detroit"
as you want. Also, you can use TermsFilter vs. wrapping in an OR filter.
Here is your query using the Elastic.js javascript DSL:

ejs.Request()
.query(
ejs.FilteredQuery(

ejs.MultiMatchQuery(["authorName","authorTags","cnt_en","permalink","t_en"],
"Ford in Detroit")
.operator('and')
.useDisMax(false),
ejs.AndFilter([
ejs.TermsFilter('authorCountry', ['United States'])
.cache(false),
ejs.TermsFilter('authorLanguage', [12])
.cache(false),
ejs.NumericRangeFilter('publishedDate')
.from(1355616000000)
.to(1358467199999)
.includeLower(true)
.includeUpper(true)
])));

Use the FullScale Labs translator to generate the full JSON:

http://docs.fullscale.co/translator/

Thanks,
Matt Weber

On Mon, Jan 21, 2013 at 3:32 PM, suleman mubarik
sulemanmubarik@gmail.comwrote:

Hi I have this search query (Ford in Detroit)

It is evaluating as (Ford AND in AND Detroit)

When I apply the stop word filter (analyzer) it become (Ford AND AND
Detroit)

When I run this

Here is the JSON which ES is evaluating

{ "filtered" : { "query" : { "constant_score" : { "query" : { "bool" : {
"must" : [ { "multi_match" : { "query" : "in", "fields" : [ "authorName",
"authorTags", "cnt_en", "permalink", "t_en" ], "type" : "boolean",
"operator" : "AND", "use_dis_max" : false } }, { "multi_match" : {
"query" : "Detroit", "fields" : [ "authorName", "authorTags", "cnt_en",
"permalink", "t_en" ], "type" : "boolean", "operator" : "AND",
"use_dis_max" : false } }, { "multi_match" : { "query" : "Ford", "fields" :
[ "authorName", "authorTags", "cnt_en", "permalink", "t_en" ], "type" :
"boolean", "operator" : "AND", "use_dis_max" : false } } ] } }, "boost" :
1.0 } }, "filter" : { "and" : { "filters" : [ { "or" : { "filters" : [ {
"term" : { "authorCountry" : "United States", "_cache" : false } } ] } }, {
"or" : { "filters" : [ { "term" : { "authorLanguage" : 12, "_cache" : false
} } ] } }, { "numeric_range" : { "publishedDate" : { "from" :
1355616000000, "to" : 1358467199999, "include_lower" : true,
"include_upper" : true } } } ] } } } }

Is there a way so the ES analyzer remove the multi_match when it find the
stop word and in that case my search term will become (Ford AND Detroit)

or any idea how can i remove this

Thanks

--

--

Thanks Matt

I am treating "Ford in Detroit" as a phrase and it is working fine for me
and I am getting correct results

But when I search Ford in Detroit I am doing a multi single word search
i.e. I am searching for Ford AND Detroit after removing the stop words .
Ford and Detroit can be in any place in text

But ES does not convert (Ford AND in AND Detroit) to (Ford AND Detroit)
after applying stop words

So I need when ES see Ford in Detroit it should treat it as Ford Detroit

Here is the json when I send "Ford in Detroit" , (Ford Detroit) or (Ford in
Detroit)

When "Ford in Detroit"

{

"filtered": {

    "query": {

        "constant_score": {

            "query": {

                "multi_match": {

                    "query": "Ford in Detroit",

                    "fields": [

                        "authorName",

                        "authorTags",

                        "cnt_en",

                        "permalink",

                        "t_en"

                    ],

                    "type": "phrase",

                    "operator": "AND",

                    "use_dis_max": false

                }

            },

            "boost": 1

        }

    },

    "filter": {

        "and": {

            "filters": [

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorCountry": "United States",

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorLanguage": 12,

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "numeric_range": {

                        "publishedDate": {

                            "from": 1356220800000,

                            "to": 1358985600000,

                            "include_lower": true,

                            "include_upper": true

                        }

                    }

                }

            ]

        }

    }

}

}

When Ford Detroit

{

"filtered": {

    "query": {

        "constant_score": {

            "query": {

                "bool": {

                    "must": [

                        {

                            "multi_match": {

                                "query": "Detroit",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        },

                        {

                            "multi_match": {

                                "query": "Ford",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        }

                    ]

                }

            },

            "boost": 1

        }

    },

    "filter": {

        "and": {

            "filters": [

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorCountry": "United States",

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorLanguage": 12,

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "numeric_range": {

                        "publishedDate": {

                            "from": 1356220800000,

                            "to": 1358985600000,

                            "include_lower": true,

                            "include_upper": true

                        }

                    }

                }

            ]

        }

    }

}

}

When Ford in Detroit

{

"filtered": {

    "query": {

        "constant_score": {

            "query": {

                "bool": {

                    "must": [

                        {

                            "multi_match": {

                                "query": "in",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        },

                        {

                            "multi_match": {

                                "query": "Detroit",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        },

                        {

                            "multi_match": {

                                "query": "Ford",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        }

                    ]

                }

            },

            "boost": 1

        }

    },

    "filter": {

        "and": {

            "filters": [

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorCountry": "United States",

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorLanguage": 12,

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "numeric_range": {

                        "publishedDate": {

                            "from": 1356220800000,

                            "to": 1358985600000,

                            "include_lower": true,

                            "include_upper": true

                        }

                    }

                }

            ]

        }

    }

}

}

On Monday, January 21, 2013 3:32:16 PM UTC-8, suleman mubarik wrote:

Hi I have this search query (Ford in Detroit)

It is evaluating as (Ford AND in AND Detroit)

When I apply the stop word filter (analyzer) it become (Ford AND AND
Detroit)

When I run this

Here is the JSON which ES is evaluating

{ "filtered" : { "query" : { "constant_score" : { "query" : { "bool" : {
"must" : [ { "multi_match" : { "query" : "in", "fields" : [ "authorName",
"authorTags", "cnt_en", "permalink", "t_en" ], "type" : "boolean",
"operator" : "AND", "use_dis_max" : false } }, { "multi_match" : {
"query" : "Detroit", "fields" : [ "authorName", "authorTags", "cnt_en",
"permalink", "t_en" ], "type" : "boolean", "operator" : "AND",
"use_dis_max" : false } }, { "multi_match" : { "query" : "Ford", "fields" :
[ "authorName", "authorTags", "cnt_en", "permalink", "t_en" ], "type" :
"boolean", "operator" : "AND", "use_dis_max" : false } } ] } }, "boost" :
1.0 } }, "filter" : { "and" : { "filters" : [ { "or" : { "filters" : [ {
"term" : { "authorCountry" : "United States", "_cache" : false } } ] } }, {
"or" : { "filters" : [ { "term" : { "authorLanguage" : 12, "_cache" : false
} } ] } }, { "numeric_range" : { "publishedDate" : { "from" :
1355616000000, "to" : 1358467199999, "include_lower" : true,
"include_upper" : true } } } ] } } } }

Is there a way so the ES analyzer remove the multi_match when it find the
stop word and in that case my search term will become (Ford AND Detroit)

or any idea how can i remove this

Thanks

--

Don't craft the Boolean queries yourself, let Elasticsearch do it within
the match query. Use the translator again...

"Ford in Detroit", notice type as "phrase"

ejs.Request()
.query(
ejs.FilteredQuery(

ejs.MultiMatchQuery(["authorName","authorTags","cnt_en","permalink","t_en"],
"Ford in Detroit")
.operator('and')
.useDisMax(false)
.type('phrase'),
ejs.AndFilter([
ejs.TermsFilter('authorCountry', ['United States'])
.cache(false),
ejs.TermsFilter('authorLanguage', [12])
.cache(false),
ejs.NumericRangeFilter('publishedDate')
.from(1355616000000)
.to(1358467199999)
.includeLower(true)
.includeUpper(true)
])));

Ford Detroit and Ford in Detroit use the same format, notice no type set.

Defaults to boolean.

ejs.Request()
.query(
ejs.FilteredQuery(

ejs.MultiMatchQuery(["authorName","authorTags","cnt_en","permalink","t_en"],
"Ford in Detroit")
.operator('and')
.useDisMax(false),
ejs.AndFilter([
ejs.TermsFilter('authorCountry', ['United States'])
.cache(false),
ejs.TermsFilter('authorLanguage', [12])
.cache(false),
ejs.NumericRangeFilter('publishedDate')
.from(1355616000000)
.to(1358467199999)
.includeLower(true)
.includeUpper(true)
])));

The match query will create the lucene boolean queries for you behind the
scenes. It tokenizes the the query string the same as the underlying field
being searched, or you can set one manually. So if you remove stopwords on
the field, they will be removed from the query string as well. Assuming
you are using an analyzer that removes stopwords, the match query above is
the same as "Ford AND Detroit".

Thanks,
Matt Weber

On Tue, Jan 22, 2013 at 3:55 PM, suleman mubarik
sulemanmubarik@gmail.comwrote:

Thanks Matt

I am treating "Ford in Detroit" as a phrase and it is working fine for me
and I am getting correct results

But when I search Ford in Detroit I am doing a multi single word search
i.e. I am searching for Ford AND Detroit after removing the stop words .
Ford and Detroit can be in any place in text

But ES does not convert (Ford AND in AND Detroit) to (Ford AND Detroit)
after applying stop words

So I need when ES see Ford in Detroit it should treat it as Ford Detroit

Here is the json when I send "Ford in Detroit" , (Ford Detroit) or (Ford
in Detroit)

When "Ford in Detroit"

{

"filtered": {

    "query": {

        "constant_score": {

            "query": {

                "multi_match": {

                    "query": "Ford in Detroit",

                    "fields": [

                        "authorName",

                        "authorTags",

                        "cnt_en",

                        "permalink",

                        "t_en"

                    ],

                    "type": "phrase",

                    "operator": "AND",

                    "use_dis_max": false

                }

            },

            "boost": 1

        }

    },

    "filter": {

        "and": {

            "filters": [

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorCountry": "United States",

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorLanguage": 12,

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "numeric_range": {

                        "publishedDate": {

                            "from": 1356220800000,

                            "to": 1358985600000,

                            "include_lower": true,

                            "include_upper": true

                        }

                    }

                }

            ]

        }

    }

}

}

When Ford Detroit

{

"filtered": {

    "query": {

        "constant_score": {

            "query": {

                "bool": {

                    "must": [

                        {

                            "multi_match": {

                                "query": "Detroit",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        },

                        {

                            "multi_match": {

                                "query": "Ford",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        }

                    ]

                }

            },

            "boost": 1

        }

    },

    "filter": {

        "and": {

            "filters": [

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorCountry": "United States",

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorLanguage": 12,

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "numeric_range": {

                        "publishedDate": {

                            "from": 1356220800000,

                            "to": 1358985600000,

                            "include_lower": true,

                            "include_upper": true

                        }

                    }

                }

            ]

        }

    }

}

}

When Ford in Detroit

{

"filtered": {

    "query": {

        "constant_score": {

            "query": {

                "bool": {

                    "must": [

                        {

                            "multi_match": {

                                "query": "in",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        },

                        {

                            "multi_match": {

                                "query": "Detroit",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        },

                        {

                            "multi_match": {

                                "query": "Ford",

                                "fields": [

                                    "authorName",

                                    "authorTags",

                                    "cnt_en",

                                    "permalink",

                                    "t_en"

                                ],

                                "type": "boolean",

                                "operator": "AND",

                                "use_dis_max": false

                            }

                        }

                    ]

                }

            },

            "boost": 1

        }

    },

    "filter": {

        "and": {

            "filters": [

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorCountry": "United States",

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "or": {

                        "filters": [

                            {

                                "term": {

                                    "authorLanguage": 12,

                                    "_cache": false

                                }

                            }

                        ]

                    }

                },

                {

                    "numeric_range": {

                        "publishedDate": {

                            "from": 1356220800000,

                            "to": 1358985600000,

                            "include_lower": true,

                            "include_upper": true

                        }

                    }

                }

            ]

        }

    }

}

}

On Monday, January 21, 2013 3:32:16 PM UTC-8, suleman mubarik wrote:

Hi I have this search query (Ford in Detroit)

It is evaluating as (Ford AND in AND Detroit)

When I apply the stop word filter (analyzer) it become (Ford AND AND
Detroit)

When I run this

Here is the JSON which ES is evaluating

{ "filtered" : { "query" : { "constant_score" : { "query" : { "bool" : {
"must" : [ { "multi_match" : { "query" : "in", "fields" : [
"authorName", "authorTags", "cnt_en", "permalink", "t_en" ], "type" :
"boolean", "operator" : "AND", "use_dis_max" : false } }, {
"multi_match" : { "query" : "Detroit", "fields" : [ "authorName",
"authorTags", "cnt_en", "permalink", "t_en" ], "type" : "boolean",
"operator" : "AND", "use_dis_max" : false } }, { "multi_match" : { "query"
: "Ford", "fields" : [ "authorName", "authorTags", "cnt_en", "permalink",
"t_en" ], "type" : "boolean", "operator" : "AND", "use_dis_max" : false } }
] } }, "boost" : 1.0 } }, "filter" : { "and" : { "filters" : [ { "or" : {
"filters" : [ { "term" : { "authorCountry" : "United States", "_cache" :
false } } ] } }, { "or" : { "filters" : [ { "term" : { "authorLanguage" :
12, "_cache" : false } } ] } }, { "numeric_range" : { "publishedDate" : {
"from" : 1355616000000, "to" : 1358467199999, "include_lower" : true,
"include_upper" : true } } } ] } } } }

Is there a way so the ES analyzer remove the multi_match when it find the
stop word and in that case my search term will become (Ford AND Detroit)

or any idea how can i remove this

Thanks

--

--

Thanks Matt

Yes it will work for this case but if i have an other case ("foo bar " the)
where "foo bar "is a phrase and the is a stop word in this i will have
again 2 multi_match and i will get no results where i should get same
result as for "foo bar"

On Monday, January 21, 2013 3:32:16 PM UTC-8, suleman mubarik wrote:

Hi I have this search query (Ford in Detroit)

It is evaluating as (Ford AND in AND Detroit)

When I apply the stop word filter (analyzer) it become (Ford AND AND
Detroit)

When I run this

Here is the JSON which ES is evaluating

{ "filtered" : { "query" : { "constant_score" : { "query" : { "bool" : {
"must" : [ { "multi_match" : { "query" : "in", "fields" : [ "authorName",
"authorTags", "cnt_en", "permalink", "t_en" ], "type" : "boolean",
"operator" : "AND", "use_dis_max" : false } }, { "multi_match" : {
"query" : "Detroit", "fields" : [ "authorName", "authorTags", "cnt_en",
"permalink", "t_en" ], "type" : "boolean", "operator" : "AND",
"use_dis_max" : false } }, { "multi_match" : { "query" : "Ford", "fields" :
[ "authorName", "authorTags", "cnt_en", "permalink", "t_en" ], "type" :
"boolean", "operator" : "AND", "use_dis_max" : false } } ] } }, "boost" :
1.0 } }, "filter" : { "and" : { "filters" : [ { "or" : { "filters" : [ {
"term" : { "authorCountry" : "United States", "_cache" : false } } ] } }, {
"or" : { "filters" : [ { "term" : { "authorLanguage" : 12, "_cache" : false
} } ] } }, { "numeric_range" : { "publishedDate" : { "from" :
1355616000000, "to" : 1358467199999, "include_lower" : true,
"include_upper" : true } } } ] } } } }

Is there a way so the ES analyzer remove the multi_match when it find the
stop word and in that case my search term will become (Ford AND Detroit)

or any idea how can i remove this

Thanks

--

You can use query_string which is similar to match queries, but does more
processing on the query string. Using query_string you can do something
like:

ejs.Request()
.query(
ejs.FilteredQuery(
ejs.QueryStringQuery(""foo bar" the")

.fields(["authorName","authorTags","cnt_en","permalink","t_en"])
.defaultOperator('and')
.useDisMax(false),
ejs.AndFilter([
ejs.TermsFilter('authorCountry', ['United States'])
.cache(false),
ejs.TermsFilter('authorLanguage', [12])
.cache(false),
ejs.NumericRangeFilter('publishedDate')
.from(1355616000000)
.to(1358467199999)
.includeLower(true)
.includeUpper(true)
])));

In this case "foo bar" will be a phrase, and the will get not be considered.

Thanks,
Matt Weber

On Wed, Jan 23, 2013 at 12:19 PM, suleman mubarik
sulemanmubarik@gmail.comwrote:

Thanks Matt

Yes it will work for this case but if i have an other case ("foo bar " the)
where "foo bar "is a phrase and the is a stop word in this i will have
again 2 multi_match and i will get no results where i should get same
result as for "foo bar"

On Monday, January 21, 2013 3:32:16 PM UTC-8, suleman mubarik wrote:

Hi I have this search query (Ford in Detroit)

It is evaluating as (Ford AND in AND Detroit)

When I apply the stop word filter (analyzer) it become (Ford AND AND
Detroit)

When I run this

Here is the JSON which ES is evaluating

{ "filtered" : { "query" : { "constant_score" : { "query" : { "bool" : {
"must" : [ { "multi_match" : { "query" : "in", "fields" : [
"authorName", "authorTags", "cnt_en", "permalink", "t_en" ], "type" :
"boolean", "operator" : "AND", "use_dis_max" : false } }, {
"multi_match" : { "query" : "Detroit", "fields" : [ "authorName",
"authorTags", "cnt_en", "permalink", "t_en" ], "type" : "boolean",
"operator" : "AND", "use_dis_max" : false } }, { "multi_match" : { "query"
: "Ford", "fields" : [ "authorName", "authorTags", "cnt_en", "permalink",
"t_en" ], "type" : "boolean", "operator" : "AND", "use_dis_max" : false } }
] } }, "boost" : 1.0 } }, "filter" : { "and" : { "filters" : [ { "or" : {
"filters" : [ { "term" : { "authorCountry" : "United States", "_cache" :
false } } ] } }, { "or" : { "filters" : [ { "term" : { "authorLanguage" :
12, "_cache" : false } } ] } }, { "numeric_range" : { "publishedDate" : {
"from" : 1355616000000, "to" : 1358467199999, "include_lower" : true,
"include_upper" : true } } } ] } } } }

Is there a way so the ES analyzer remove the multi_match when it find the
stop word and in that case my search term will become (Ford AND Detroit)

or any idea how can i remove this

Thanks

--

--