Fields, keywords and text

[iLemming (Ag Ibragimov)]

I'm dealing with a headache where it seems I can't find the correct set of parameters for the analyzer.

So here's the problem:

I have an index where some fields are type of text and some fields are "type": "keyword".

Let's say I have data like this:

{
  "title": "quick lazy dog" // text
  "source": "internet"      //keyword
}

Now, when I search using a query:

"query_string": {
    "query": "quick lazy dog",
    "default_operator": "and",
    "fields": ["title", "source"],
}

It works. But then, if I add a "stop" word. Like so:

"query_string": {
    "query": "the quick lazy dog",
    "default_operator": "and",
    "fields": ["title", "source"],
}

It doesn't work anymore. It won't remove "the". And the issue it seems that there are different types of fields mixed. Because when I do it like this:

"query_string": {
    "query": "the quick lazy dog",
    "default_operator": "and",
    "fields": ["title", "some_other_text_field"],
}

It works just fine.

I've been trying to configure custom analyzers, tweaking filters and tokenizer params, but I can't get it right.
Someone please help me!

Ah, btw. I'm trying to get it right for ES 5.6
we're currently migrating to ES 7, I think the semantics would slightly differ, but right now I need to solve it primarily for 5.6

[hunsw (Attila)]

I'm on 7.12 (until evening when I will be on 8.0 ), but if I interpret it right, your query is basically (let's narrow it down to 2 words only, quick dog):

(title:quick OR source:quick) AND (title:dog OR source:dog)

When you add the or any other word not appearing in your title field, it turns into

(title:quick OR source:quick) AND (title:dog OR source:dog) AND (title:someword OR source:someword)

There may be some differences due to your source field being a keyword, but what I suggest is using the explicit query in the query field so it's clear for everyone (for you too) what is the real query you are executing.

So for your example, you could search for:
query: "title:(the OR quick OR dog) AND source:\"The Quick Dog\""
(OR is the default, so the quick dog works equally fine.)

Relevant doc: Query string query | Elasticsearch Guide [7.14] | Elastic