Problem with filters and facets

Elastic Stack Elasticsearch
1

Hi everyone,
I'm using ES 1.7, and i'm experiencing some issues with filters.
each doc in our index has a language (PRL), Board of Appeal(BOA), and many other filter.
Our application is developed in JAVA and deployed in 5 environments. as an example we have TEST and INTG.
In both environments, a default search is working fine, the application gets responses from elasticsearch, but in INTG environment, it's not working with PRL and BOA and some other filters but it's working in TEST.
The mapping and the package developed in both TEST and INTG are the same.

Can you please provide me an other cause that can be happen?
Thank you so much!

Best regards,
Mohammed ESSABRI

2

Some comments:

  • It's hard to know without seeing any code
  • We don't support anymore 1.7. You should upgrade to 6.x
  • Facets... I did not touch facets for years!
3

Thank you @dadoonet for your reply, but the client is working with 1.7 and it's not allowed by him to upgrade.

Please check a part of the mapping below:

image
image.png1385×669 12.4 KB

image

Many thanks!

4

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with </> icon. Check the preview window.

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

5

Dear @dadoonet,

I'm sorry about the images, so please check below the mapping of the index:
{
"index_name" : {
"mappings" : {
"default" : {
"_all" : {
"enabled" : false
},
"properties" : { }
},
"decision" : {
"transform" : {
"script" : "['decision_content', 'TLE', 'ORD', 'RES', 'REM', 'CWD', 'HDN', 'KEY', 'HDW', 'FSU'].each { field -> ctx.source[field + '' + ctx._source['lang']] = ctx._source[field] }",
"lang" : "groovy"
},
"dynamic_templates" : [ {
"de" : {
"mapping" : {
"analyzer" : "german",
"type" : "string"
},
"match" : "_de",
"match_mapping_type" : "string"
}
}, {
"fr" : {
"mapping" : {
"analyzer" : "french",
"type" : "string"
},
"match" : "_fr",
"match_mapping_type" : "string"
}
}, {
"en" : {
"mapping" : {
"analyzer" : "english",
"type" : "string"
},
"match" : "*_en",
"match_mapping_type" : "string"
}
} ],
"_all" : {
"enabled" : false
},
"_id" : {
"path" : "path"
},
"properties" : {
"APL" : {
"type" : "string"
},
"APN" : {
"type" : "string",
"index" : "not_analyzed"
},
"ART" : {
"type" : "string"
},
"BOA" : {
"type" : "string",
"index" : "not_analyzed"
},
"CSN" : {
"type" : "string",
"analyzer" : "case_insensitive_keyword"
},
"CWD" : {
"type" : "string"
},
"CWD_de" : {
"type" : "string",
"analyzer" : "german"
},
"CWD_en" : {
"type" : "string",
"analyzer" : "english"
},
"CWD_fr" : {
"type" : "string",
"analyzer" : "french"
},
"DAT" : {
"type" : "date",
"format" : "yyyyMMdd"
},
"DCI" : {
"type" : "string",
"analyzer" : "ipc_analyzer"
},
"DDO" : {
"type" : "date",
"format" : "yyyyMMdd"
},
"DIS" : {
"type" : "string",
"index" : "not_analyzed"
},
"FSU" : {
"type" : "string"
},
"FSU_de" : {
"type" : "string",
"analyzer" : "german"
},
"FSU_en" : {
"type" : "string",
"analyzer" : "english"
},
"FSU_fr" : {
"type" : "string",
"analyzer" : "french"
},
"HDN" : {
"type" : "string"
},
"HDN_de" : {
"type" : "string",
"analyzer" : "german"
},
"HDN_en" : {
"type" : "string",
"analyzer" : "english"
},
"HDN_fr" : {
"type" : "string",
"analyzer" : "french"
},
"HDW" : {
"type" : "string"
},
"HDW_de" : {
"type" : "string",
"analyzer" : "german"
},
"HDW_en" : {
"type" : "string",
"analyzer" : "english"
},
"HDW_fr" : {
"type" : "string",
"analyzer" : "french"
},
"IPC" : {
"type" : "string",
"analyzer" : "ipc_analyzer"
},
"KEY" : {
"type" : "string"
},
"KEY_de" : {
"type" : "string",
"analyzer" : "german"
},
"KEY_en" : {
"type" : "string",
"analyzer" : "english"
},
"KEY_fr" : {
"type" : "string",
"analyzer" : "french"
},
"OFJ" : {
"type" : "string",
"analyzer" : "ofj_analyzer"
},
"OPP" : {
"type" : "string"
},
"ORD" : {
"type" : "string"
},
"ORD_de" : {
"type" : "string",
"analyzer" : "german"
},
"ORD_en" : {
"type" : "string",
"analyzer" : "english"
},
"ORD_fr" : {
"type" : "string",
"analyzer" : "french"
},
"PRL" : {
"type" : "string",
"analyzer" : "case_insensitive_keyword"
},
"REM" : {
"type" : "string"
},
"REM_de" : {
"type" : "string",
"analyzer" : "german"
},
"REM_en" : {
"type" : "string",
"analyzer" : "english"
},
"REM_fr" : {
"type" : "string",
"analyzer" : "french"
},
"RES" : {
"type" : "string"
},
"RES_de" : {
"type" : "string",
"analyzer" : "german"
},
"RES_en" : {
"type" : "string",
"analyzer" : "english"
},
"RES_fr" : {
"type" : "string",
"analyzer" : "french"
},
"TLE" : {
"type" : "string"
},
"TLE_de" : {
"type" : "string",
"analyzer" : "german"
},
"TLE_en" : {
"type" : "string",
"analyzer" : "english"
},
"TLE_fr" : {
"type" : "string",
"analyzer" : "french"
},
"applicantname" : {
"type" : "string"
},
"applicationno" : {
"type" : "string"
},
"archival" : {
"type" : "boolean"
},
"art" : {
"type" : "string"
},
"board" : {
"type" : "string"
},
"case_number_keyword" : {
"type" : "string",
"analyzer" : "case_insensitive_keyword"
},
"case_number_structured" : {
"properties" : {
"csn1" : {
"type" : "string",
"analyzer" : "case_insensitive_keyword"
},

// i couldn't provide the whole mapping due to the limite of number of characters
i'll do it in the next comment

Best regards,
Mohammed ESSABRI

6

The rest of the mapping

      "csn2" : {
        "type" : "integer"
      },
      "csn3" : {
        "type" : "integer"
      }
    }
  },
  "casenumber" : {
    "type" : "string"
  },
  "catchwords" : {
    "type" : "string"
  },
  "citeddecisions" : {
    "type" : "string"
  },
  "decision_content" : {
    "type" : "string"
  },
  "decision_content_de" : {
    "type" : "string",
    "store" : true,
    "analyzer" : "german",
    "copy_to" : [ "main_search_field_de" ]
  },
  "decision_content_en" : {
    "type" : "string",
    "store" : true,
    "analyzer" : "english",
    "copy_to" : [ "main_search_field_en" ]
  },
  "decision_content_fr" : {
    "type" : "string",
    "store" : true,
    "analyzer" : "french",
    "copy_to" : [ "main_search_field_fr" ]
  },
  "decisiondate" : {
    "type" : "string"
  },
  "dis" : {
    "type" : "string"
  },
  "headnote" : {
    "type" : "string"
  },
  "headword" : {
    "type" : "string"
  },
  "highlight_decision_content" : {
    "type" : "string"
  },
  "invention_title" : {
    "type" : "string"
  },
  "ipcclass" : {
    "type" : "string"
  },
  "keywords" : {
    "type" : "string"
  },
  "lang" : {
    "type" : "string",
    "index" : "not_analyzed"
  },
  "main_search_field_de" : {
    "type" : "string",
    "analyzer" : "german"
  },
  "main_search_field_en" : {
    "type" : "string",
    "analyzer" : "english"
  },
  "main_search_field_fr" : {
    "type" : "string",
    "analyzer" : "french"
  },
  "ofj" : {
    "type" : "string"
  },
  "opponentname" : {
    "type" : "string"
  },
  "order" : {
    "type" : "string"
  },
  "path" : {
    "type" : "string",
    "analyzer" : "file_path"
  },
  "proceedingslanguage" : {
    "type" : "string"
  },
  "reasons" : {
    "type" : "string"
  },
  "remark" : {
    "type" : "string"
  },
  "size" : {
    "type" : "integer"
  },
  "summary" : {
    "type" : "string"
  },
  "test-field" : {
    "type" : "integer"
  },
  "title" : {
    "type" : "string"
  },
  "title_de" : {
    "type" : "string",
    "store" : true,
    "analyzer" : "german",
    "copy_to" : [ "main_search_field_de" ]
  },
  "title_en" : {
    "type" : "string",
    "store" : true,
    "analyzer" : "english",
    "copy_to" : [ "main_search_field_en" ]
  },
  "title_fr" : {
    "type" : "string",
    "store" : true,
    "analyzer" : "french",
    "copy_to" : [ "main_search_field_fr" ]
  },
  "type" : {
    "type" : "string",
    "index" : "not_analyzed"
  }
}

}
}
}
}

Best regards,

7

Also i showed the query that the app is sending to ES and that's it:
without filter: it's working

{
"from" : 0,
"size" : 6,
"query" : {
"filtered" : {
"query" : {
"bool" : {
"must" : [ {
"bool" : {
"should" : [ {
"bool" : {
"should" : [ {
"query_string" : {
"query" : "",
"fields" : [ "title_", "title_en^5.0" ]
}
}, {
"query_string" : {
"query" : "",
"fields" : [ "title" ]
}
} ],
"boost" : 3.0
}
}, {
"bool" : {
"should" : [ {
"query_string" : {
"query" : "",
"fields" : [ "decision_content_", "decision_content_en^5.0" ]
}
}, {
"query_string" : {
"query" : "",
"fields" : [ "decision_content" ]
}
} ]
}
} ]
}
}, {
"bool" : { }
} ]
}
},
"filter" : {
"type" : {
"value" : "decision"
}
}
}
},
"post_filter" : {
"and" : {
"filters" : [ {
"terms" : {
"lang" : [ "en" ]
}
}, {
"or" : {
"filters" : [ {
"type" : {
"value" : "section"
}
}, {
"type" : {
"value" : "decision"
}
}, {
"type" : {
"value" : "annotation"
}
} ]
}
} ]
}
},
"aggregations" : {
"unfiltered" : {
"filter" : {
"match_all" : { }
},
"aggregations" : {
"by_language" : {
"terms" : {
"field" : "lang",
"size" : 0
}
},
"by_content_type" : {
"filters" : {
"filters" : {
"annotations" : {
"type" : {
"value" : "annotation"
}
},
"books" : {
"type" : {
"value" : "section"
}
},
"indexName" : {
"type" : {
"value" : "decision"
}
}
}
}
},
"by_book" : {
"filter" : {
"type" : {
"value" : "section"
}
},
"aggregations" : {
"by_book_type" : {
"terms" : {
"field" : "type",
"size" : 0
}
}
}
},
"by_indexName" : {
"filter" : {
"type" : {
"value" : "decision"
}
},
"aggregations" : {
"by_indexName_csn1" : {
"terms" : {
"field" : "case_number_structured.csn1",
"size" : 0
}
}
}
}
}
},
"filtered" : {
"filter" : {
"and" : {
"filters" : [ {
"terms" : {
"lang" : [ "en" ]
}
}, {
"or" : {
"filters" : [ {
"type" : {
"value" : "section"
}
}, {
"type" : {
"value" : "decision"
}
}, {
"type" : {
"value" : "annotation"
}
} ]
}
} ]
}
},
"aggregations" : {
//related to agregation
}
}
},
"highlight" : {
// related to highlighting
}
}

Best regards,

8

And here is the request with the filter : is not working
{
"from" : 0,
"size" : 6,
"query" : {
"filtered" : {
"query" : {
"bool" : {
"must" : [ {
"bool" : {
"should" : [ {
"bool" : {
"should" : [ {
"query_string" : {
"query" : "",
"fields" : [ "title_", "title_en^5.0" ]
}
}, {
"query_string" : {
"query" : "",
"fields" : [ "title" ]
}
} ],
"boost" : 3.0
}
}, {
"bool" : {
"should" : [ {
"query_string" : {
"query" : "",
"fields" : [ "decision_content_", "decision_content_en^5.0" ]
}
}, {
"query_string" : {
"query" : "",
"fields" : [ "decision_content" ]
}
} ]
}
} ]
}
}, {
"bool" : {
"must" : {
"multi_match" : {
"query" : "EN",
"fields" : [ "proceedingslanguage_*", "proceedingslanguage_en^5.0" ],
"type" : "best_fields"
}
}
}
} ]
}
},
"filter" : {
"type" : {
"value" : "decision"
}
}
}
},
"post_filter" : {
"and" : {
"filters" : [ {
"terms" : {
"lang" : [ "en" ]
}
}, {
"or" : {
"filters" : [ {
"type" : {
"value" : "section"
}
}, {
"type" : {
"value" : "decision"
}
}, {
"type" : {
"value" : "annotation"
}
} ]
}
} ]
}
},
"aggregations" : {
"unfiltered" : {
"filter" : {
"match_all" : { }
},
"aggregations" : {
"by_language" : {
"terms" : {
"field" : "lang",
"size" : 0
}
},
"by_content_type" : {
"filters" : {
"filters" : {
"annotations" : {
"type" : {
"value" : "annotation"
}
},
"books" : {
"type" : {
"value" : "section"
}
},
"indexName" : {
"type" : {
"value" : "decision"
}
}
}
}
},
"by_book" : {
"filter" : {
"type" : {
"value" : "section"
}
},
"aggregations" : {
"by_book_type" : {
"terms" : {
"field" : "type",
"size" : 0
}
}
}
},
"by_indexName" : {
"filter" : {
"type" : {
"value" : "decision"
}
},
"aggregations" : {
"by_indexName_csn1" : {
"terms" : {
"field" : "case_number_structured.csn1",
"size" : 0
}
}
}
}
}
},
"filtered" : {
"filter" : {
"and" : {
"filters" : [ {
"terms" : {
"lang" : [ "en" ]
}
}, {
"or" : {
"filters" : [ {
"type" : {
"value" : "section"
}
}, {
"type" : {
"value" : "decision"
}
}, {
"type" : {
"value" : "annotation"
}
} ]
}
} ]
}
},
"aggregations" : {
// related to agregations
}
}
},
"highlight" : {
// related to the highlighting
}
}

The reponse hre is empty the number of Hits is 0.
If you need to provide you the whole response, please feel free to ask me.

9

many thanks,

Best regards,
Mohammed ESSABRI

10

Please format your code as explained to make it readable.
If it's too big text share it on gist.github.com instead.

11

Hi @dadoonet,

Thank you for your support!

Here is the code in github, please check the following link

Best regards,
Mohammed ESSABRI

12

We are wasting a lot of time here.

I said first that you should use gist.github.com. Not creating a new github repository.
Then some files are not formatted.

Then, let me repeat myself:

Please read what is described there. Look at the short example.
This kind of example is easy to copy and paste in Kibana in order to reproduce your problem and eventually fix it.

Without this, I can't test locally and help.

Please keep in mind that the simpler your example is, the more chance you have to get a fast response. Otherwise, there are tons of fields that are not involved in your problem and it will take much more time to analyse it.

13

Dear @dadoonet,

Thank you for your feedback, and sorry for the misunderstanding.

I have created the scripts as requested, can you please check in the following link?

Thanks and best regards,
Mohammed ESSABRI

14

Please, try to keep the example as simple as possible.

You have tons of fields here.

Please format your code as explained to make it readable.

The queries are not formatted and not easily readable.

I don't have a 1.7 version on my laptop but I'm unsure this can work anyway:

curl  -XPOST 'localhost:9200/dg' -d '{
  // ...
}'

May be someone else has time to look at it the way it is now though. I'm afraid I don't.

15

Dear @dadoonet,

Sorry, but If i will make it more simple then it's, may be i'll remove details that contain the source of the error i have.
I have copied it as it's generated by ES client.

But i think now it can be used directly by copying it and past in ES client.

Best regards,
Mohammed ESSABRI

16

The thing is that I'm not using elasticsearch 1.7 and I don't mean to reinstall it as it's not supported anymore so I can't try your script.
I can just try to read your script.

But as it's too long and badly formatted (see the queries), it would require too much time.

Also your script does not contain any data that are indexed in elasticsearch so I can't reproduce anything.

What we can do in the short term to move forward if you still need help: start to reproduce a similar problem on elasticsearch 6.3 and if you can, share the Kibana Dev console script here so we can help.

You opened this thread 6 days ago and we did not move at all on it.
My advice is that you take some time to reproduce on 6.x and this will totally worth it.

17

Thank you for your help!

Kibana is not working in my machine, it shows the following exception: [license][xpack] license information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. AuthentificationException[unable to authenticate user[user_name] for REST request [/_xpack]...

Alternatively, i separated the old script to have only the part producing the problem to be smaller, can you please check it?
The problem is in multi_match, here is the script.

Thank you!

18

I wanna just add something:

In the mapping, we have the following field "proceedingslanguage", but we don't have "proceedingslanguage_en", "proceedingslanguage_fr", or "proceedingslanguage_de" which "proceedingslanguage_*" represents.

The Java app that sends this script to ES is deployed in 5 environments, and in test environment, it's working fine and getting the result, but int the other environments, we don't get any result.

Using curl, we get this reply:

   {
      "took" : 10,
      "timed_out" : false,
      "_shards" : {
        "total" : 27,
        "successful" : 27,
        "failed" : 0
      },
      "hits" : {
        "total" : 0,
        "max_score" : null,
        "hits" : [ ]
      }
    }

Thank you!

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.