# Why doesn't this document match? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 8, 2018, 7:37pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583 "2018-08-08T19:37:54Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 8, 2018, 7:37pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/1 "2018-08-08T19:37:54Z")
I create a document in 6.3.1 using ``` output { elasticsearch { hosts => "localhost" index => "deleteme8" } } input { generator { count => 1 message => "Foo: 'UserID:xx0001'" } } ``` If in Kibana I request "message: UserID" it does not find the document. I found this out when running categorization jobs in ML and the link to show examples would not load any documents. ML is under the impression that that document does contain UserID. There is no template for the index. I figure MLs expectations of how that document will be analyzed do not match reality. Is this an ML issue or an ES issue? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [August 10, 2018, 12:39pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/2 "2018-08-10T12:39:28Z")
I think we'd need to drop down a level to see the elasticsearch mapping and the raw JSON doc to make any headway on this. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2018, 1:35pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/3 "2018-08-10T13:35:33Z")
A search for \* in the index returns this ``` "hits": [ { "_index": "deleteme8", "_type": "doc", "_id": "ZyUGG2UBe2VO2Oy1ehDH", "_score": 1, "_source": { "message": "Foo: 'UserID:xx0001'", "@timestamp": "2018-08-08T19:32:13.828Z", "@version": "1", "host": "...", "sequence": 0 } } ] ``` GET deleteme8/\_mapping returns this ``` { "deleteme8": { "mappings": { "doc": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "host": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "sequence": { "type": "long" } } } } } } ``` Is that what you are asking for? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [August 10, 2018, 1:47pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/4 "2018-08-10T13:47:09Z")
Thanks for that. The issue is the user ID and the value are indexed as a single term - you can see the way a field is indexed using the \_analyze API: ``` POST test/_analyze { "field":"message", "text": "Foo: 'UserID:xx0001'" } ``` The output of which is: ``` { "tokens": [ { "token": "foo", "start_offset": 0, "end_offset": 3, "type": "", "position": 0 }, { "token": "userid:xx0001", "start_offset": 6, "end_offset": 19, "type": "", "position": 1 } ] } ``` Searching for `message:UserID*` works but may not be very efficient if you have lots of unique user ids of this form. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 10, 2018, 1:51pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/5 "2018-08-10T13:51:18Z")
So the fact that machine learning expects to be able to search for UserID would be an ML defect? As I said, when I click on the link to show examples it returns nothing, because if it categorized that message it would do a search for "message: Foo AND message: UserID" to find examples of it. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [August 10, 2018, 2:05pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/6 "2018-08-10T14:05:18Z")
Taking a guess I'd say the categorisation job uses a different analyzer to the one used in your search index and ML winds up discovering tokens that can't then be searched for in a drill-down. I think it would make sense to re-post this in the ML forum as a new "Can't drill down from categorisation job" issue rather than a core elasticsearch matching issue. ---
**Author:** ![droberts195](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/droberts195/32/17692_2.png) [@droberts195](https://discuss.elastic.co/u/droberts195)\ **Post date:** [August 10, 2018, 2:26pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/7 "2018-08-10T14:26:37Z")
Even though it's primarily about categorizing non-English logs, you might find [this blog](https://www.elastic.co/blog/categorizing-non-english-log-messages-in-machine-learning-for-elasticsearch) interesting. The key part in relation to what you're suffering from is: > Two things to be aware of when customizing the `categorization_analyzer` are: > > - Although techniques such as lowercasing, stemming and decompounding work well for search, for categorizing machine-generated log messages it’s best not to do these things. For example, stemming rules out the possibility of distinguishing “service starting” from “service started”. In human-generated text this could be appropriate, as people use slightly different words when writing about the same thing. But for machine-generated log messages from a given program, different words mean a different message. > - The tokens generated by the `categorization_analyzer` need to be sufficiently similar to those generated by the analyzer used at index time that when you search for them you’ll match the original message. This is required in order for drilldown from category definitions to the original data to work. It seems that you've run into this _without_ customizing anything because the `ml_classic` tokenizer splits on colons but the `standard` tokenizer doesn't. You can make things work by recreating your job with a custom `categorization_analyzer` that uses the `standard` tokenizer. Basically, add a section into your job JSON like this: ```auto "categorization_analyzer" : { "tokenizer" : "standard", "filter" : [ { "type" : "pattern_replace", "pattern": "^[0-9].*" }, { "type" : "stop", "stopwords" : [ "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun", "January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December", "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec", "GMT", "UTC" ] } ] } ``` (If you know the field you're categorizing doesn't contain dates then you can omit the stop filter to make it more concise and efficient.) The [docs](https://www.elastic.co/guide/en/elastic-stack-overview/6.3/ml-configuring-categories.html#ml-configuring-analyzer) for the `categorization_analyzer` show an example in the context of a full job config if you need it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 7, 2018, 2:26pm UTC](https://discuss.elastic.co/t/why-doesnt-this-document-match/143583/8 "2018-09-07T14:26:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.