Problem with raw field

Elastic Stack Kibana
1

Hi,

I have a problem using a raw field for term aggregation in a data table visualization. Whenever I try to view the data for a specific raw field used in a search through the visualization, I get no results. However, if I use the same search in Discover, I get back a result with entries and the raw field that I wanted to show in the visualization is populated correctly.

Any ideas to why the raw field data isn't showing up in the visualization?

2

Can you post a screenshot of your visualization configuration /query ?

I'm assuming you mean you have a .raw not_analyzed mapping that you are using in your datatable ?
And this shows no results ?
You are linking this visualization to a saved search that Does yield results ?

3

Sure, see attached images.

Correct, I mean a .raw field.

Screen Shot 2015-09-09 at 09.05.17.png2878×1220 166 KB
Screen Shot 2015-09-09 at 09.06.30.png2880×1640 308 KB

4

That is rather befalling... the only reason I can think of , is that there is some kind of disconnect between your raw and analyzed field (that you are seeing in Discover page). Maybe a mapping error ?

On Discover; In the left pane , click in the settings (gear) next to Available Fields and uncheck the "Hide Missing Fields".
Now find your detailedLogMessage.raw and click on it.

It will display something like

This field is present in your elasticsearch mapping but not in any documents in the search results. You may still be able to visualize or search on it.

Click on the Visualize button just below the warning. Do you get any results back for that ?

5

I followed the steps that you suggested and sure enough, I got the message that you described. However, when I click the Visualize button, no results are found.

6

Then there must be something wrong with your .raw mapping.
Although I would have thought the field would be missing completely if the mapping was wrong :-/

Can you find and paste the mapping for the index ?

7

Here is the mapping for that specific field:

"detailedLogMessage": {
  "type": "string",
  "norms": {
    "enabled": false
  },
  "fields": {
    "raw": {
      "type": "string",
      "index": "not_analyzed",
      "ignore_above": 256
    }
  }
}

Could it be the ignore_above property that causes issues? I just checked and the strings that doesn't show up in the visulization are indeed longer than 256 characters.

8

Yes, that seems to be the case. I initially thought ignore_above ignores the bits of text longer than specified value (i.e. truncates down to the specified length) - but in reality (and according to the docs) the value is completely ignored if above that length.

See this thread.

:open_mouth: - I'll have to go review all My mappings now.

9

Thanks a lot for helping out getting to the bottom with this. I also thought that it would just truncate the string but now its quite clear why I am missing some values in the reports...

10

Follow up question: the index is created by Logstash, can I somehow control the ignore_above for specific fields?

Edit: Changed the property in the index template used by logstash which solved the problem.

11

If the mapping specifies ignore_above then there's not much you can do afterwards to rectify the situation.
Kibana only displays whats already indexed by elasticsearch.

Unfortunately I have no experience with logstash (I index my documents directly from my own c++ program) - but I'm sure you should be able to change the way logstash maps the index... maybe ask the question in the logstash group ?

12

How you get so many properties for mapping for a field ?

My mapping only looks like this

GET http://localhost:9200/logindex/_mapping/logtype/field/host?pretty
{
"logindex" : {
"mappings" : {
"logtype" : {
"host" : {
"full_name" : "host",
"mapping" : {
"host" : {
"type" : "string"
}
}
}
}
}
}
}

How "norms" and "fields" show up in yours and not mine. I push my logs to ES using logstash only. Why my mapping looks different. because of this I am not able to follow some posts. Are you using some other endpoint ?
any help appreciated.
thanks

13

Hi

I have the same problem but my ignore_above is only 2

"page": { "type": "string", "norms": { "enabled": false }, "fields": { "raw": { "type": "string", "index": "not_analyzed", "ignore_above": 2 } } },

why is that?