What datatype to select

Elastic Stack Elasticsearch
1

Hello, while importing my dataset I have this column which consist of numerical and string values. When setting the datatype in mappings to keyword its not showing the numerical values and by changing it to double its showing only numerical values. Please help figure out what datatype to choose to query all the values of a column. My kibana version is 8.5. Please help me resolve the issue.

2

Hi @Hasan

We need more details...

How are you ingesting the data?

Please show sample documents/ data samples.

What exactly do you mean "its not showing"... Not showing where / how?

keyword will accept text characters or numerical

3

While adding the control to my dashboard that particular control is not showing all the values of that field.
image

Here its only showing 66 available items. However, it consist of around 2000 values.
I have kept the type keyword in the mappings as it contains both numerical and character type values.

Also, I'm uploading the data through csv file and have changed the type of this column to text already.

4

Hi @Hasan

I need more details.. .there are a number of reasons...

Can you show me the mapping in Kibana Dev Tools

GET /my-index-name

Also Can you show me a couple sample lines of the input CSV a couple with the number a couple with the text .

Is there a @timestamp field? If so the control will only show options that are available during the time shown in the Time Picker

5

Here is the mapping for my current index.
image

Yes @timestamp field is required.

CSV file column with the number and text values.
image

6

Hi @Hasan Thanks for the information.

Next time please do not post images of text. They can be hard to read and impossible for us to help/ debug. For instance, I could not cut and paste and try that mapping myself. Next Time please use text and format it.

I forgot to ask, What version of the stack are you using?

I notice the selections that you show do not look like the selections in the CSV are you sure you selected the correct field?

You did not answer the question about the time picker? Or is there any other filter applied to the dashboard that's limiting the selection? KQL Filter, Filter Panel etc...

Screen Shot 2022-12-28 at 8.34.19 AM
Screen Shot 2022-12-28 at 8.34.19 AM2816×1558 343 KB

What happens if you tried to type in a value that you know into the control box that you do not see in the list? Does it show up?

Can you run this? It will show the number of distinct values

GET my-index/_search
{
  "size": 0, 
  "aggs": {
    "options": {
      "cardinality": {
        "field": "ITEM_CODE"
      }
    }
  }
}
7

Thanks @stephenb for your timely response. Yes I am using the @timestamp field and my records are being filtered according to the filtered time slot.

However, The issue got resolved to much extent when I changed the format of the file while uploading. Earlier I was using CSV UTF-8 format and changing it to only CSV Format made it successfully read most of the records in controls.

The only problem now is that I want to make my controls search case sensitive and I have achieved it using this

{
"settings": {
"index": {
"number_of_shards": 1,
"number_of_replicas": 1,
"analysis": {
"normalizer": {
"normalizer_case_insensitive": {
"type": "custom",
"filter": ["lowercase"]
}
}
}
}
},

But now its only showing the lower case values not upper case ones. What if I remove "filter": ["lowercase"] will solve my problem?

8

Yes get rid of that, type keyword is case sensitive by default.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.