Map keyword field in kibana 6.3.0

deepsing · October 10, 2018, 10:33am

Hi
In latest verson of kibana 6.3.0, there are separate fields. One is without "keyword" and another is with "keyword". I want to map both fields and showed a single one.

dadoonet · October 10, 2018, 11:24am

You need to adapt your mapping and remove most likely the keyword one if you don't use it.

deepsing · October 10, 2018, 12:27pm

Hi @dadoonet

Could you please provide me an example.

deepsing · October 12, 2018, 2:33pm

Hi @dadoonet

Could you please provide me an example?

dadoonet · October 12, 2018, 3:01pm

In your template there's no keyword field so probably this field is generated by your application or anything which is calling elasticsearch.
Check your ingestion layer.

If you don't find it, could you share the output of:

GET Input-*/_search?size=1

deepsing · October 12, 2018, 4:15pm

Hi @dadoonet

dadoonet · October 12, 2018, 5:19pm

Could you run:

GET /Input-*/_search?size=1
{
    "query": {
        "exists" : { "field" : "@version.keyword" }
    }
}

deepsing · October 13, 2018, 2:12am

  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "skipped": 0,
    "failed": 0
  },

dadoonet · October 13, 2018, 7:46am

What is the output of:

GET Input-2018.10.12/_mapping

deepsing · October 13, 2018, 8:44am

Hi @dadoonet

{
  "Input-2018.10.13": {
    "mappings": {
      "doc": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },

dadoonet · October 13, 2018, 9:29am

What is the output of:

GET /_template

deepsing · October 13, 2018, 9:40am

Hi @dadoonet

"logstash": {
    "order": 0,
    "index_patterns": [
      "Input-%{+YYYY.MM.dd}"
    ],

dadoonet · October 13, 2018, 9:42am

Can you share the full output of the previous request? Here you copied only one part I think. At least this is not consistent with what you pasted so far.

deepsing · October 13, 2018, 9:44am

Hi @dadoonet

Because of text limit, full information is not coming.

{
  "kibana_index_template:.kibana": {
    "order": 0,
    "index_patterns": [
      ".kibana"
    ],
    "settings": {
      "index": {
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1"
      }
    },

dadoonet · October 13, 2018, 9:45am

Then share it as a gist on gist.github.com and paste the link here.

deepsing · October 13, 2018, 9:52am

Hi @dadoonet

https://gist.github.com/hello/6e1803ea3a692ac45a7e0d6175e91fb4

dadoonet · October 13, 2018, 10:14am

I think I understand now.

In the logstash template you have:

    "index_patterns": [
      "event-%{+YYYY.MM.dd}"
    ],

Where I believe it should be:

    "index_patterns": [
      "event-*"
    ],

So the logstash template is not applied to your index when the index is created the first time which explains why you have all default elasticsearch behavior.

Also, I don't understand why you have been saying that the index name is Input-2018.10.13 if actually the template is applied on event-*. That does not make sense to me.

deepsing · October 13, 2018, 10:19am

Hi @dadoonet

I want to use index pattern event-%{+YYYY.MM.dd}, but it is not matching with kibana index pattrn.

That is why I am using event-*.
Please let me know right index pattern.

dadoonet · October 13, 2018, 11:48am

No I meant your elasticsearch index template is incorrect. Fix it.

deepsing · October 14, 2018, 3:03am

Hi @dadoonet

I am using below index in logstash.conf

output {
    elasticsearch { 
	    index => "event-%{+YYYY.MM.dd}" 
	    hosts => ["localhost"]
        template => "c:/elasticsearc/template.json"		
	}

Do I also need to change it?

Actually I want to make the index in the form of "event-YYYY.MM.dd"

Thanks
Deepak