Help troubleshoot this template please

Elastic Stack Logstash
1

I am trying to make sure everything is indexed but the username is not analyzed. This is not working. what am i doing wrong?

{
"template" : "twitter",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256},
"user.name": {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"user.name": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

2

user.name indicates a nested field. Is it a sub-field of object user?

3

YEs it is.

4

I want to index everything except the user.name subfield which is a string. I am not sure if this would be done in the mapping or properties.

5

You need to follow the pattern outlined by geoip.location:

"geoip" : {
  "type" : "object",
  "dynamic": true,
  "properties" : {
    "location" : { "type" : "geo_point" }
  }
}

with user, as it is an object:

"user" : {
  "type" : "object",
  "dynamic": true,
  "properties" : {
    "name" : { "type": "string", "index": "not_analyzed" }
  }
}

You can add whatever other mapping properties once you've defined user as an object. This is roughly what needs to be done.

6

This is what I have but it still is analyzing it:
{
"template": "twitter",
"order": 1,
"settings": {
"number_of_shards": 1
},
"mappings": {
"tweet": {
"_all": {
"enabled": false
},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties": {
"user" : {
"type" : "object",
"dynamic": true,
"properties" : {
"name" : { "type": "string", "index": "not_analyzed" }
}
},
"text": {
"type": "string"
},
"coordinates": {
"properties": {
"coordinates": {
"type": "geo_point"
},
"type": {
"type": "string"
}
}
}
}
}
}
}

7

This mapping template will only be applied to new indices created. Are you running your check against an index which might have the former mapping applied?

8

I delete everything and reload again.

9

You should attach your mapping (not the template) here so we can see what it looks like.

10

I thought the template contained the mapping as well -

11

The template contains the mapping to be applied to new indices. Using the _mapping API for Elasticsearch allows you to see the mapping currently applied to an entire index, or even just a type within an index. In order to troubleshoot this, we need to know what mapping was actually applied.