Changing the analyzer type fields are no longer aggregatable


(Cam) #1

Hi All,
I changed the index mapping for my index, and now I can't perform visualisations on the fields!

Example index template:

PUT /_template/my_logstash
{
"order": 1000,
"index_patterns": [
  "logstash-*",
  "docker-*",
  "syslog-*",
  "ironport-*",
  "radius-*",
  "firewall-*"
],
"settings": {
  "analysis": {
    "analyzer": {
	  "keyword_lowercase": {
	    "tokenizer": "keyword",
	      "filter": ["lowercase"]
	},
	  "whitespace_lowercase": {
	    "tokenizer": "whitespace",
	      "filter": ["lowercase"]
	  }
    }
  }
},
"mappings": {
  "doc": {
    "dynamic": "true",
    "properties": {
      "bytes": {
        "type": "integer"
      },
      "bytes_in": {
        "type": "integer"
      },
      "dest_ip": {
        "type": "ip"
      },
      "src_ip": {
        "type": "ip"
      },
      "host": {
        "type": "text",
		"analyzer": "whitespace_lowercase"
      },
      "logsource": {
        "type": "text",
		"analyzer": "whitespace_lowercase"
      },
      "program": {
        "type": "text",
		"analyzer": "whitespace_lowercase"
      },
<TRUNCATED>

But on newly created indexes (and recent data into existing indexes) I can no longer use the host 'field' to split data etc.
Looking at a newly created index (with data), the 'host' field exists in the documents, and in Kibana, but in Kibana's index-patterns 'host' is searchable ONLY.
There is no 'host.keyword' like their used to be....

What can I do to fix this?


Correct src_ip field datatype to 'ip'
Create custom analyzer in index template
(Mark Walkom) #2

No, because you have explicitly set the mapping of the host field so it takes precedence over anything more generic.

You can change what you have there and add a .keyword field using https://www.elastic.co/guide/en/elasticsearch/reference/6.4/multi-fields.html


(Cam) #3

Thanks Warkolm,
I think what this forum lacks is people actually saying how they do things, so that when beginners google questions, they can actually find the answers they need.

So what I ended up doing was going to kibana > Dev Tools > Console
Then inputting:

PUT /_template/my_logstash
{
"order": 1000,
"index_patterns": [
  "logstash-*",
  "docker-*",
  "syslog-*",
  "ironport-*",
  "radius-*",
  "test-*",
 "firewall-*"
],
"settings": {
  "analysis": {
    "analyzer": {
      "keyword_lowercase": {
        "tokenizer": "keyword",
          "filter": ["lowercase"]
    },
      "whitespace_lowercase": {
        "tokenizer": "whitespace",
          "filter": ["lowercase"]
      }
    }
  }
},
"mappings": {
  "doc": {
    "dynamic": "true",
    "properties": {
      "bytes": {
        "type": "integer"
      },
      "bytes_in": {
        "type": "integer"
      },
      "bytes_out": {
        "type": "integer"
      },
      "dest_port": {
        "type": "integer"
      },
      "src_port": {
        "type": "integer"
      },
      "dest_translated_ip": {
        "type": "ip"
      },
      "src_translated_ip": {
        "type": "ip"
      },
      "dest_ip": {
        "type": "ip"
      },
      "src_ip": {
        "type": "ip"
      },
      "host": {
        "type": "text",
        "analyzer": "whitespace_lowercase",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "logsource": {
        "type": "text",
        "analyzer": "whitespace_lowercase",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "program": {
        "type": "text",
        "analyzer": "whitespace_lowercase",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "source": {
        "type": "text",
        "analyzer": "keyword_lowercase",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "sourcetype": {
        "type": "text",
        "analyzer": "whitespace_lowercase",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "username": {
        "type": "text",
        "analyzer": "whitespace_lowercase",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "geoip": {
        "properties": {
          "ip": {
            "type": "ip"
          },
          "location": {
            "type": "geo_point"
          },
          "latitude": {
            "type": "half_float"
          },
          "longitude": {
            "type": "half_float"
          }
        }
      }
    }
  }
}
}

(Mark Walkom) #4

If you provide too info much it can be overwhelming and people just give up, if it's too little the it can be underwhelming and the same thing can happen.

It's terribly hard to guess what level of knowledge someone has, but if you are after something specific then being explicit really helps us help you :slight_smile: