Elastic ignoring shard settings

[Callahan]

Hi,

So, because I ended up with way, way too many shards on my 3 node cluster due to not really understanding the consequences of having too many shards and therefore accepting the default of 5, I took the decision to wipe out my indexes and start again.

The steps I have taken
Most of my Logstash confs end with the output looking similar to the below. Any that reference their own templates, also specify in said template that I only want 1 shard and 2 replicas.

output {
 if [type] == "fileserverlogs" {
    elasticsearch {
      hosts => ["elasticsearch-server:9200"]
      index => "fileserverlogs"
	  template => "C:\logstash\config\templates\default-template.json"
	  template_overwrite => false
    }
}
}

Within the C:\logstash\config\templates\default-template.json file, I have specified (amongst other things):

{
  "template" : "logstash-*",
  "version" : 60001,
  "settings" : {
    "index.refresh_interval" : "5s",
	"number_of_shards" : "1",
	"number_of_replicas" : "2"
  },

Equally, with my winlogbeat instances, I have the winlogbeat.yml set up with the following key lines:

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1

#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["172.16.2.25:5516"]

I exported the winlogbeat template from the winlogbeat 6.2.1 as so:

PS> .\winlogbeat.exe export template --es.version 6.2.1 | Out-File -Encoding UTF8 winlogbeat.template.json

then pushed it to ES (omitting the version name as I don't want to have to update my winlogbeat YMLs on all my servers. Currently, they are set to use a template called "winlogbeat")

PS > Invoke-RestMethod -Method Put -ContentType "application/json" -InFile winlogbeat.template.json -Uri http://localhost:9200/_template/winlogbeat

I checked the template was successfully uploaded by using GET _template/ - This showed the "winlogbeat" template successfully uploaded alongside the kibana template.

Next I changed the default template on the Logstash server so that any confs that I used in the future with the standard logstash-template, would also be set to 1 shard. This too was confirmed after starting up LS service. Results shown below:

  "winlogbeat": {
    "order": 1,
    "index_patterns": [
      "winlogbeat-*"
    ],
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "10000"
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "1",
        "number_of_replicas": "2"
      }
    },

=========

     "logstash": {
        "order": 0,
        "version": 60001,
        "index_patterns": [
          "logstash-*"
        ],
        "settings": {
          "index": {
            "number_of_shards": "1",
            "number_of_replicas": "2",
            "refresh_interval": "5s"

I fired up Logstash again along with the winlogbeat instances. Data started flowing in again. I watched the shards. Each index STILL has the default 5 shards and 2 replicas.

I've obviously missed something. Where else in this chain to I need to specify that I only want one shard per index? They are daily indices so they don't get big. Split up in 5 parts makes them KB in size which is what caused my issues in the first place!

Thanks for any help you can offer.