Template indexing issue

Elastic Stack Elasticsearch
1

Hi all,

We are using elastic search to collect logs and analyze. For this purpose, we are using filebeat and metricbeat. Initially, when we had an issue in indexing the template, support people helped in resolving it. I have a few basic doubts, Before indexing when we want to index the template with a required number of shards, say shard count 1. I used the following template,

{
  "template": "te*",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z YYYY"
        }
      }
    }
  }
}

But whenever the logs are being generated for the next day, or a new index is being created it doesn't follow the template with shards count 1 instead it follows the default count 5. can somebody help?

note: I tried with "template":"metricbeat-" and filebeat- in different template as well. i.e. naming it as template 1 and 2. But it doesn't follow the shard count specified.

2

That will only match indices that start in te, etc test, ted-$DATE.

3

In that case, I tried the same by replacing the template name with metricbeat-* and filebeat-*. (to index logs on daily basis). So we noticed that when the logs are being generated for the next day it didn't index and stopped getting the data. Not sure what went wrong

4

Did you check the logs?

6 
2018/01/15 20:52:25.695474 logger.go:22: INFO   done
2018/01/15 20:52:36.621633 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63757216 beat.memstats.memory_alloc=32064464 beat.memstats.memory_total=42492377144 libbeat.config.module.running=3 libbeat.output.write.bytes=238 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119 libbeat.pipeline.events.retry=50
2018/01/15 20:53:06.621727 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63750048 beat.memstats.memory_alloc=31877808 beat.memstats.memory_total=42492394456 libbeat.config.module.running=3 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119
2018/01/15 20:53:36.621650 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63750048 beat.memstats.memory_alloc=31894936 beat.memstats.memory_total=42492411584 libbeat.config.module.running=3 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119
2018/01/15 20:54:06.621653 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63750048 beat.memstats.memory_alloc=31908632 beat.memstats.memory_total=42492425280 libbeat.config.module.running=3 libbeat.output.read.errors=1 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119
2018/01/15 20:54:36.621682 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63750048 beat.memstats.memory_alloc=31928808 beat.memstats.memory_total=42492445456 libbeat.config.module.running=3 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119
2018/01/15 20:54:55.696177 output.go:74: ERR Failed to connect: Get https://0ab4555b01b8035ba8bcf9a18f84d506.ap-southeast-2.aws.found.io:443: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2018/01/15 20:54:55.696668 logger.go:22: INFO retryer: send unwait-signal to consumer
2018/01/15 20:54:55.696685 logger.go:22: INFO   done
2018/01/15 20:54:55.696698 logger.go:22: INFO retryer: send wait signal to consumer
2018/01/15 20:54:55.696708 logger.go:22: INFO   done
2018/01/15 20:55:06.621648 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63675152 beat.memstats.memory_alloc=31960920 beat.memstats.memory_total=42492582032 libbeat.config.module.running=3 libbeat.output.write.bytes=238 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119 libbeat.pipeline.events.retry=50
2018/01/15 20:55:36.621722 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=63675152 beat.memstats.memory_alloc=31977848 beat.memstats.memory_total=42492598960 libbeat.config.module.running=3 libbeat.pipeline.clients=6 libbeat.pipeline.events.active=4119

These are few logs if you meant this. If not please let me know what is required.

7

That seems relevant, was that at the time it tried to create the new indices?

8

now :frowning: since it stopped to get the logs I deleted the old template. I think now it is using the predefined template

9

To explain in detail. I just want to change the size of shards to 1 because we are using 1 node and when we are using the predefined template we get the unassigned shards count to 5 and the cluster state turns yellow. This affects the performance and wanted to resolve that.

10

That won't help, it's yellow because the replicas are not set.

You should copy the existing template Logstash uses and then customise it :slight_smile:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.