Reindex indices

elastic: 8.15.1
kibana: 8.15.1
logstash: 8.15.1
Ubuntu 22.04LTS

Hello folks,

I've made a mistake with the way I ingest logstash pipelines, and now I've exhausted my maximum shard count (1000/1000 used). I deleted a couple of older indexes to get elastic/kibana/logstash back up and running, however I want to retain what data I have and so far haven't found a good way to do it.

My googling has lead me to the conclusion that I should be able to combine the daily indices I have into monthly ones, however each time I try and combine them I get an "Empty response from server" from curl.

The curl command I'm trying is-

curl -k -X POST -u "elastic:<password>" "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
{
  "conflicts": "proceed",
  "source": {
    "index": "index-2024.10.*"
  },
  "dest": {
    "index": "index-2024.10",
    "op_type": "create"
  }
}
'

A little further digging tells me this is most likely because the _source mapping is missing, however I never explicitly disabled it and I thought it was enabled by default.

I tried running a few calls to check mappings, but they all returned empty results as well

curl -k -X GET -u "elastic:<password>" "localhost:9200/*/_mapping?pretty"
curl -k -X GET -u "elastic:<password>" "localhost:9200/_all/_mapping?pretty"
curl -k -X GET -u "elastic:<password>" "localhost:9200/_mapping?pretty"
curl -k -X GET -u "elastic:<password>" "localhost:9200/index-2024.10.17/_mapping"

I had a look at the index in index management in Kibana and the list of mappings there doesn't show a _source mapping, but I'm not sure if that's a Kibana thing or not.

Is there any way to retroactively enable the _source mapping? Or is there any other way to combine indices without it?

Hi @Logistic_dilated

Most likely Elastic is running on HTTPS not HTTP...how did you install elasticsearch?

try

curl -k -v -u elastic https://localhost:9200

1 Like

Look at your index templates to see if source is enabled. You can also use the get mapping API on some of the indices and see if this indicates source being enabled or disabled.

In newer version there is a commercial feature called synthetic source that allows documents to be rebuilt based on indexed data.

It is however not possible to retroactively enable source and the documentation clearly describes the implications of diabling source.

I would recommend changing to monthly indices (if that will give you an appropriate index and shard size) for all new data (that will reduce shard count growth rate) and then over time delete old indices as required if you indeed have disabled source and can not reindex as a result.