Help reindexing to new index

Elastic Stack Elasticsearch
#1

I have a logstash .conf file whos outputs originally look like this:

if "PAN-OS_config" in [tags] {
    elasticsearch {
        index => "config"
        hosts => ["localhost:9200"]
    }

This created indexes in Elasticsearch such as "config", "traffic", "threat", etc. Since I want to setup log rotation and I need (and I read it's best practice) to use have indexes with %{+YYYY.MM} appended to them. I've changed the .conf and updated the index to:

index => "log-panos-config-%{+YYYY.MM}"

However, when I try to reindex the old index to the new, I get the following error:

{
  "index" : "logs-panos-config-%{+YYYY.MM}",
  "type" : "_doc",
  "id" : "CTqktnYBBWs-cule_Kv7",
  "cause" : {
    "type" : "illegal_argument_exception",
    "reason" : "data_stream [logs-panos-config-%{+YYYY.MM}] must be lowercase"
  },
  "status" : 400
}

Here is what I'm running to reindex:

    curl -u user:pass -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": "panos-config"
      },
      "dest": {
        "index": "logs-panos-config-%{+YYYY.MM}"
      }
    }
    '

I'm assuming the "must be lowercase" is in reference to the "YYYY.MM" but I'm not sure what to do to get the date appended.

#2

Why not use ILM? :slight_smile:

#3

I'm looking around ILM now...getting a major headache lol. So am I trying to take this single index and convert it into a data stream?

#4

Not sure if this is correct but I went into "Manage" for the "config" index and added a lifecycle policy I created. It gave me a message about no alias existing but it seemed to add the policy anyways?

#5

The date expression you have there isn't converted in a query automatically. Logstash will do it for you, but you need to be explicit when talking to Elasticsearch.

#6

I changed it from the expression to "2020.12" and then received these errors:

    {
      "index" : "logs-panos-config-2020.12",
      "type" : "_doc",
      "id" : "CTqktnYBBWs-cule_Kv7",
      "cause" : {
        "type" : "illegal_argument_exception",
        "reason" : "only write ops with an op_type of create are allowed in data streams"
      },
      "status" : 400
    }
#7

I may be over complicating this...Basically I changed the output index on my logstash .conf file so now new logs are coming to the new index. Now I need a way to get the logs in the old index into the new one and have my visualizations point to the new index instead of the old one. Lol and I have no clue how.

#8

I take that back. New logs are not coming into the new index (which is actually a data stream now). The logstash log is showing the same "only write ops with an op_type of create are allowed in data streams" error and saying it couldn't index event to Elasticsearch.

#9

I've solved my reindex issue. I was able to successfully change the index, reindex the old into the new, and delete the old. I then changed the UUID for my visualizations to point to the new index UUID.

1 Like
#10

Great to hear! Can you share what solved it, as it might help someone in future?