Help reindexing to new index

Elastic Stack Elasticsearch
1

I have a logstash .conf file whos outputs originally look like this:

if "PAN-OS_config" in [tags] {
    elasticsearch {
        index => "config"
        hosts => ["localhost:9200"]
    }

This created indexes in Elasticsearch such as "config", "traffic", "threat", etc. Since I want to setup log rotation and I need (and I read it's best practice) to use have indexes with %{+YYYY.MM} appended to them. I've changed the .conf and updated the index to:

index => "log-panos-config-%{+YYYY.MM}"

However, when I try to reindex the old index to the new, I get the following error:

{
  "index" : "logs-panos-config-%{+YYYY.MM}",
  "type" : "_doc",
  "id" : "CTqktnYBBWs-cule_Kv7",
  "cause" : {
    "type" : "illegal_argument_exception",
    "reason" : "data_stream [logs-panos-config-%{+YYYY.MM}] must be lowercase"
  },
  "status" : 400
}

Here is what I'm running to reindex:

    curl -u user:pass -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": "panos-config"
      },
      "dest": {
        "index": "logs-panos-config-%{+YYYY.MM}"
      }
    }
    '

I'm assuming the "must be lowercase" is in reference to the "YYYY.MM" but I'm not sure what to do to get the date appended.

2

Why not use ILM? :slight_smile:

3

I'm looking around ILM now...getting a major headache lol. So am I trying to take this single index and convert it into a data stream?

4

Not sure if this is correct but I went into "Manage" for the "config" index and added a lifecycle policy I created. It gave me a message about no alias existing but it seemed to add the policy anyways?

5

The date expression you have there isn't converted in a query automatically. Logstash will do it for you, but you need to be explicit when talking to Elasticsearch.

6

I changed it from the expression to "2020.12" and then received these errors:

    {
      "index" : "logs-panos-config-2020.12",
      "type" : "_doc",
      "id" : "CTqktnYBBWs-cule_Kv7",
      "cause" : {
        "type" : "illegal_argument_exception",
        "reason" : "only write ops with an op_type of create are allowed in data streams"
      },
      "status" : 400
    }
7

I may be over complicating this...Basically I changed the output index on my logstash .conf file so now new logs are coming to the new index. Now I need a way to get the logs in the old index into the new one and have my visualizations point to the new index instead of the old one. Lol and I have no clue how.

8

I take that back. New logs are not coming into the new index (which is actually a data stream now). The logstash log is showing the same "only write ops with an op_type of create are allowed in data streams" error and saying it couldn't index event to Elasticsearch.

9

I've solved my reindex issue. I was able to successfully change the index, reindex the old into the new, and delete the old. I then changed the UUID for my visualizations to point to the new index UUID.

1 Like
10

Great to hear! Can you share what solved it, as it might help someone in future?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.