Rejecting mapping update


(rohan joseph) #1

Hi,

I have upgraded my elasticsearch to 6.3.2. After an upgrade i am trying to reindex my data. But i am getting below error.

  "cause": {
    "type": "illegal_argument_exception",
    "reason": "Rejecting mapping update to [logstash-YYYY.DD.MM-reindexed] as the final mapping would have more than 1 type: [kube-logs, logs]"
  },

(Noémi Ványi) #2

What is your Filebeat version?


(rohan joseph) #3

Below is my Filebeat Configuration

filebeat.config.inputs:
enabled: true
path: "/etc/filebeat-additional/"
filebeat.inputs:

  • type: log
    paths: "/var/log/containers/*.log"
    tags: ["kube-logs","syslog"]
    symlinks: true
    json.message_key: log
    json.keys_under_root: true
    json.add_error_key: true
    multiline.pattern: '^\s'
    multiline.match: after
    output.logstash:
    hosts: ["${LOGSTASH_HOSTS:'logstash:5043'}"]
    timeout: 15
    logging:
    level: ${LOG_LEVEL:warning}
    processors:
  • drop_fields:
    fields: ["host"]

(rohan joseph) #4

My filebeat version is 6.3.2


(rohan joseph) #5

I am using below command for reindexing

POST /_reindex
{

"source": {

"index": "logstash-YYYY.MM.DD"

},

"dest": {

"index": "logstash-YYYY.MM.DD-reindexed"

}

}


(Noémi Ványi) #6

What were the versions of Elasticsearch and Filebeat before you updated?


(rohan joseph) #7

Before updating the version were 5.2.2


(Noémi Ványi) #8

You have two types: kube-logs and logs. You need to create separate indices for each type.


(rohan joseph) #9

Hi @kvch

My filebeat version is sending only one type. After upgrade I have restored the indexes. please could let me how to separate two indices?

Regards


(rohan joseph) #10

Can anyone help me on the same for reindexing


(Adrian Serrano) #11

So right now you have a 5.x index that contains two different types, and you want to reindex every type into a different new index.

I understand that the way of achieving this is to use the type filtering in the reindex API as explained here: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

Something like:

POST _reindex
{
  "source": {
    "index": "logstash-YYYY.MM.DD",
    "type": "logs",
  },
  "dest": {
    "index": "logstash-YYYY.MM.DD-logs-reindexed"
  }
}

and

POST _reindex
{
  "source": {
    "index": "logstash-YYYY.MM.DD",
    "type": "kube-logs",
  },
  "dest": {
    "index": "logstash-YYYY.MM.DD-kube-logs-reindexed"
  }
}

You can also map all your documents to the same index, by removing the type information, as explained here:

Otherwise, you will get more authoritative answers in #elasticsearch


(Siddaram Kj) #12

From elastic search 6 in each index we can have only one type.so create two index in new index.


(rohan joseph) #13

I guess the below link has helped me a ton.

I tried creating the index with different type as it was mapped in my index. But was ending up in confusions.
But below reindex api helped me to have single t type.

POST _reindex
{
"source": {
"index": "old"
},
"dest": {
"index": "new"
},
"script": {
"source": """
ctx._source.type = ctx._type;
ctx._type = 'doc';
""",
"lang": "painless"
}
}

Thanks a ton again


(rohan joseph) #14

True. But data of same date with multiple created confusion.


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.