Unable to store the data in index


(kish) #23

Could you tell us what logs are required to analysis further. I hope i have shared the required info. Kindly let me know what logs are required. So that i can try reproduce and share it.

And is there any way we can modify the index type without deleting it?


(David Pilato) #24

I don't need logs at this stage.
I need a script that I can play to reproduce the problem.

And is there any way we can modify the index type without deleting it?

No. You need to reindex. You can look at the reindex API for that.


(kish) #25

here is the script kind which help you to create the index and try to add the data. In script 1st data set will be added(which is having ipv4) while the 2nd set will not added(which is having ipv6)

curl -X PUT "localhost:9200/register" -H 'Content-Type: application/json' -d'
{
"mappings":{
"register":{
	"properties":{
		"Uuid":{
			"type":"keyword"
		},
		"NodeType":{
			"type":"keyword"
		},
		"Address":{
			"type":"ip"
		},
		"NodeName":{
			"type":"keyword"
		}
	}
}
}
}'

curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "1.2.3.4"
}'
curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'

And i saw the reindex portion and it say that index name will be changed which actaully is not possible for my application as we have many program accessing with old index name.

Could you kindly let me know how we can change the indextype & mapping type alone.
say at present i have below mapping
> {

            	"settings":{
            	},
            	"mappings":{
            		"**register**":{
            			"properties":{
            				"Uuid":{
            					"type":"keyword"
            				},
            				"NodeType":{
            					"type":"keyword"
            				},
            				"Address":{
            					"type":"**keyword**"
            				},
            				"NodeName":{
            					"type":"keyword"
            				}
            			}
            		}
            	}
            }`

and i want to convert it to below mapping(highlighted the required change part)

{
	"settings":{
	},
	"mappings":{
		"**register1**":{
			"properties":{
				"Uuid":{
					"type":"keyword"
				},
				"NodeType":{
					"type":"keyword"
				},
				"Address":{
					"type":"**ip**"
				},
				"NodeName":{
					"type":"keyword"
				}
			}
		}
	}
}`

(David Pilato) #26

Does this script fails for you? Or works?

And i saw the reindex portion and it say that index name will be changed which actaully is not possible for my application as we have many program accessing with old index name.

Yes. You need anyway to create a new index.
This is why we are highly recommending in our trainings and talks to use aliases from the start. So you can switch the alias to a new index without stopping your application.

Here I'm afraid you will have to drop the index and rebuild it.

What you can do to optimize time wise your operation is to:

  • Reindex in a new index like register-01
  • When done, delete index register
  • Create an alias named register for index register-01

The last 2 operations should take a very little time.

I of course suggest that you test this scenario and your script in a test environment.


(kish) #27

It is both, like i have three block in that script

  1. Creating register index -> it works fine
  2. Putting (ipv4)data to register index -> it works fine
  3. Putting (ipv6)data to register index -> it fails

Thanks
-Kishan


(David Pilato) #28

I just tried this script in Kibana and everything works fine. At least with 6.3.0.

DELETE /register
PUT /register
{
  "mappings": {
    "register": {
      "properties": {
        "Uuid": {
          "type": "keyword"
        },
        "NodeType": {
          "type": "keyword"
        },
        "Address": {
          "type": "ip"
        },
        "NodeName": {
          "type": "keyword"
        }
      }
    }
  }
}
POST /register/register
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "1.2.3.4"
}

POST /register/register
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}
GET /register/_search

This gives:

{
  "took": 38,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "register",
        "_type": "register",
        "_id": "0w0eAmQBkD3wIOkZZcWE",
        "_score": 1,
        "_source": {
          "NodeName": "cfx",
          "Uuid": "yuidje",
          "NodeType": "cscf",
          "Address": "2a00:8a00:a000:4000:0:0:a67:5c16"
        }
      },
      {
        "_index": "register",
        "_type": "register",
        "_id": "0g0eAmQBkD3wIOkZTsUa",
        "_score": 1,
        "_source": {
          "NodeName": "cfx",
          "Uuid": "yuidje",
          "NodeType": "cscf",
          "Address": "1.2.3.4"
        }
      }
    ]
  }
}

(Christian Dahlqvist) #29

Are you by any chance on a very old version of Elasticsearch? If I recall correctly I think support for IPv6 addresses was added in Elasticsearch 5.0.


(kish) #31

I tried in my setup but it is failing, kindly let me know what i am missing

root@nhss01vm01oam02> curl -XDELETE 'localhost:9200/register'
{"acknowledged":true}root@nhs PUT "localhost:9200/register" -H 'Content-Type: application/json' -d'
{
"mappings":{
"register":{
"properties":{
"Uuid":{
"type":"keyword"
},
"NodeType":{
"type":"keyword"
},
"Address":{
"type":"ip"
},
"NodeName":{
"type":"keyword"
}
}
}
}
}'
{"acknowledged":true,"shards_acknowledged":true}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -XGET 'localhost:9200/register/_mapping'
{"register":{"mappings":{"register":{"properties":{"Address":{"type":"ip"},"NodeName":{"type":"keyword"},"NodeType":{"type":"keyword"},"Uuid":{"type":"keyword"}}}}}}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
> {
> "NodeName" : "cfx",
> "Uuid" : "yuidje", 
> "NodeType" : "cscf",
> "Address" : "1.2.3.4"
> }'
{"_index":"register","_type":"register","_id":"AWQCUJGgZGhrCD-vGBa4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
> {
> "NodeName" : "cfx",
> "Uuid" : "yuidje", 
> "NodeType" : "cscf",
> "Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
> }'
{"_index":"register","_type":"register","_id":"AWQCULhjZGhrCD-vGBa5","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
> {
>   "query": { "match_all": {} }
> }'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

(kish) #32

Version i am using is

root@nhss01vm01oam02> curl -XGET 'localhost:9200'
{
"name" : "nhss01vm01oam02",
"cluster_name" : "elastic-OAMUNIT",
"cluster_uuid" : "zQialZKYQJKOQUnR4ChZXw",
"version" : {
"number" : "5.1.1",
"build_hash" : "5395e21",
"build_date" : "2017-02-21T12:09:24.490Z",
"build_snapshot" : true,
"lucene_version" : "6.3.0"
},
"tagline" : "You Know, for Search"
}


(kish) #33

Kindly check and let me know, i am afraid that there is any issue in ES 5.1.1 version


(kish) #34

And also i am to see that data is getting deleted without any intervention. Not sure what happening with this index. Kindly help

root@nhss01vm01oam02> curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
{
"NodeName" : "cfx",           
"Uuid" : "yuidje", 
"NodeType" : "cscf",
"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'
{"_index":"register","_type":"register","_id":"AWQCV9wOZGhrCD-vGBbF","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}

 root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
    {
          "query": { "match_all": {} }
        }'
        {
          "took" : 2,
          "timed_out" : false,
          "_shards" : {
            "total" : 5,
            "successful" : 5,
            "failed" : 0
          },
          "hits" : {
            "total" : 1,
            "max_score" : 1.0,
            "hits" : [
              {
                "_index" : "register",
                "_type" : "register",
                "_id" : "AWQCV9wOZGhrCD-vGBbF",
                "_score" : 1.0,
                "_source" : {
                  "NodeName" : "cfx",
                  "Uuid" : "yuidje",
                  "NodeType" : "cscf",
                  "Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
                }
              }
            ]
          }
        }

  root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
        {
          "query": { "match_all": {} }
        }'
        {
          "took" : 1,
          "timed_out" : false,
          "_shards" : {
            "total" : 5,
            "successful" : 5,
            "failed" : 0
          },
          "hits" : {
            "total" : 0,
            "max_score" : null,
            "hits" : [ ]
          }
        }

(David Pilato) #35

Can you test with a more recent version?