Unable to store the data in index

kish · June 15, 2018, 5:29am

Could you tell us what logs are required to analysis further. I hope i have shared the required info. Kindly let me know what logs are required. So that i can try reproduce and share it.

And is there any way we can modify the index type without deleting it?

dadoonet · June 15, 2018, 5:32am

I don't need logs at this stage.
I need a script that I can play to reproduce the problem.

And is there any way we can modify the index type without deleting it?

No. You need to reindex. You can look at the reindex API for that.

kish · June 15, 2018, 5:45am

here is the script kind which help you to create the index and try to add the data. In script 1st data set will be added(which is having ipv4) while the 2nd set will not added(which is having ipv6)

curl -X PUT "localhost:9200/register" -H 'Content-Type: application/json' -d'
{
"mappings":{
"register":{
	"properties":{
		"Uuid":{
			"type":"keyword"
		},
		"NodeType":{
			"type":"keyword"
		},
		"Address":{
			"type":"ip"
		},
		"NodeName":{
			"type":"keyword"
		}
	}
}
}
}'

curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "1.2.3.4"
}'
curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'

And i saw the reindex portion and it say that index name will be changed which actaully is not possible for my application as we have many program accessing with old index name.

Could you kindly let me know how we can change the indextype & mapping type alone.
say at present i have below mapping
> {

            	"settings":{
            	},
            	"mappings":{
            		"**register**":{
            			"properties":{
            				"Uuid":{
            					"type":"keyword"
            				},
            				"NodeType":{
            					"type":"keyword"
            				},
            				"Address":{
            					"type":"**keyword**"
            				},
            				"NodeName":{
            					"type":"keyword"
            				}
            			}
            		}
            	}
            }`

and i want to convert it to below mapping(highlighted the required change part)

{
	"settings":{
	},
	"mappings":{
		"**register1**":{
			"properties":{
				"Uuid":{
					"type":"keyword"
				},
				"NodeType":{
					"type":"keyword"
				},
				"Address":{
					"type":"**ip**"
				},
				"NodeName":{
					"type":"keyword"
				}
			}
		}
	}
}`

dadoonet · June 15, 2018, 6:01am

Does this script fails for you? Or works?

And i saw the reindex portion and it say that index name will be changed which actaully is not possible for my application as we have many program accessing with old index name.

Yes. You need anyway to create a new index.
This is why we are highly recommending in our trainings and talks to use aliases from the start. So you can switch the alias to a new index without stopping your application.

Here I'm afraid you will have to drop the index and rebuild it.

What you can do to optimize time wise your operation is to:

Reindex in a new index like register-01
When done, delete index register
Create an alias named register for index register-01

The last 2 operations should take a very little time.

I of course suggest that you test this scenario and your script in a test environment.

kish · June 15, 2018, 6:12am

It is both, like i have three block in that script

Creating register index -> it works fine
Putting (ipv4)data to register index -> it works fine
Putting (ipv6)data to register index -> it fails

Thanks
-Kishan

dadoonet · June 15, 2018, 6:26am

I just tried this script in Kibana and everything works fine. At least with 6.3.0.

DELETE /register
PUT /register
{
  "mappings": {
    "register": {
      "properties": {
        "Uuid": {
          "type": "keyword"
        },
        "NodeType": {
          "type": "keyword"
        },
        "Address": {
          "type": "ip"
        },
        "NodeName": {
          "type": "keyword"
        }
      }
    }
  }
}
POST /register/register
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "1.2.3.4"
}

POST /register/register
{
	"NodeName" : "cfx",
	"Uuid" : "yuidje", 
	"NodeType" : "cscf",
	"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}
GET /register/_search

This gives:

{
  "took": 38,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "register",
        "_type": "register",
        "_id": "0w0eAmQBkD3wIOkZZcWE",
        "_score": 1,
        "_source": {
          "NodeName": "cfx",
          "Uuid": "yuidje",
          "NodeType": "cscf",
          "Address": "2a00:8a00:a000:4000:0:0:a67:5c16"
        }
      },
      {
        "_index": "register",
        "_type": "register",
        "_id": "0g0eAmQBkD3wIOkZTsUa",
        "_score": 1,
        "_source": {
          "NodeName": "cfx",
          "Uuid": "yuidje",
          "NodeType": "cscf",
          "Address": "1.2.3.4"
        }
      }
    ]
  }
}

Christian_Dahlqvist · June 15, 2018, 6:36am

Are you by any chance on a very old version of Elasticsearch? If I recall correctly I think support for IPv6 addresses was added in Elasticsearch 5.0.

kish · June 15, 2018, 7:20am

I tried in my setup but it is failing, kindly let me know what i am missing

root@nhss01vm01oam02> curl -XDELETE 'localhost:9200/register'
{"acknowledged":true}root@nhs PUT "localhost:9200/register" -H 'Content-Type: application/json' -d'
{
"mappings":{
"register":{
"properties":{
"Uuid":{
"type":"keyword"
},
"NodeType":{
"type":"keyword"
},
"Address":{
"type":"ip"
},
"NodeName":{
"type":"keyword"
}
}
}
}
}'
{"acknowledged":true,"shards_acknowledged":true}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -XGET 'localhost:9200/register/_mapping'
{"register":{"mappings":{"register":{"properties":{"Address":{"type":"ip"},"NodeName":{"type":"keyword"},"NodeType":{"type":"keyword"},"Uuid":{"type":"keyword"}}}}}}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
> {
> "NodeName" : "cfx",
> "Uuid" : "yuidje", 
> "NodeType" : "cscf",
> "Address" : "1.2.3.4"
> }'
{"_index":"register","_type":"register","_id":"AWQCUJGgZGhrCD-vGBa4","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
> {
> "NodeName" : "cfx",
> "Uuid" : "yuidje", 
> "NodeType" : "cscf",
> "Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
> }'
{"_index":"register","_type":"register","_id":"AWQCULhjZGhrCD-vGBa5","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}root@nhss01vm01oam02> 
root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
> {
>   "query": { "match_all": {} }
> }'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

kish · June 15, 2018, 7:22am

Version i am using is

root@nhss01vm01oam02> curl -XGET 'localhost:9200'
{
"name" : "nhss01vm01oam02",
"cluster_name" : "elastic-OAMUNIT",
"cluster_uuid" : "zQialZKYQJKOQUnR4ChZXw",
"version" : {
"number" : "5.1.1",
"build_hash" : "5395e21",
"build_date" : "2017-02-21T12:09:24.490Z",
"build_snapshot" : true,
"lucene_version" : "6.3.0"
},
"tagline" : "You Know, for Search"
}

kish · June 15, 2018, 7:23am

Kindly check and let me know, i am afraid that there is any issue in ES 5.1.1 version

kish · June 15, 2018, 7:30am

And also i am to see that data is getting deleted without any intervention. Not sure what happening with this index. Kindly help

root@nhss01vm01oam02> curl -X POST "localhost:9200/register/register" -H 'Content-Type: application/json' -d'
{
"NodeName" : "cfx",           
"Uuid" : "yuidje", 
"NodeType" : "cscf",
"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'
{"_index":"register","_type":"register","_id":"AWQCV9wOZGhrCD-vGBbF","_version":1,"result":"created","_shards":{"total":2,"successful":2,"failed":0},"created":true}

 root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
    {
          "query": { "match_all": {} }
        }'
        {
          "took" : 2,
          "timed_out" : false,
          "_shards" : {
            "total" : 5,
            "successful" : 5,
            "failed" : 0
          },
          "hits" : {
            "total" : 1,
            "max_score" : 1.0,
            "hits" : [
              {
                "_index" : "register",
                "_type" : "register",
                "_id" : "AWQCV9wOZGhrCD-vGBbF",
                "_score" : 1.0,
                "_source" : {
                  "NodeName" : "cfx",
                  "Uuid" : "yuidje",
                  "NodeType" : "cscf",
                  "Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
                }
              }
            ]
          }
        }

  root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
        {
          "query": { "match_all": {} }
        }'
        {
          "took" : 1,
          "timed_out" : false,
          "_shards" : {
            "total" : 5,
            "successful" : 5,
            "failed" : 0
          },
          "hits" : {
            "total" : 0,
            "max_score" : null,
            "hits" : [ ]
          }
        }

dadoonet · June 15, 2018, 7:33am

Can you test with a more recent version?

system · July 13, 2018, 7:33am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to store fields Elasticsearch	4	288	July 6, 2017
Storing fields in the index? Elasticsearch	3	296	July 6, 2017
Index data storage into a cluster Elasticsearch	4	218	November 26, 2023
Elastic Store Data Kibana	2	177	July 7, 2023
How elastic store he indexed data Elasticsearch	2	454	December 28, 2019

Unable to store the data in index

Related topics