Unable to store the data in index

kish · June 11, 2018, 10:15am

Hello -

I have the below format to store the data and in my logs i able to see that data is stored but when i execute indices or see data stored i am getting nothing.

API used to store the data:

put1, err := elasticSearchClient.Index().
			Index(RegisterIndex).
			Type(RegisterDocType).
			BodyJson(reg).
			Do(elasticSearchContext)
		if err != nil {
			log.Println("Failed to store register entry, ", err.Error())
			return err
		}
log.Println("Indexed", put1.Id, "to index", put1.Index, "type \n", put1.Type)

Here RegisterIndex is 'register'
And in my logs i able to see that data is stored through the index id

2018/06/11 15:31:04 register info----> {HLRCALLP-01 RegSyveOS-nokiahsshlr01 hlrcallp 2a00:8a00:a000:4000:0:0:c0a8:4616}
2018/06/11 15:31:04 Indexed AWPuSrmbVIR4QuMOWYGv to index register type register

But when i check the incide i see the doc count as still zero.

health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   register          wvt1sSx2QnyhqDKSVURfOg   5   1          **0**            0      1.4kb           766b

And my mapping is us follows

 curl -X GET "localhost:9200/register/_mapping/"
`{"register":{"mappings":{"register":{"properties":{"Address":{"type":"ip"},"NodeName":{"type":"keyword"},"NodeType":{"type":"keyword"},"Uuid":{"type":"keyword"}}}}}}`

root@nhss01vm01oam02> curl http://localhost:9200/register/_mapping/register/field/Address?pretty
{
  "register" : {
    "mappings" : {
      "register" : {
        "Address" : {
          "full_name" : "Address",
          "mapping" : {
            "Address" : {
              "type" : "ip"
            }
          }
        }
      }
    }
  }
}

dadoonet · June 12, 2018, 12:24pm

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

kish · June 12, 2018, 1:24pm

I have edited to my best. kindly check

dadoonet · June 12, 2018, 9:06pm

Great. Next time don't use both quotes > and code formatting. It's not necessary.

Is there any way to trace exactly what is sent to elasticsearch when you call you index API from your client? I mean the REST call that is done behind the scene (URL + Body)?

Sounds like it's successful but would like to double-check.

What gives:

GET register/register/AWPuSrmbVIR4QuMOWYGv

When you run that in Kibana?

kish · June 13, 2018, 7:11am

We do not use Kibana here.
REST API call:

func saveRegisterInfo(reg RegisterInfo) (err error) {
	log.Println("Received data -> ", reg)
	if !registerExists(reg) {
		put1, err := elasticSearchClient.Index().
			Index(RegisterIndex).
			Type(RegisterDocType).
			BodyJson(reg).
			Do(elasticSearchContext)
		if err != nil {
			log.Println("Failed to store register entry, ", err.Error())
			return err
		}
		log.Println("err ->", err)
		log.Println("Indexed", put1.Id, "to index", put1.Index, "type \n", put1.Type)
	} else {
		log.Println("Registration exists, skipped")
		// TODO: check and remove multiple records and store new one.
	}
	return err
}

Data sent which is captured in log, recent log

 2018/06/13 12:33:28 Received data ->  {HLRCALLP-01 RegSyveOS-nokiahsshlr01 hlrcallp 2a00:8a00:a000:4000:0:0:c0a8:4616}
 2018/06/13 12:33:28 err -> <nil>
 2018/06/13 12:33:28 Indexed AWP39NZKp9wSKrdECrkB to index register type       register

And output requested

root@nhss01vm01oam02> curl -XGET "http://localhost:9200/register/register/AWP39NZKp9wSKrdECrkB?pretty"
{
  "_index" : "register",
  "_type" : "register",
  "_id" : "AWP39NZKp9wSKrdECrkB",
  "found" : false
 }

I even tried to search in all index

root@nhss01vm01oam02> curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "ids" : {
            "values" : ["AWP39NZKp9wSKrdECrkB"]
        }
    }
}'
{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 65,
    "successful" : 65,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

Some extra info, which might be useful

root@nhss01vm01oam02> curl -XGET 'localhost:9200/_cat/health?v&pretty'
epoch      timestamp cluster         status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1528874138 12:45:38  elastic-OAMUNIT green           3         3    130  65    0    0        0             0                  -                100.0%
(reverse-i-search)`in': curl http://localhost:9200/register/_mapp^Cg/register/field/Address?pretty
root@nhss01vm01oam02> curl -X GET "localhost:9200/register/_mapping?pretty&pretty"
{
  "register" : {
    "mappings" : {
      "register" : {
        "dynamic" : "strict",
        "properties" : {
          "Address" : {
            "type" : "ip"
          },
          "NodeName" : {
            "type" : "keyword"
          },
          "NodeType" : {
            "type" : "keyword"
          },
          "Uuid" : {
            "type" : "keyword"
          }
        }
      }
    }
  }
}
root@nhss01vm01oam02> curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size

green  open   register          bGAIpg0_RyqbUeEH0bJFYQ   5   1          0            0      1.5kb           795b

kish · June 13, 2018, 1:17pm

Am i missing something?

xeraa · June 13, 2018, 4:39pm

Can you run a GET on the ID you are getting back from your program right after inserting the data (should translate to curl -XGET http://localhost:9200/register/register/AWP39NZKp9wSKrdECrkB)?

PS: Please no pings. We are doing our best to help, but daily pings about your issue are not the way to do this.

kish · June 13, 2018, 5:48pm

Already i have posted the results , posting again.

curl -XGET http://localhost:9200/register/register/AWP39NZKp9wSKrdECrkB?pretty"
{ "_index" : "register", "_type" : "register", "_id" : "AWP39NZKp9wSKrdECrkB", "found" : false }

xeraa · June 13, 2018, 9:12pm

No, do it from within your program, since this seems to write data somewhere (right after the line log.Println("Indexed", put1.Id, "to index", put1.Index, "type \n", put1.Type)). If Elasticsearch is returning you an ID, it should index this doc. I just want to ensure that you are talking to the same Elasticsearch instance, nothing else in the rest of your program can delete the data before you query it,...

kish · June 14, 2018, 7:14am

I tried to print along with program
Code snippet

> func saveRegisterInfo(reg RegisterInfo) (err error) {
> 	if !registerExists(reg) {
> 		put1, err := elasticSearchClient.Index().
> 			Index(RegisterIndex).
> 			Type(RegisterDocType).
> 			BodyJson(reg).
> 			Do(elasticSearchContext)
> 		if err != nil {
> 			log.Println("Failed to store register entry, ", err.Error())
> 			return err
> 		}
> 		log.Println("Indexed", put1.Id, "to index", put1.Index, "type \n", put1.Type)
> 		doc, err := elasticSearchClient.Get().Index(put1.Index).Type(put1.Type).Id(put1.Id).Do(elasticSearchContext)
> 		if err != nil {
> 			log.Println("testing log ->", err)
> 		}
> 		log.Println("testing log err -> ", err)
> 		log.Println("testing log doc -> ", doc)
> 	} else {
> 		log.Println("Registration exists, skipped")
> 		// TODO: check and remove multiple records and store new one.
> 	}
> 	return err
> }

Log Dumped:

2018/06/14 12:39:11 -> register
2018/06/14 12:39:11 register info----> {HLRCALLP-01 RegSyveOS-nokiahsshlr01 hlrcallp 2a00:8a00:a000:4000:0:0:c0a8:4616}
2018/06/14 12:39:11 Indexed AWP9IHFlkjFB93J3Ns4Y to index register type
register
2018/06/14 12:39:11 testing log err ->
2018/06/14 12:39:11 testing log doc -> &{register register AWP9IHFlkjFB93J3Ns4Y 0xc4206a2740 0xc42061d3a0 true map[] }
2018/06/14 12:39:11 <- register

I able to see TRUE here. But when see actually the data it is empty.

root@nhss01vm01oam02> curl -XGET http://localhost:9200/register/register/AWP9IHFlkjFB93J3Ns4Y
{"_index":"register","_type":"register","_id":"AWP9IHFlkjFB93J3Ns4Y","found":false}

> root@nhss01vm01oam02> curl -XPOST 'localhost:9200/register/_search?pretty' -d '
> {
>   "query": { "match_all": {} }
> }'
> {
>   "took" : 2,
>   "timed_out" : false,
>   "_shards" : {
>     "total" : 5,
>     "successful" : 5,
>     "failed" : 0
>   },
>   "hits" : {
>     "total" : 0,
>     "max_score" : null,
>     "hits" : [ ]
>   }
> }
> 
> root@nhss01vm01oam02> curl -X GET "localhost:9200/register/_mapping/"
> {"register":{"mappings":{"register":{"properties":{"Address":{"type":"ip"},"NodeName":{"type":"keyword"},"NodeType":{"type":"keyword"},"Uuid":{"type":"keyword"}}}}}}root@nhss01vm01oam02>

dadoonet · June 14, 2018, 7:38am

Here you are most likely sending an empty JSON in reg IMO.

Which means that elasticsearch is skipping it as there is nothing to index.

kish · June 14, 2018, 9:51am

No, I just tried to print in the same fucntion itself and below are the logs. here 'reg-info' is the one inside the function call.

> 
> func saveRegisterInfo(reg RegisterInfo) (err error) {
> 	log.Println("reg-info ->", reg)
> 	if !registerExists(reg) {
> 		put1, err := elasticSearchClient.Index().
> 			Index(RegisterIndex).
> 			Type(RegisterDocType).
> 			BodyJson(reg).
> 			Do(elasticSearchContext)
> 		if err != nil {
> 			log.Println("Failed to store register entry, ", err.Error())
> 			return err
> 		}
2018/06/14 15:18:16 -> register
2018/06/14 15:18:16 register info----> {lavnf-cscf01 Dw1hO5S3 cscf 3a00::10}
2018/06/14 15:18:16 reg-info -> {lavnf-cscf01 Dw1hO5S3 cscf 3a00::10}
2018/06/14 15:18:16 Indexed AWP9shX9lX2VvIMTDcEO to index register type
register
2018/06/14 15:18:16 testing log err ->
2018/06/14 15:18:16 testing log doc -> &{register register AWP9shX9lX2VvIMTDcEO 0xc4205f7f50 0xc4206432a0 true map }
2018/06/14 15:18:16 <- register
2018/06/14 15:18:16 Request completed: POST /api/register Register 16.27µs
2018/06/14 15:18:16 <-- Returning
^C
root@lavnf-oam02> curl -XGET "localhost:9200/register/regsiter/AWP9shX9lX2VvIMTDcEO"
{"_index":"register","_type":"regsiter","_id":"AWP9shX9lX2VvIMTDcEO","found":false}root@lavnf-oam02>

dadoonet · June 14, 2018, 10:06am

Please remember what I told you:

{lavnf-cscf01 Dw1hO5S3 cscf 3a00::10}

Does not look like a correct JSON to me. May be that's just a logger issue though.

I'm afraid I can't help more unless you can answer this:

kish · June 14, 2018, 10:18am

I understand that you were asking what is the data sent from client to the program. I will try to explain in the logs itself.

Full code from the point program receive from client to till store the data

func Register(w http.ResponseWriter, req *http.Request, vars map[string]string) {
log.Println("-> register")
defer log.Println("<- register")

var reg RegisterInfo
err := json.Unmarshal(byte(vars["Body"]), &reg)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
if err != nil {
w.WriteHeader(http.StatusBadRequest)
log.Println("Json unmarshalling failed: ", err.Error())
writeError(w, "Json unmarshalling failed: "+err.Error())
return
}
if reg.Address == "" || reg.Uuid == "" || reg.NodeType == "" {
log.Println("Missing fields in register input - ", reg)
w.WriteHeader(http.StatusBadRequest)
writeError(w, "Missing fields in register input, "+err.Error())
return
}
log.Println("register info---->", reg)
err = saveRegisterInfo(reg)
if err == nil {
w.WriteHeader(http.StatusOK)
writeSucces(w, "Registered")
} else {
w.WriteHeader(http.StatusInternalServerError)
log.Println("Failed to save register info, " + err.Error())
writeError(w, "Failed to save register info, "+err.Error())
}
// TODO: handle already registered case
}
func saveRegisterInfo(reg RegisterInfo) (err error) {
log.Println("reg-info ->", reg)
if !registerExists(reg) {
put1, err := elasticSearchClient.Index().
Index(RegisterIndex).
Type(RegisterDocType).
BodyJson(reg).
Do(elasticSearchContext)
if err != nil {
log.Println("Failed to store register entry, ", err.Error())
return err
}
log.Println("Indexed", put1.Id, "to index", put1.Index, "type \n", put1.Type)
doc, err := elasticSearchClient.Get().Index(put1.Index).Type(put1.Type).Id(put1.Id).Do(elasticSearchContext)
if err != nil {
log.Println("testing log ->", err)
}
log.Println("testing log err -> ", err)
log.Println("testing log doc -> ", doc)
} else {
log.Println("Registration exists, skipped")
// TODO: check and remove multiple records and store new one.
}
return err
}

Logs below

2018/06/14 15:18:16 --> Entering
2018/06/14 15:18:16 Request received: POST /api/register Register 2018-06-14 15:18:16.761873558 +0530 IST
2018/06/14 15:18:16 --> Entering
2018/06/14 15:18:16 Vars received map
2018/06/14 15:18:16 No form elements in request
2018/06/14 15:18:16 <-- Returning
2018/06/14 15:18:16 --> Entering
2018/06/14 15:18:16 <-- Returning
2018/06/14 15:18:16 Authentication Failed: Credentials not set
2018/06/14 15:18:16 -> register
2018/06/14 15:18:16 register info----> {lavnf-cscf01 Dw1hO5S3 cscf 3a00::10} -> this is the data received from client
2018/06/14 15:18:16 reg-info -> {lavnf-cscf01 Dw1hO5S3 cscf 3a00::10} -> this is the data inside the saveRegisterInfo fucn
2018/06/14 15:18:16 Indexed AWP9shX9lX2VvIMTDcEO to index register type
register
2018/06/14 15:18:16 testing log err ->
2018/06/14 15:18:16 testing log doc -> &{register register AWP9shX9lX2VvIMTDcEO 0xc4205f7f50 0xc4206432a0 true map }
2018/06/14 15:18:16 <- register
2018/06/14 15:18:16 Request completed: POST /api/register Register 16.27µs
2018/06/14 15:18:16 <-- Returning

The same program works for ipv4 i.e if the reg is {lavnf-cscf01 Dw1hO5S3 cscf 1.2.3.4}

dadoonet · June 14, 2018, 10:37am

Ok. Then if elasticsearch can't index it, it will not give you back an _id.

So can you reproduce the problem please with a script that we can easily run in Kibana?

Like:

DELETE index
PUT index/_doc/1
{
  "foo": "bar"
}
GET index/_search
{
  "query": {
    "match": {
      "foo": "bar"
    }
  }
}

Note: please don't use citation to format your code but follow instructions here: Unable to store the data in index

kish · June 14, 2018, 11:30am

I have never used Kibana. But what i understood from your request is to provide data in curl format. Kindly let me know if missed something.

curl -X PUT "localhost:9200/test" -H 'Content-Type: application/json' -d'
{
    "mappings":{
		"type1":{
			"properties":{
				"Uuid":{
					"type":"keyword"
				},
				"NodeType":{
					"type":"keyword"
				},
				"Address":{
					"type":"ip"
				},
				"NodeName":{
					"type":"keyword"
				}
			}
		}
	}
}'

curl -X PUT "localhost:9200/test" -H 'Content-Type: application/json' -d'
{
	"NodeName" : "cfx" 
	"Uuid" : "yuidje" 
	"NodeType" : "cscf"
	"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'

But when i tried on my setup it is work as expected. Data is getting stored.

Christian_Dahlqvist · June 14, 2018, 11:40am

What if you add the type like this?

curl -X PUT "localhost:9200/test/type1" -H 'Content-Type: application/json' -d'
{
	"NodeName" : "cfx" 
	"Uuid" : "yuidje" 
	"NodeType" : "cscf"
	"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'

kish · June 14, 2018, 11:43am

Actually it is working with below format. In my post i missed the /type/doc_numer.

root@lavnf-oam02> curl -X PUT "localhost:9200/test/type1/3" -H 'Content-Type: application/json' -d'
{
"NodeName" : "cfx" ,
"Uuid" : "yuidje" ,
"NodeType" : "cscf",
"Address" : "2a00:8a00:a000:4000:0:0:a67:5c16"
}'

kish · June 14, 2018, 2:22pm

I found the solution atleast working fine for me.
I am using the indexname & indextype same like register/register. But when i changed to different names say like register/registerx, then it is working as expected.

Not sure how it is related but seems to an issue.

dadoonet · June 14, 2018, 3:04pm

Again, if you can reproduce from scratch we can help. Here it's hard.
I can just tell that's probably related to the mapping but this is a wild guess.

Topic		Replies	Views
Elastic Store Data Kibana	2	177	July 7, 2023
How exactly Elastic search stores the data? Elasticsearch	4	1011	July 6, 2017
Elasticsearch stores plain text Elasticsearch	2	387	December 29, 2020
Exception while storing data more than 32K Elasticsearch	5	4907	February 19, 2018
Retrieve data from elasticsearch 5.6.0 using java API Elasticsearch	3	642	February 14, 2020

Unable to store the data in index

Related Topics