Cloud elastic account,Reindex docs

Ilyoskhuja · August 7, 2019, 8:21am

I use cloud.elastic.co, my first account working (with reindexing docs and search) ,

but now I created new account with same version

I didnt change reindexing code but i got error
2019-08-06 22:56:40.131 +05:00 [Warning] Invalid NEST response built from a unsuccessful (400) low level call on POST: /atiruz/_search?typed_keys=true

# Audit trail of this API call:

* [1] BadResponse: Node: https://f87fd4280a7944--------cd160f6f03ee54.europe-west1.gcp.cloud.es.io:9243/ Took: 00:00:00.1334420

# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: POST /atiruz/_search?typed_keys=true. ServerError: Type: search_phase_execution_exception Reason: "all shards failed"

# Request:

{"from":0,"query":{"bool":{"filter":[{"geo_bounding_box":{"_name":"Location","boost":1.1,"location":{"bottom_right":{"lat":41.23020955415123,"lon":69.34550008544922},"top_left":{"lat":41.368716908561794,"lon":69.13469991455077}}}]},"size":26,"sort":[{"createdDate":{"order":"desc"}}]}

# Response:

{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to find geo_point field [location]","index_uuid":"01aoWJJfSeSLyhgXJX-uIg","index":"atiruz"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"atiruz","node":"RELnlMkQRVWY5BXnJ91ngw","reason":{"type":"query_shard_exception","reason":"failed to find geo_point field [location]","index_uuid":"01aoWJJfSeSLyhgXJX-uIg","index":"atiruz"}}]},"status":400}

I tested with my first account same code , I reindex all docs and its working without any error, but when I reindex with my second account it cant reindex properly

forloop · August 7, 2019, 9:01pm

The response indicates that the "location" field is missing in the index

Can you show the mapping that exists in the index? You can get this by issuing

GET /atiruz

in Kibana Dev Console

Ilyoskhuja · August 8, 2019, 2:25am

{
* 		"took": 7,
* 		"timed_out": false,
* 		"_shards": {
    * 		"total": 5,
    * 		"successful": 5,
    * 		"skipped": 0,
    * 		"failed": 0
* 		},
* 		"hits": {
    * 		"total": {
        * 		"value": 1143,
        * 		"relation": "eq"
    * 		},
    * 		"max_score": 1,
    * 		"hits": [
        * 		{
            * 		"_index": ".kibana_1",
            * 		"_type": "_doc",
            * 		"_id": "space:default",
            * 		"_score": 1,
            * 		"_source": {
                * 		"space": {
                    * 		"name": "Default",
                    * 		"description": "This is your default space!",
                    * 		"color": "#00bfb3",
                    * 		"_reserved": true
                * 		},
                * 		"type": "space",
                * 		"references": [ ],
                * 		"updated_at": "2019-08-05T10:45:10.572Z"
            * 		}
        * 		}
        * 		,
        * 		{
            * 		"_index": ".kibana_1",
            * 		"_type": "_doc",
            * 		"_id": "maps-telemetry:maps-telemetry",
            * 		"_score": 1,
            * 		"_source": {
                * 		"maps-telemetry": {
                    * 		"mapsTotalCount": 0,
                    * 		"timeCaptured": "2019-08-07T00:00:03.697Z",
                    * 		"attributesPerMap": {
                        * 		"dataSourcesCount": {
                            * 		"min": 0,
                            * 		"max": 0,
                            * 		"avg": 0
                        * 		},
                        * 		"layersCount": {
                            * 		"min": 0,
                            * 		"max": 0,
                            * 		"avg": 0
                        * 		},
                        * 		"layerTypesCount": { },
                        * 		"emsVectorLayersCount": { }
                    * 		}
                * 		},
                * 		"type": "maps-telemetry",
                * 		"references": [ ],
                * 		"updated_at": "2019-08-07T00:00:03.697Z"
            * 		}
        * 		}
        * 		,
        * 		{
            * 		"_index": ".kibana_task_manager",
            * 		"_type": "_doc",
            * 		"_id": "oss_telemetry-vis_telemetry",
            * 		"_score": 1,
            * 		"_source": {
                * 		"type": "task",
                * 		"task": {
                    * 		"taskType": "vis_telemetry",
                    * 		"state": "{"runs":3}",
                    * 		"params": "{}",
                    * 		"attempts": 3,
                    * 		"scheduledAt": "2019-08-05T10:45:54.160Z",
                    * 		"runAt": "2019-08-08T00:00:00.000Z",
                    * 		"status": "idle"
                * 		},
                * 		"kibana": {
                    * 		"uuid": "69bdc359-ea65-4c48-8cd9-8c673e2a1a35",
                    * 		"version": 7010199,
                    * 		"apiVersion": 1
                * 		}
            * 		}
        * 		}
        * 		,
        * 		{
            * 		"_index": ".kibana_task_manager",
            * 		"_type": "_doc",
            * 		"_id": "Maps-maps_telemetry",
            * 		"_score": 1,
            * 		"_source": {
                * 		"type": "task",
                * 		"task": {
                    * 		"taskType": "maps_telemetry",
                    * 		"state": "{"runs":1,"stats":{"mapsTotalCount":0,"timeCaptured":"2019-08-07T00:00:03.697Z","attributesPerMap":{"dataSourcesCount":{"min":0,"max":0,"avg":0},"layersCount":{"min":0,"max":0,"avg":0},"layerTypesCount":{},"emsVectorLayersCount":{}}}}",
                    * 		"params": "{}",
                    * 		"attempts": 0,
                    * 		"scheduledAt": "2019-08-05T10:45:52.473Z",
                    * 		"runAt": "2019-08-08T00:00:00.000Z",
                    * 		"status": "idle"
                * 		},
                * 		"kibana": {
                    * 		"uuid": "69bdc359-ea65-4c48-8cd9-8c673e2a1a35",
                    * 		"version": 7010199,
                    * 		"apiVersion": 1
                * 		}
            * 		}
        * 		}
        * 		,
      
        * 		{
            * 		"_index": "atiruz",
            * 		"_type": "_doc",
            * 		"_id": "2793",
            * 		"_score": 1,
            * 		"_source": {
                * 		"location": {
                    * 		"lat": 41.29574805579102,
                    * 		"lon": 69.19265627861022
                * 		},
                * 		"id": 2793,
                * 		"buildTitle": "Продажа Квартиры: Район - Чиланзар Улица/Квартал - 14",
                * 		"createdDate": "2019-05-03T14:05:34.7268961",
                * 		"totalRooms": 3,
                * 		"description": "Продажа Квартиры: #Район - Чиланзар Улица/Квартал - 14 Ориентир - около дороги #Комнат - 3 Этаж - 2 Этажей - 4 Площадь м² - 65 Состояние - чистое Описание - балкон 1.5×3 #Цена - 33 000 $ Телефон - +998977338845",
                * 		"ownerPrice": 33000,
                * 		"size": 65,
                * 		"sizeName": "kv",
                * 		"moneyType": "Dollar",
                * 		"buildAction": "Sotiladi",
                * 		"buildType": "Kvartira",
                * 		"address": "#Район - Чиланзар Улица/Квартал - 14",
                * 		"buildingRooms": [
                    * 		{
                        * 		"roomType": 12,
                        * 		"roomCount": 2
                    * 		}
                    * 		,
                    * 		{
                        * 		"roomType": 14,
                        * 		"roomCount": 1
                    * 		}
                * 		],
                * 		"buildingPhotos": [
                    * 		{
                        * 		"title": "ec76d27c-445e-4cb6-a6da-ed7ab999c693.jpg",
                        * 		"path": "Storage/Building/Photo\bp1c3iio.mrp.jpg"
                    * 		}
                * 		]
            * 		}
        * 		}
    * 		]
* 		}
}

I created new index and everything is working, but I wonder, where is problem I used same code,why is happened?

forloop · August 8, 2019, 3:27am

Please format your code, logs or configuration files using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel to help see what it looks like.

The response you've posted looks like search results. Please show the mapping that exists in the index atiruz, with

GET /atiruz

The response will look something like

{
  "atiruz" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "acceptedAnswerId" : {
          "type" : "integer"
        },
        "answerCount" : {
          "type" : "integer"
        },
        "title" : {
          "type" : "text",
          "norms" : false,
          "fields" : {
            "raw" : {
              "type" : "keyword"
            }
          },
          "analyzer" : "expand"
        }
      }
    },
    "settings" : {
      "index" : {
        "refresh_interval" : "1s",
        "number_of_shards" : "3",
        "provided_name" : "posts",
        "creation_date" : "1565087735315",
        "analysis" : {
        }
      }
    }
  }
}

although the properties are going to be different in your case

Ilyoskhuja · August 9, 2019, 8:52pm

{

* "atiruz": {
  * "mappings": {
    * "properties": {
      * "address": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "buildAction": {
        * "type": "integer"},
      * "buildTitle": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "buildType": {
        * "type": "integer"},
      * "buildingPhotos": {
        * "properties": {
          * "path": {
            * "type": "text",
            * "fields": {
              * "keyword": {
                * "type": "keyword",
                * "ignore_above": 256}}},
          * "title": {
            * "type": "text",
            * "fields": {
              * "keyword": {
                * "type": "keyword",
                * "ignore_above": 256}}}}},
      * "buildingRooms": {
        * "properties": {
          * "roomCount": {
            * "type": "integer"},
          * "roomType": {
            * "type": "integer"}}},
      * "createdDate": {
        * "type": "date"},
      * "description": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "districtRu": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "districtUz": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "id": {
        * "type": "integer"},
      * "location": {
        * "type": "geo_point"},
      * "moneyType": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "ownerPrice": {
        * "type": "double"},
      * "size": {
        * "type": "double"},
      * "sizeName": {
        * "type": "text",
        * "fields": {
          * "keyword": {
            * "type": "keyword",
            * "ignore_above": 256}}},
      * "totalRooms": {
        * "type": "integer"}}}}

}

Sometimes it doesn't create index with location field

forloop · August 12, 2019, 7:42am

How are you creating the index? Can you please show the source code that you're using?

Ilyoskhuja · August 16, 2019, 4:43am

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

using Nest;
using XonadonUz.Models;

namespace XonadonUz
{public static class ElasticsearchExtensions
    {
        public static void AddElasticsearch(
            this IServiceCollection services, IConfiguration configuration)
        {
//honadon https:/----------------------------
            var url = configuration["https://f
            var defaultIndex ="======";
           
        //    var defaultIndex = configuration["example_index"];
        //     var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var list = new List<string>();
            var settings = new ConnectionSettings(

                new Uri("https://f
                .DefaultIndex(defaultIndex)
                //honadon
              //  .BasicAuthentication("=====", "=============")
                ///atiruz
                .BasicAuthentication("======", "==============")

                
                  
    .DisableDirectStreaming() 
    .OnRequestCompleted(apiCallDetails => 
    {
        // log out the request and the request body, if one exists for the type of request
        if (apiCallDetails.RequestBodyInBytes != null)
        {
            list.Add(
                $"{apiCallDetails.HttpMethod} {apiCallDetails.Uri} " +
                $"{Encoding.UTF8.GetString(apiCallDetails.RequestBodyInBytes)}");
        }
        else
        {
            list.Add($"{apiCallDetails.HttpMethod} {apiCallDetails.Uri}");
        }

        // log out the response and the response body, if one exists for the type of response
        if (apiCallDetails.ResponseBodyInBytes != null)
        {
            list.Add($"Status: {apiCallDetails.HttpStatusCode}" +
                     $"{Encoding.UTF8.GetString(apiCallDetails.ResponseBodyInBytes)}");
        }
        else
        {
            list.Add($"Status: {apiCallDetails.HttpStatusCode}");
        }
    });










            // AddDefaultMappings(settings);

            var client = new ElasticClient(settings);

            services.AddSingleton<IElasticClient>(client);

            CreateIndex(client, defaultIndex);
        }

        private static void AddDefaultMappings(ConnectionSettings settings)
        {
            settings
                .DefaultMappingFor<ElasticSearchModel>(m => m);
                   
        }

        private static void CreateIndex(IElasticClient client, string indexName)
        {
            var createIndexResponse = client.Indices.Create("honadonz12", c => c
                          .Map<ElasticSearchModel>(mm => mm
                         .AutoMap()
                            )
                            .RequestConfiguration(r => r
                            .DisableDirectStreaming()
    )
                  );

                   Console.WriteLine("Writeline is :" + createIndexResponse);
        }
    }
}

forloop · August 19, 2019, 6:53am

I think what may be happening here is that a document is sent to be indexed into an Elasticsearch index before that index exists, which causes Elasticsearch to create the index and infer the mapping from the document it receives.

This behaviour can be changed to not automatically create the index when a document is sent to be indexed, and instead explicitly require the index be created beforehand, by updating cluster settings with

PUT _cluster/settings
{
    "persistent": {
        "action.auto_create_index": "false" 
    }
}

forloop · August 19, 2019, 7:00am

Ilyoskhuja:

var createIndexResponse = client.Indices.Create("honadonz12", c => c
    .Map<ElasticSearchModel>(mm => mm
        .AutoMap()
    ) 
    .RequestConfiguration(r => r
        .DisableDirectStreaming()  
    ) 
);

I'd recommend also

checking to see if the index already exists before trying to create it

if (!client.Indices.Exists("honadonz12").Exists)
{
    // create the index		
}

checking that the index was successfully created, by looking at the createIndexResponse

 var createIndexResponse = client.Indices.Create("honadonz12", c => c
 	.Map<ElasticSearchModel>(mm => mm
 		.AutoMap()
 	)
 	.RequestConfiguration(r => r
 		.DisableDirectStreaming()
 	)
 );
 
 if (!createIndexResponse.IsValid)
 {
 	// do something
 }

system · September 16, 2019, 7:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.