Hello there,
I'm new to Elasticsearch and I'm trying to set up a cloud solution of Elasticsearch on azure with .Net 6.0 and a NEST, but I encoutered a problem when working with 7.17.4 NEST and 8.x version of Elasticsearch (same problem on local and cloud). I also use ingest pipeline to index pdf and word documents.
I can establish connection to the server sucessfully, same creating index and pipeline from code and even successfuly index a file. The problem comes when I'm trying to search for anything... From my understanding of the exception, the Json file is in a wrong format?
The search:
var searchResult = client.Search<Document>();
Exception:
JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
I'm stuck on this error for some time now. Can anybody help me, please?
My design class looks like this, where the Attachment is a NEST class.
public class Document
{
public string Id { get; set; }
public string Title { get; set; }
public string Content { get; set; }
public Attachment Attachment { get; set; }
}
I register my client as a singleton service like this:
var settings = new ConnectionSettings(Configuration.GetSection("Elasticsearch")["CloudId"],
new ApiKeyAuthenticationCredentials(Configuration.GetSection("Elasticsearch")["EncodedApiKey"]))
.DefaultIndex(Configuration.GetSection("Elasticsearch")["DefaultIndex"])
.DefaultMappingFor<Document>(m => m
.IndexName(Configuration.GetSection("Elasticsearch")["IndexName"]))
.EnableApiVersioningHeader(true)
.NodePredicate(n => n.IngestEnabled = true);
var client = new ElasticClient(settings);
services.AddSingleton(client);
There by the wiki I understand, that using .EnableApiVersioningHeader(true) should be the solution to get this working, but it does not work in my case 
Index is created like this:
var createIndexResponse = client.Indices.Create(Constants.ElasticDocumentsIndex, descriptor =>
{
return descriptor.Map<Document>(mapping => mapping
.Properties(properties => properties
.Text(textField => textField.Name(document => document.Id))
.Text(textField => textField.Name(document => document.Title))
.Text(textField => textField.Name(document => document.Content))
.Object<Attachment>(attachment => attachment
.Name(document => document.Attachment)
.AutoMap())));
});
Pipeline:
var putPipelineResponse = client.Ingest.PutPipeline(Constants.ElasticDocumentsPipeline, p => p
.Description(Constants.ElasticDocumentsPipelineDescription)
.Processors(pr => pr
.Attachment<Document>(a => a
.Field(f => f.Content)
.TargetField(f => f.Attachment))
.Remove<Document>(x => x.Field("content"))));
Indexing a file with base64 string:
var response = client.Index(new Document
{
Id = string.Format("{0}_{1}", "TestCont", filename),
Title = filename,
Content = base64File
}, i => i.Pipeline(Constants.ElasticDocumentsPipeline));
The thing is, when I use the web UI API console and run GET /documents/_search it works fine... and I get the indexed file:
{
"hits": {
"hits": [
{
"_score": 1,
"_id": "TestCont_A3TestFile.pdf",
"_source": {
"attachment": {
"content_length": 218,
"language": "en",
"format": "application/pdf; version=1.7",
"author": "Roman CERNY",
"modified": "2022-07-04T11:46:29Z",
"content": "This is a test A3 word file. \n\n \n\nWe will make some improvements in project and select a few new employees for pilot. \n\nImprovement will contain some other information and the review will be conducted on site.",
"content_type": "application/pdf",
"date": "2022-07-04T11:46:29Z",
"creator_tool": "Microsoft® Word for Microsoft 365"
},
"id": "TestCont_A3TestFile.pdf",
"title": "A3TestFile.pdf"
},
"_index": "documents"
}
],
"total": {
"relation": "eq",
"value": 1
},
"max_score": 1
Debug info:
# FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST on https://5178b7d0452742b3bfd909815bad8750.westeurope.azure.elastic-cloud.com/documents/_search?typed_keys=true
- [1] ProductCheckOnStartup: Took: 00:00:01.3838201
- [2] ProductCheckSuccess: Node: https://5178b7d0452742b3bfd909815bad8750.westeurope.azure.elastic-cloud.com/ Took: 00:00:01.3515972
- [3] BadResponse: Node: https://5178b7d0452742b3bfd909815bad8750.westeurope.azure.elastic-cloud.com/ Exception: JsonParsingException Took: 00:00:00.3082421
# Audit exception in step 3 BadResponse:
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsValueSeparatorWithVerify()
at Nest.AttachmentFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
# Inner Exception: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsValueSeparatorWithVerify()
at Nest.AttachmentFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
# Exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
---> Elasticsearch.Net.Utf8Json.JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsValueSeparatorWithVerify()
at Nest.AttachmentFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Nest.ElasticClient.Search[TDocument](ISearchRequest request)
at U3t.PDCA.BusinessLogic.Services.ElasticSearchService.Search(String query) in D:\dev\U3t.PlanDoCheckAct\U3t.PDCA.BusinessLogic\Services\ElasticSearchService.cs:line 320
at U3t.PDCA.Controllers.SearchController.SearchDocument(String query) in D:\dev\U3t.PlanDoCheckAct\U3t.PlanDoCheckAct\Controllers\SearchController.cs:line 64
at lambda_method7(Closure , Object , Object[] )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
