JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:428

Hello there,

I'm new to Elasticsearch and I'm trying to set up a cloud solution of Elasticsearch on azure with .Net 6.0 and a NEST, but I encoutered a problem when working with 7.17.4 NEST and 8.x version of Elasticsearch (same problem on local and cloud). I also use ingest pipeline to index pdf and word documents.
I can establish connection to the server sucessfully, same creating index and pipeline from code and even successfuly index a file. The problem comes when I'm trying to search for anything... From my understanding of the exception, the Json file is in a wrong format?

The search:

var searchResult = client.Search<Document>();

Exception:

JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414

I'm stuck on this error for some time now. Can anybody help me, please?

My design class looks like this, where the Attachment is a NEST class.

 public class Document
    {
        public string Id { get; set; }
        public string Title { get; set; } 
        public string Content { get; set; }
        public Attachment Attachment { get; set; }
    }

I register my client as a singleton service like this:

var settings = new ConnectionSettings(Configuration.GetSection("Elasticsearch")["CloudId"],
                new ApiKeyAuthenticationCredentials(Configuration.GetSection("Elasticsearch")["EncodedApiKey"]))
                .DefaultIndex(Configuration.GetSection("Elasticsearch")["DefaultIndex"])
                .DefaultMappingFor<Document>(m => m
                    .IndexName(Configuration.GetSection("Elasticsearch")["IndexName"]))
                .EnableApiVersioningHeader(true)
                .NodePredicate(n => n.IngestEnabled = true);

var client = new ElasticClient(settings);
services.AddSingleton(client);

There by the wiki I understand, that using .EnableApiVersioningHeader(true) should be the solution to get this working, but it does not work in my case :frowning:

Index is created like this:

var createIndexResponse = client.Indices.Create(Constants.ElasticDocumentsIndex, descriptor =>
            {
                return descriptor.Map<Document>(mapping => mapping
                    .Properties(properties => properties
                        .Text(textField => textField.Name(document => document.Id))
                        .Text(textField => textField.Name(document => document.Title))
                        .Text(textField => textField.Name(document => document.Content))
                        .Object<Attachment>(attachment => attachment
                            .Name(document => document.Attachment)
                            .AutoMap())));
            });

Pipeline:

var putPipelineResponse = client.Ingest.PutPipeline(Constants.ElasticDocumentsPipeline, p => p
               .Description(Constants.ElasticDocumentsPipelineDescription)
               .Processors(pr => pr
                   .Attachment<Document>(a => a
                       .Field(f => f.Content)
                       .TargetField(f => f.Attachment))
                   .Remove<Document>(x => x.Field("content"))));

Indexing a file with base64 string:

var response = client.Index(new Document
                {
                    Id = string.Format("{0}_{1}", "TestCont", filename),
                    Title = filename,
                    Content = base64File
                }, i => i.Pipeline(Constants.ElasticDocumentsPipeline));

The thing is, when I use the web UI API console and run GET /documents/_search it works fine... and I get the indexed file:

{
  "hits": {
    "hits": [
      {
        "_score": 1,
        "_id": "TestCont_A3TestFile.pdf",
        "_source": {
          "attachment": {
            "content_length": 218,
            "language": "en",
            "format": "application/pdf; version=1.7",
            "author": "Roman CERNY",
            "modified": "2022-07-04T11:46:29Z",
            "content": "This is a test A3 word file. \n\n \n\nWe will make some improvements in project and select a few new employees for pilot. \n\nImprovement will contain some other information and the review will be conducted on site.",
            "content_type": "application/pdf",
            "date": "2022-07-04T11:46:29Z",
            "creator_tool": "Microsoft® Word for Microsoft 365"
          },
          "id": "TestCont_A3TestFile.pdf",
          "title": "A3TestFile.pdf"
        },
        "_index": "documents"
      }
    ],
    "total": {
      "relation": "eq",
      "value": 1
    },
    "max_score": 1

Debug info:

# FailureReason: Unrecoverable/Unexpected BadResponse while attempting POST on https://5178b7d0452742b3bfd909815bad8750.westeurope.azure.elastic-cloud.com/documents/_search?typed_keys=true
 - [1] ProductCheckOnStartup: Took: 00:00:01.3838201
 - [2] ProductCheckSuccess: Node: https://5178b7d0452742b3bfd909815bad8750.westeurope.azure.elastic-cloud.com/ Took: 00:00:01.3515972
 - [3] BadResponse: Node: https://5178b7d0452742b3bfd909815bad8750.westeurope.azure.elastic-cloud.com/ Exception: JsonParsingException Took: 00:00:00.3082421
# Audit exception in step 3 BadResponse:
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsValueSeparatorWithVerify()
   at Nest.AttachmentFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
   at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
# Inner Exception: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsValueSeparatorWithVerify()
   at Nest.AttachmentFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
   at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
# Exception:
Elasticsearch.Net.UnexpectedElasticsearchClientException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
 ---> Elasticsearch.Net.Utf8Json.JsonParsingException: expected:',', actual:'"application/pdf; version=1.7"', at offset:414
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadIsValueSeparatorWithVerify()
   at Nest.AttachmentFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
   at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   at Nest.ElasticClient.Search[TDocument](ISearchRequest request)
   at U3t.PDCA.BusinessLogic.Services.ElasticSearchService.Search(String query) in D:\dev\U3t.PlanDoCheckAct\U3t.PDCA.BusinessLogic\Services\ElasticSearchService.cs:line 320
   at U3t.PDCA.Controllers.SearchController.SearchDocument(String query) in D:\dev\U3t.PlanDoCheckAct\U3t.PlanDoCheckAct\Controllers\SearchController.cs:line 64
   at lambda_method7(Closure , Object , Object[] )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()

Hi @Roman_Cerny.

This is a known compatibility issue for the v7 client with b8 server due to changes in the response. We don't have a fix currently, but a workaround is to define your own Attachment type, rather than using the built-in one. See this issue for context.

1 Like

Hi,
Thank you for this suggestion, after I implemented my own Attachment type, it started working. :slight_smile: