Nest 6.8.3 getting InvalidProgramException sometimes

I see an internal ticket was created(00487754) and this topic was closed. We were getting this in our prod instance yesterday. Nest 6.6 getting InvalidProgramException sometimes

Envionrment:
AWS ECS Fargate
Docker container
linux - debian(mcr.microsoft.com/dotnet/core/aspnet:3.1)
.Net Core 3.1
NEST 6.8.3

Stack trace:

Message Common Language Runtime detected an invalid program.stackTrace at [Elasticsearch.Net](http://elasticsearch.net/).Transport `1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters) at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) at [Elasticsearch.Net](http://elasticsearch.net/).Transport` 1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
at [Elasticsearch.Net](http://elasticsearch.net/).ElasticLowLevelClient.DoRequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
at [Elasticsearch.Net](http://elasticsearch.net/).ElasticLowLevelClient.SearchAsync[TResponse](String index, String type, PostData body, SearchRequestParameters requestParameters, CancellationToken ctx)
at Nest.LowLevelDispatch.SearchDispatchAsync[TResponse](IRequest `1 p, SerializableData` 1 body, CancellationToken ct)
at Nest.ElasticClient.b__563_0[T,TResult](ISearchRequest p, SerializableData `1 d, CancellationToken c) at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.DispatchAsync[TRequest,TQueryString,TResponse,TResponseInterface](TRequest request, CancellationToken cancellationToken, Func` 3 responseGenerator, Func`4 dispatch)
at App.Core.Search.SearchRepository.SearchAsync(BenchmarkSearchQuery search, CancellationToken cancellationToken) in /src/App/Core/Search/SearchRepository.cs:line 39
at App.Controllers.SearchController.SearchAsync(BenchmarkSearchRequest request, CancellationToken cancellationToken) in /src/App/Controllers/SearchController.cs:line 77
at lambda_method(Closure , Object )
at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Logged|12_1(ControllerActionInvoker invoker)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

Are you able to capture a dump when this happens, using something like dotnet-dump and share it with me? Feel free to send me a private message to discuss how to share.

My initial suspicion is that we're looking at a platform issue. There aren't any recent changes in NEST 6.x that I can see which might be the cause of this. It looks like it may possibly be related to:

The error message Common Language Runtime detected an invalid program. implies the generation of invalid IL. Does it happen all the time, or only sometimes? If it's only sometimes, that would imply the invalid IL is generated at runtime.

Json.NET, which 6.x uses internally, will use Reflection.Emit to generate IL at runtime if the platform supports Reflection.Emit:

but Json.Net has supported this for a long time so I don't think this is at fault, and I don't believe any other client dependencies use Reflection.Emit.

Are you aware of any dependencies that you have that may instrument at runtime? For example, I believe ApplicationInsights instruments at runtime using the Profiling APIs.

Our system is immutable and on Fargate so I do not believe I am able to get a dotnet-dump. This does not happen all the time. We had it happen in another service as well. It has only ever happened with the SearchAsync call in the Nest client. I have attempted to move to 6.8.5.

It seems to go away on a fresh deploy. This service is not worked on very often so it tends to happen more here.

We use .Net trace from Datadog for APM. I have a ticket in with them to see if they are causing the issue. This was the stacktrace from Nov 2019. This has not happened since then. It may be them but I saw the one post on here and figured I would try all avenues to make sure I get the resolution. You can see their clrprofiler at the bottom. That is not in the new stacktrace and this is the only time we saw it with their stuff in it.

System.InvalidProgramException: Common Language Runtime detected an invalid program.
System.InvalidProgramException: Common Language Runtime detected an invalid program.
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.DoRequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.SearchAsync[TResponse](String index, PostData body, SearchRequestParameters requestParameters, CancellationToken ctx)
   at Nest.LowLevelDispatch.SearchDispatchAsync[TResponse](IRequest`1 p, SerializableData`1 body, CancellationToken ct)
   at Nest.ElasticClient.SearchAsyncb__563_0[T,TResult](ISearchRequest p, SerializableData`1 d, CancellationToken c)
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.DispatchAsync[TRequest,TQueryString,TResponse,TResponseInterface](TRequest request, CancellationToken cancellationToken, Func`3 responseGenerator, Func`4 dispatch)
   at App.Core.Management.Search.Firms.ElasticsearchManagementFirmSearchManager.SearchAsync(User user, SearchRequest request, CancellationToken cancellationToken) in /src/App/Core/Management/Search/Firms/ElasticsearchManagementFirmSearchManager.cs:line 69
   at App.Controllers.V1.Management.SearchController.Firms(SearchRequest model, CancellationToken cancellationToken) in /src/App/Controllers/V1/Management/SearchController.cs:line 36
   at lambda_method(Closure , Object )
   at Microsoft.Extensions.Internal.ObjectMethodExecutorAwaitable.Awaiter.GetResult()
   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ExceptionContext context)
   at Rethrow(Object )
   at Datadog.Trace.ClrProfiler.Integrations.AspNetCoreMvc2Integration.Rethrow(Object context, Int32 opCode, Int32 mdToken)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Rethrow(Object )
   at Datadog.Trace.ClrProfiler.Integrations.AspNetCoreMvc2Integration.Rethrow(Object context, Int32 opCode, Int32 mdToken)

Not sure if this matters but when we moved from 2.* to 3.1 .dotnet core we made sure to keep newtonsoft(it was breaking some of our endpoints)

Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson

services.AddMvc().AddNewtonsoftJson();

services.AddControllers().AddNewtonsoftJson();

If I do a fresh deploy this goes away for awhile but keeps coming back.
We are using https://github.com/DataDog/dd-trace-dotnet/releases/tag/v1.16.1 . They claim it is not them.

We are on <PackageReference Include="NEST" Version="6.8.5" />

Started happening again in the past week. Still confused on what is causing it. Do you know what the resolution to this was? Nest 6.6 getting InvalidProgramException sometimes

Here is a snippet of the code failing:

public async Task<SearchDocument[]> GetAsync(BenchmarkGetQuery<string> query,
            CancellationToken cancellationToken)
        {
            var response = await _elasticClient.SearchAsync<SearchDocument>(s => s
                    .Size(100)
                    .Query(q => BuildTermQuery(Infer.Field<SearchDocument>(f => f.SourceBenchmarkId.Suffix("keyword")), query.SourceBenchmarkId)
                    ).Sort(sort => sort.Descending(p => p.AsOf)), cancellationToken
            ).ConfigureAwait(false);

            return response.Documents.ToArray();
        }

private TermQuery BuildTermQuery<T>(Field field, T value)
        {
            return new TermQuery
            {
                Field = field,
                Value = value
            };
        }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.