Nest 6.8.3 getting InvalidProgramException sometimes

Not sure if this matters but when we moved from 2.* to 3.1 .dotnet core we made sure to keep newtonsoft(it was breaking some of our endpoints)

Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson

services.AddMvc().AddNewtonsoftJson();

services.AddControllers().AddNewtonsoftJson();

If I do a fresh deploy this goes away for awhile but keeps coming back.
We are using https://github.com/DataDog/dd-trace-dotnet/releases/tag/v1.16.1 . They claim it is not them.

We are on <PackageReference Include="NEST" Version="6.8.5" />

Started happening again in the past week. Still confused on what is causing it. Do you know what the resolution to this was? Nest 6.6 getting InvalidProgramException sometimes