We are using Jaeger with ELK stack in Azure. Recently updated the Jaeger version to 1.52 and Elasticsearch version to 8.11. After upgrade the Jaeger UI is not showing any services not even the Jaeger services. But we can search key word like "trace ID" etc through global search in Jaeger UI , its show the correct result/traceability.
we extracted error from jaeger-query and found http request from jaeger-query to ES failing with 400, given below:
Error: {
"caller": "app/http_handler.go:493",
"error": "search services failed: elastic: Error 400 (Bad Request): all shards failed [type=search_phase_execution_exception]",
"level": "error",
"msg": "HTTP handler, Internal Server Error",
"stacktrace": "github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:493\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:164\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.traceResponseHandler.func2\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/http_handler.go:537\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleFunc.WithRouteTag.func3\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:285\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:229\ngo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1\n\tgo.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/handler.go:81\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.1/mux.go:212\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.additionalHeadersHandler.func4\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/additional_headers_handler.go:28\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/jaegertracing/jaeger/cmd/query/app.createHTTPServer.CompressHandler.CompressHandlerLevel.func6\n\tgithub.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2136\ngithub.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\tgithub.com/gorilla/handlers@v1.5.1/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009",
"ts": 1709657320.93561 }
Please provide the your suggestion how can I debug this to find the http request from jaeger-query to Elasticsearch to find the solution.