X-opaque-id not displayed in search slow log

Elastic Stack Elasticsearch
1

Hello. First of all, I want to mention that I have read the similar posts Cannot see user "id" in slowlogs (X-Opaque-ID) and Elasticsearch X-Opaque-Id Not in application.log Exception but there are no responses there, so I'm posting my question

so, we are using Amazon ES v7.7 and are experiencing slow queries often. Both indexing and search types. I understood from the documentation that we could attach an HTTP Header X-Opaque-Id to the request that will reflect in the slow log as an id at the end of the line

I tried setting this header and our application logs confirm it's indeed attached to the request but the slow logs don't retain the id. Below is a sample log

[2021-08-20T02:32:22,691][TRACE][index.search.slowlog.query.oyyhvC_jSOa-vV6-ns6ihQ] [2d9ef1eb5e635897f792affedacc870b] [attr.mutjwian][4] took[1.8s], took_millis[1895], total_hits[973864+ hits], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[{"from":0,"size":24,"post_filter":{"bool":{"must":[{"term":{"ourId":{"value":189,"boost":1.0}}},{"bool":{"must":[{"term":{"ourAttributes.our_id.long":{"value":189,"boost":1.0}}},{"term":{"deleted":{"value":"false","boost":1.0}}},{"term":{"specialIds":{"value":6182,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"_source":{"includes":["newId"],"excludes":[]},"sort":[{"fewAttributes.modified.date":{"order":"desc","missing":"_last","unmapped_type":"keyword"}}]}], id[]

Please ask any questions for further info. Thank you!
@pgomulka tagging you here for visibility

2

could there be a proxy that is filtering that header in your system architecture?
can you confirm that X-Opaque-Id is present on a response headers?

try to turn on http tracing in elasticsearch by following this

if x-opaque-id was present on a request, it should be printed out in these logs
be careful if this is your prod env, it can be very noisy. Try this in test first

1 Like
3

@pgomulka thank you for the pointers

I'm not getting the response header corresponding to the opaque id. so, it might be dropped in between?

I tried to enable TRACE logging but I got an HTTP 401 status back. Let me check further and update here

4

hello, @pgomulka

Looks like AWS ES restricts the options we could provide while updating the cluster settings. so, I can't enable trace logging. I tried hitting requests in my local ES and ES echoes the header back

so with AWS ES, we're stuck for now. I'll try to ask a question in AWS forums and see why the VPC domain they expose doesn't passthrough the headers we send and if there's a way to override it

5

Raised a question on AWS Forums - https://forums.aws.amazon.com/thread.jspa?messageID=993728&#993728

6

FYI the aws service is not Elasticsearch, we are unable to help with that sorry to say.

7

hello, @warkolm

I'm aware of the differences between the two offerings. This problem is a restriction explicitly placed by AWS and I totally understand Elastic as a company can't solve it

thank you!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.