ThreadContext loses headers between threads

[Sergey_Ivanov]

Hello,

We're using our custom plugin for Elasticsearch and we faced the following problem after upgrade from ES 5.6.6 to 6.5.0:

Looks like ThreadContext loses headers between http_server_worker and updateTask threads.

For example, we store custom header in RestHandler like:

 @Override
public void handleRequest(RestRequest restRequest, RestChannel restChannel, NodeClient nodeClient)
  throws Exception {
    try {
      threadContext.putHeader("user", "test");      
 ......

and then we try to get this header in IndexEventListener implementation:

  @Override
  public void afterIndexCreated(IndexService indexService) {
    final String user = threadContext.getHeader("user");
....

But headers are empty.

Could you please help, is it expected behavior? I didn't find any information about these changes in change-log for update to 6.5.0 but it worked fine on ES 5.6.6.

[Sergey_Ivanov]

In addition, we found the issue which probably affected this behavior:
Use system context for cluster state update tasks (#31241)

and specifically here is this code:

final Supplier<ThreadContext.StoredContext> supplier = threadContext.newRestorableContext(false);
try (ThreadContext.StoredContext ignore = threadContext.stashContext()) {
        threadContext.markAsSystemContext();

        List<Batcher.UpdateTask> safeTasks = tasks.entrySet().stream()
            .map(e -> taskBatcher.new UpdateTask(config.priority(), source, e.getKey(), safe(e.getValue()), executor))
            .map(e -> taskBatcher.new UpdateTask(config.priority(), source, e.getKey(), safe(e.getValue(), supplier), executor))
            .collect(Collectors.toList());
        taskBatcher.submitTasks(safeTasks, config.timeout());
    } catch (EsRejectedExecutionException e) {

threadContext.stashContext() - saves old and creates new ThreadContext where we have no access to previous thread context's headers in updateTask thread.

Could you please explain, how we can obtain these headers inupdateTask thread now (after changes of this issue)?