Sending request to one index, writing to multiple indices

Elastic Stack Elasticsearch
21

Can you please quote some issues specifically that it can cause eventually. As I am not understanding why this can cause problem and why will it not be considered as a feature request.

22

In Elasticsearch each indexing request will either succeed or fail. There is no concept of partial success or failure. If you allowed a single indexing request to write to multiple indices, you could end up with one succeeding and one failing, which would break this fundamental principle. This could be due to the user not having privileges to write to one of the indices or that one of the indices is not available for writing due to various issues. Adding something like this is therfore most likely not straightforward.

This is something I would recommend handling client side or by introducing a custom proxy that can handle this logic as well as errors that may occur.

1 Like
23

But in my case, I am well aware that both the indices are writable and both the indices are present.
Thus I think that for my use case I can add something like this.
Although I am having hard time implementing the same. I had tried the following:

  • Firstly sending a request with the processor and then again sending it without processors or vice versa. But I am time and again facing the issue stating that the channel is already closed.
  • To overcome this I created two channels in AbstractHttpServerTransport.java and run the dispatchRequest method twice in handleIncomingRequest but it shows some thread timer issue.
  • I tried hard coding the http Request at the entry point of the which according to my learning is channelRead0 method in Netty4HttpRequestHandler (Correct me if I am wrong). What I tried doing is updating the incoming HttpRequest to /_bulk request and updated the request body accordingly but this is also breaking the code.
  • Running evaluatePipelines twice once with processors and once without also don't work because this method only modifies our coming request body, it is not actually responsible for indexing as per my learnings of the codebase.

I am sure some of the above approaches I mentioned are not good at all and will probably break a lot of things but I have mentioned it just to be clear what I have tried.
I would really appreciate if someone can suggest some way I can do the same. Also can someone tell which class is the entry point for any request that we send. Is it Netty4HttpRequestHandler as I mentioned earlier ?

24

I would, as I have stated earlier, recommend handling this outside of Elasticsearch. If you create a custom build of Elasticsearch you will likely not be able to get much help here as any issues you encounter may very well be unique to you and not reproducable on a standard Elasticsearch build.

I am not familiar with the code base so can not help with your proposed changes will leave this thread.

25

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.