Transfer data/fields from one index to another

Elastic Stack Kibana
1

Can anybody help me moving the data from one index to another in Kibana.
I have an index "abc" with lot number of documents and say 100 fields and I want to extract 10 fields out of it and with some filter applied to remove unwanted events and place all this in another index "xyz". Is this thing possible in Kibana?
Also, can "Transforms" help me in this?

2

@Nisha2297 Thanks for your question!
You can use a _reindex API with queries to transfer data to another index with a different mapping.
There are a few handy examples in the docs that you could adapt to your use case. The document queries form part of your request body.
Please note that the mapping and settings for the new index into which you're going to "transfer" data needs to be defined up-front.

2 Likes
3

Thanks for your valuable inputs!
I explored the _reindex API and I am able to transfer data with selected fields into another index.
But my source index has a streaming data coming from Kafka and once I run the API and transfer the data into new index, the new index won't get updated afterwards with more events. I hope I am clear with my doubt. Please help me find a solution for this. It would be really helpful :slight_smile:

4

Ah, ok, I think I have an idea of what you want to do. What you're looking for is a way to continuously index the incoming data into two indices, where you want to forward some modified data from the source index into a new index.

I can think of two possible things to explore, although there might be others.

  • Setup and use ILM to automatically "forward' your data from the source to the target index. In this case, you'll first need to set up all the transforms and data extraction definitions in ingest pipelines for the target. I personally haven't tried this approach yet but conceptually it might work.
  • Alternatively, if you don't need the data as it is in the _source index, then you could try just setting up the ingest pipelines to take the raw data and transform it how you need to before indexing.

If neither of these work, maybe the elasticsearch team has ideas on how you can do what you want to. We could cross post this discussion in the elasticsearch discuss forum room or you could ask in there separately.

2 Likes
5

Thanks for your reply :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.