Creating a second index with subset of fields from first index

Elastic Stack Elasticsearch
1

Hello Elastic community,

I'm seeking advice on how to efficiently create a second index containing only a subset of fields from a primary index based on certain conditions.

To provide some context, let's say I have a primary index containing documents with 10 fields. However, I'm interested in creating a secondary index that includes only two specific fields (e.g., client IP and client MAC) from the primary index, under the condition that both of these fields exist in the document.

It's important to note that I'm unable to use Logstash for this task due to certain constraints.

Essentially, when a particular condition is met (e.g., both client IP and client MAC fields exist), I'd like to write a document to the secondary index with only these two fields.

I'd appreciate any advice or suggestions on how to approach this task efficiently using Elasticsearch.

Thank you in advance for your help and insights!

Best regards

2

Hi @yago82

Did you try with Reindex select fields with a source filter.

POST _reindex
{
  "source": {
    "index": "my-index-000001",
    "_source": ["user.id", "_doc"]
  },
  "dest": {
    "index": "my-new-index-000001"
  }
}
3

The easiest way to do that is during indexing, so this could be easily done if you were using Logstash.

Without Logstash you may be able to do that using transforms.

With transform you can create a search in discovery with your condition, and use this search as the data source for your transform.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.