Search From Multiple Indexes

Elastic Stack Elasticsearch
1

Is there a way to search records from multiple indexes?

Here is requirement, we are receiving 200 GB data per day, Also, i need to delete data based day wise,
so i am planing to build index on daily bases, so i can easily remove entire index.

if user search for 1 week of data, i need to search data from 7 different Indexes.
How can i search from multiple Indexes?

Also, if i delete data from existing records, will index rebuild automatically or I need to do it manually?

Regards,
Gaurav Patel

2

Just use index aliases. That's very convenient.
Or search in foo-* that will work as well.

3

Thanks a lot @dadoonet for quick reply, I am new to elasticsearch and with .net background. Could you please provide any reference or sample of it to go through. it will help me lot.

4

Searching in multiple indices. Imagine this scenario:

DELETE foo-1,foo-2,foo-3
POST foo-1/_doc
{
  "foo": "bar"
}
POST foo-2/_doc
{
  "foo": "bar"
}
POST foo-3/_doc
{
  "foo": "bar"
}

To search within all indices, run:

GET /_search

To search in indices starting with foo-, run:

GET foo-*/_search

About index aliases, read: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

5

Thank you very much @dadoonet, for your quick response and support. Rally appreciate.

6

Hi @dadoonet

Below Indexes are created as per guidance and data is inserted into it.

e.g
logs-2018-5-25,

timestamp": "2018-05-25T01:24:18.595Z
SourceId: 1
Data:"Sample Data"

logs-2018-5-28,
timestamp": "2018-05-28T01:24:18.595Z
SourceId: 1
Data:"Sample Data"

logs-2018-5-31
timestamp": "2018-05-31T01:24:18.595Z
SourceId: 1
Data:"Sample Data"

If i want to search records for date "2018-05-25" and "2018-05-28", Is their any way i can set those value into "Routing" options of "SearchRequest". or is their any other way so that instead of looking into all available Indexes under Aliases?

7

Add a date range query in your query.

8

Hi, @dadoonet,

I added date into filers, see below code. Even though its going to search in all indexes instead of two. See attached image.

var queryForm = new TermQuery
{
Field = "extendedData.name",
Value = "bothnull1"
};

        var rangeQuery = new DateRangeQuery
        {
            Field = "timestamp",
            GreaterThanOrEqualTo = new DateTime(2018, 05, 25),
            LessThanOrEqualTo = new DateTime(2018, 06, 28)
        };

        
        SearchRequest requ = new SearchRequest("callogs-*");            
        requ.Query = (queryForm && rangeQuery);

image

9

Yes. Is that a problem? Specifically if you add that within a filter clause of a bool query.

10

I mean that if you really want to set on which indices you want to work, you should do something like:

SearchRequest requ = new SearchRequest("callogs-2018-05-25","callogs-2018-05-26", ...);

Or use date math like: https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html

11

Thanks a lot @dadoonet for your quick response and guidance. let me go through it.

Rally appreciate your support and help.

12

Hi @dadoonet

Need your guidance for some points.

  1. Can i pass sql query to Search() method of ElasticClient?

    Like "Select * from callogs where ...." to Search() method of ElasticClient? I am using "NEST" package for
    my .net applicaton?

  2. My application provide dynamically creation facility, application convert user criteria into sql server WHERE condition runtime , is their way i can convert sql WHERE condition into Elastic compatible

  3. What parameter i need to pass to NOT convert filed to lower case while creating index?
    e.g Public Class Logs{

public string AppUserName{get;set;}
public string AssemblyName{get;set;}

}

When i pass object of Employee for indexing it convert "AppUserName" to "appUserName" and "AssemblyName" to "assemblyName", how can i ignore converting property name to lower case.

  1. I read some where that Elastic provide queuing facility, it is true? can you please provide me some detail of it.
13

@Gaurav_Patel

rather than add additional questions onto a specific question about multiple indices, it would be better to open a new question(s); they'll get more visibility as new questions, and it'll also be easier for others to find the answers if they have similar questions in the future.

14

Sure Sir, let me post as separate questions.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.