App-search with data feed from dataset/index

Running ECK 8.2

The idea is to use data in Elasticsearch using e.g. filebeat httpjson to load Elasticsearch with data/dataset/index.

The question is, how to use this data for app-search?

New to Elasticsearch / app search and feels strange that this is so "disconnected". Using that that already exists would speed up or simplify a lot, and "it's the same product". Should be able to link the data and if a reindex is needed some kind of event might be used?

Found some posts but no solution.

Hi @frolko, I am happy to say that what you want is now already possible, albeit as a Technical Preview feature. You can read more about it in our documentation:

Also, this guide about using Elasticsearch search API instead of the Search API that's part of App Search might be of use:

Hope that helps!

Thanks. Must be something I'm missing.

I've created a http json with pagination called 'accounts'. And it goes thru a pipeline reformat the data to match search needs (e.g. 'accountType' is renamed 'account_type')

Automatically these are created

  • data stream: logs-accounts-default
  • indices: .ds-logs-accounts-default-2022.06.20-000001

Now somehow to get the indecies to be named 'search-' as a start.
If I run PUT .ds-logs-accounts-default-2022.06.20-000001/_alias/search-accounts I get the error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "The provided expressions [.ds-logs-accounts-default-2022.06.20-000001] match a backing index belonging to data stream [logs-accounts-default]. Data stream backing indices don't support aliases."
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "The provided expressions [.ds-logs-accounts-default-2022.06.20-000001] match a backing index belonging to data stream [logs-accounts-default]. Data stream backing indices don't support aliases."
  },
  "status" : 400
}

Testing the data stream 'PUT logs-accounts-default/_alias/search-accounts' works but is not selectable when trying to create 'app search engine'

Hi @frolko.

So the basic requirements are as follows:

  1. alias should start with search- prefix
  2. user should have permissions for that index and alias
  3. user needs to expand “advanced settings” when adding an engine to see this option
  4. the feature is only available in 8.2+

I tried to reproduce the situation in the following way:

  1. Created a simple index called my_index on Enterprise Search 8.2
  2. Indexed a few documents into it
  3. Created an alias called search-test
  4. Used the Advanced options to create an engine using that alias

It worked. An engine was created.
I think that what's different in your setup is the fact that your alias points to the data stream, and not an index. And the underlying index can't be aliased because it's just not supported.