App-search with data feed from dataset/index

[frolko (Fredrick)]

Running ECK 8.2

The idea is to use data in elasticsearch using e.g. filebeat httpjson to load elasticsearch with data/dataset/index.

The question is, how to use this data for app-search?

New to elasticsearch / app search and feels strange that this is so "disconnected". Using that that already exists would speed up or simplify a lot, and "it's the same product". Should be able to link the data and if a reindex is needed some kind of event might be used?

Found some posts but no solution.

Is it possible to query app-search documents through elastic search API?

I have self-hosted app search instance with the Elastic stack (ELK+Appsearch). I can't see app-search index in Kibana. I'm wondering if there is any other way to manage the documents app-search is working on apart from app-search API? Can I use Logstash to keep app-search documents in sync with my SQL Server database?

Can you hook an existing Elasticsearch index to Elastic App search

I want to integrate App Search in my project. App search runs fine out of the box. But I have an application that is running on Elasticsearch, I was wondering if I could use this existing index as a backend for App search? If No, is it because documents are indexed differently for App Search engine compared to Elasticsearch? Also, why can't I access App search documents from my elasticsearch cluster (localhost:9200)?

[maryna.cherniavska]

Hi @frolko, I am happy to say that what you want is now already possible, albeit as a Technical Preview feature. You can read more about it in our documentation:

Elasticsearch index engines (technical preview) | Elastic App Search...

Also, this guide about using Elasticsearch search API instead of the Search API that's part of App Search might be of use:

Elasticsearch search for App Search Guide | Elastic App Search Documentation...

Hope that helps!

[frolko (Fredrick)]

Thanks. Must be something I'm missing.

I've created a http json with pagination called 'accounts'. And it goes thru a pipeline reformat the data to match search needs (e.g. 'accountType' is renamed 'account_type')

Automatically these are created

• data stream: logs-accounts-default
• indices: .ds-logs-accounts-default-2022.06.20-000001

Now somehow to get the indecies to be named 'search-' as a start.
If I run PUT .ds-logs-accounts-default-2022.06.20-000001/_alias/search-accounts I get the error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "The provided expressions [.ds-logs-accounts-default-2022.06.20-000001] match a backing index belonging to data stream [logs-accounts-default]. Data stream backing indices don't support aliases."
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "The provided expressions [.ds-logs-accounts-default-2022.06.20-000001] match a backing index belonging to data stream [logs-accounts-default]. Data stream backing indices don't support aliases."
  },
  "status" : 400
}

Testing the data stream 'PUT logs-accounts-default/_alias/search-accounts' works but is not selectable when trying to create 'app search engine'

[maryna.cherniavska]

Hi @frolko.

So the basic requirements are as follows:

1. alias should start with search- prefix
2. user should have permissions for that index and alias
3. user needs to expand “advanced settings” when adding an engine to see this option
4. the feature is only available in 8.2+

I tried to reproduce the situation in the following way:

1. Created a simple index called my_index on Enterprise Search 8.2
2. Indexed a few documents into it
3. Created an alias called search-test
4. Used the Advanced options to create an engine using that alias

It worked. An engine was created.
I think that what's different in your setup is the fact that your alias points to the data stream, and not an index. And the underlying index can't be aliased because it's just not supported.