Best practice for Engine Lifecycle Management in App Search

Elastic Search
1

I´m using App Search to create engine, write and read documents. So no direct interaction with Elasticsearch at all.

The question is what would be the best practice to define "Engine Lifecycle Management"? I can´t see that there is anything like this in the App Search Web UI, neither in the App Search REST API. So I guess I need to define ILM in Elasticsearch? Since App Search is responsible for creation of the Elasticsearch index, corresponding to the Engine, it would be useful to get input on how this ILM shall be defined.

Any documentation or suggestions on this?

2

App Search doesn't assume any lifecycle for engines. If there is one, it should be managed by the engines API. The product leans on ILM for its logs and analytics, though. You can see more here in the docs: https://www.elastic.co/guide/en/enterprise-search/current/log-retention.html

3

Can you please elaborate on this one? I cant see any ILM related details in the Engine API.

My point is that App Search create index in Elasticsearch corresponding to the Engine user has created in App Search. I assume that this index has none ILM defined and that can (most probably will) cause issues over time. Is my understanding and assumption valid? If so, what is the recommended solution for this, when using App Search for creation of indexes?

4

Engines (basically a container for content and settings) are assumed to be around forever. Time-based data like an engine's analytics and logs are managed by ILM. If an engine needs to be deleted, that needs to be done via the Engines API.

It sounds like you may have a specific use-case or concern in mind, as generally engines do not have lifecycles and it does not cause issues. Is there something specific you are worried about or experiencing?

5

It might be that I'm planning to use the App Search for the wrong use case. So let me explain what my plan is so it is more clear.

High level data flow:
Messages on Kafka Topic -- read -> Kotlin app -- write -> App Search -- read -> Search and visualization of data in Web App

Since our Kafka Topic contains real time data and is frequently updated, let say 100 messages per second, we want to keep the data in App Search for 30 days only.

Im little bit surprised when you say that "Engines are assumed to be around forever" by design. What use cases have you considered in this design decision?

6

Elasticsearch may be better suited for this use-case. App Search is tailored for use-cases with a corpus like articles or products rather than time-based data that is phased in and out. It can certainly work for what you describe though. You could, for instance, create an engine per day and delete it after 30 days (via the API).

See this for more about the intent of the product: https://www.elastic.co/app-search/

7

Well, Elasticsearch is already in use (behind the scenes) when using App Search, and App Search is just a facade over Elasticsearch. So Im little bit confused that App Search sets some limits to its use cases when underlying core component (Elasticsearch) already provides support for handling these concerns.

The benefits I see using App Search in this use case is:

  • It can work as backend-for-frontend when building new web app where you want to search and read data from Elasticsearch, but don't want to have direct access from Web App to Elasticsearch

  • It provides out of box React/UI components so you don't need to reinvent the wheel in your Web App for searching, displaying results and talk with App Search

  • It has useful Admin Web UI so you can easily get more insight of the usage of your data

I'm sorry If I sound rude, but I just don't understand how you can assume that an Engine can be around for ever, no matter use case, without defining properly ILM in Elasticsearch.

Regarding creating Engine per day: I would not consider this as feasible solution at all. Because you would basically build ILM functionallity by hand (reinventing the wheel) and you will be limited to search documents per day, since basic license limit search per engine

8

Thanks for clarifying!

Engines are assumed to be permanent-ish objects in App Search because App Search is opinionated in its use cases, which generally includes common search uses for applications and websites. Of course, that still covers a lot of ground, but it does have some limits – when those limits don't work for some users, Elasticsearch is a great choice.

But the upside of being opinionated in use case is that App Search comes with optimizations out-of-the-box that give results that are relevant and easily tuneable, robust analytics, and much more. In short, App Search is much more than just a facade for Elasticsearch, the same way that a forklift is much more than just a golf cart in a costume – it can do a lot of the heavy lifting for you :wink: .

To your specific use case, if you'd prefer to cycle indices via ILM, then Elasticsearch is a great choice! If you'd prefer not to cycle through engines, another option is to write a controller that deletes documents in a persistent engine (you could search / delete for docs older than a specific date, for example).

Hope that helps!

1 Like
9

Thanks clarifying answer, @nickchow! I wish this information about what use cases App Search is suited for and its boundaries were more clearly documented.

Last question: what would be the downside of defining manually an ILM in Elasticsearch of the engine's index? I understand that this is not recommanded solution and can break in upcoming releases, but that could be the easiest solution on short term.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.