Is it possible to query app-search documents through elastic search API?

Elastic Search
1

I have self-hosted app search instance with the Elastic stack (ELK+Appsearch). I can't see app-search index in Kibana. I'm wondering if there is any other way to manage the documents app-search is working on apart from app-search API? Can I use Logstash to keep app-search documents in sync with my SQL Server database?

2

Hi @Ali_Modern :wave:

Have you looked into the App Search output plugin: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elastic_app_search.html?

3

Hey @orhantoy!

Thanks for providing that. That is a helpful plugin to feed new/updated documents into App Search.
However, we're trying to keep it fully synced with a master data source. Documents updated/removed/created in master need to be reflected in app-search. So, we're looking for a full CRUD capability.

Create/Update -> DONE. API or Logstash plugin. We're able to create as many as we want.
Read -> Limited. You can only list up to 10k (not to mention you need 100 requests of getting 100 documents each time). In our use case, we have 14k.
Delete -> Blocked. We can't delete what we can't list in the first place. Once created, a document after 10k becomes a ghost that only appears in search results.

That said, when we were looking into alternatives to the official App Search API which was obviously designed for front-end experience (explains paging), we were looking into alternatives that work better with machines. E.g. API that could return the entire dataset in one request somehow.

That led us to think we could access the documents from the bottom; Elastic search instance that presumably stores the documents (and doesn't show it as an index in Kibana). So to summarize, the question is:

Is it possible to manage (CRUD) app-search documents from anywhere else other than its official API?

4

I'm having similar challenges using the App Search Document API. App search hides the underlying index 'somehow' with the 'system indices'. You can find it via https://localhost:5601/app/kibana#/management/elasticsearch/index_management/indices and select the 'include system indices' option. To find it more quickly knowing how many documents it may have could help, but searching with the term 'search-engine' will give you a list of all App Search managed search engines/indices. This is a convention observed and not an API and has the caveats that come with such embedded knowledge/observation. Any manipulation of the underlying index MAY cause App Search to misbehave. Note: I'm just a user of this system and don't know the inner workings!

It seems App Search doesn't take advantage of index 'alias' feature of Elasticsearch for index rebuilding, so it's probable that it creates a new index in Elasticsearch upon 'settings' change etc. This was gleaned from the dialog presented when poking around with Kibana 'lifecycle policy' feature:
Add lifecycle policy to ".ent-search-engine-5f29af24d9eadc7a567c6d11"
Lifecycle policy

kibana-event-log-policy
Index has no aliases
Policy kibana-event-log-policy is configured for rollover, but index .ent-search-engine-5f29af24d9eadc7a567c6d11 does not have an alias, which is required for rollover.
5

After going through a few Schema changes via the App Search UI at http://localhost:3002/as#/engines/my-engine-name/schema/edit I observed the addition of mappings but no removal. Also underlying Elasticsearch index was unchanged.

Changing the created_at schema type from text to dateand then from date to text produced the following mappings entries (from ui at https://localhost:5601/app/kibana#/management/elasticsearch/index_management/indices) :

      "created_at$date": {
          "type": "date"
        },
        "created_at$string": {
          "type": "text",
          "index_options": "offsets",
          "fields": {
            "delimiter": {
              "type": "text",
              "term_vector": "with_positions_offsets",
              "index_options": "offsets",
              "analyzer": "iq_text_delimiter",
              "position_increment_gap": 100
            },
            "enum": {
              "type": "keyword",
              "ignore_above": 2048
            },
            "intragram": {
              "type": "text",
              "index_options": "docs",
              "analyzer": "iq_intragram"
            },
            "joined": {
              "type": "text",
              "term_vector": "with_positions_offsets",
              "index_options": "offsets",
              "analyzer": "iq_text_bigram",
              "position_increment_gap": 100
            },
            "prefix": {
              "type": "text",
              "term_vector": "with_positions_offsets",
              "index_options": "docs",
              "analyzer": "i_prefix",
              "search_analyzer": "q_prefix"
            },
            "stem": {
              "type": "text",
              "term_vector": "with_positions_offsets",
              "index_options": "offsets",
              "analyzer": "iq_text_stem",
              "position_increment_gap": 100
            }
          },
          "analyzer": "iq_text_base",
          "position_increment_gap": 100
        }
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

7

Hey all - we've recently introduced new functionality on two fronts that will certainly help with the use case at hand:

  1. A way to use existing Elasticsearch indices with App Search. You can still create and managed indices (via engines) with App Search, but this gives you more flexibility on where and how the data is indexed. If some restaurants are bring your own wine, App Search is now bring your own Elasticsearch index :slight_smile:

  2. A way to use the Elasticsearch Query DSL for any App Search engine. This means even the most demanding of queries can be issued against App Search engines, with all the automatic goodness of Analytics capture, for example.

Hope this helps you all on this thread, and perhaps any other peer that stumbles upon this thread!

-- Jonas