Elastic App Search and Spring Boot

[mirko.romstadt]

Hi,

we are running a Spring Boot application that is using spring-data-elasticsearch to index its models in an elasticsearch instance. This is working fine, we can index the models via ElasticsearchOperations.index(...).

In the future we would like to use Elastic App Search instead of directly indexing to Elasticsearch, but as far as I see there is no Java client yet. So for now I tried to call the Elastic App Search API with Unirest to index the documents of the application in the search engine. I am using the ElasticsearchConverter.mapObject(model) to convert our models into a org.springframework.data.elasticsearch.core.document.Document and then use this Document as the body of the Unirest call. Our models are all camelCase, so the resulting fields are camel case, too. On invoking the API call, I get "Fields can only contain lowercase letters, numbers, and underscores". We also persist the "_class" of the model, this leads to "Fields cannot have a leading underscore". My questions here are:

• Why does the Elastic App Search API not support camel case fields and fields beginning with a "_"? Why is this different to the Elasticsearch API?
• What is the best way to implement Elastic App Search within a Spring Boot application?
• Do I need to write an own converter for our models to match the field conventions of Elastic App Search?

Thanks for your help!

Mirko

[Scrilling]

Hi Mirko, I'm glad to hear you're interested in using App Search!

App Search is a solution built on top of Elasticsearch, and as such maintains a separate suite of APIs and integration points that are not always compatible across both products. Unfortunately, this means the spring-data-elasticsearch project does not currently support App Search.

You are definitely on a valid path forming up App Search API requests from within Spring Boot! Unirest should serve your needs just fine. I recommend reviewing the App Search docs for documents (distinct from Elasticsearch Document objects) along with the API reference guide. If you havn't already, I also recommend playing with the App Search sample engines through the UI to get an idea of which capabilities you'd like to integrate with in your Spring application.

To answer your specific questions,

Why does the Elastic App Search API not support camel case fields and fields beginning with a "_"? Why is this different to the Elasticsearch API?

This is likely due to the fact that App Search is built with Ruby and as such follows standard Ruby conventions as opposed to Elasticsearch which is mostly within the Java ecosystem. I appreciate the inconsistency is confusing and have taken note.

What is the best way to implement Elastic App Search within a Spring Boot application?

You are on the right path looking into the App Search APIs and forming up requests with Unirest. Because you can not leverage the classes provided by the spring-data-elasticsearch plugin, you'll be mostly working with JSON. I recommend using a JSON library with support for Spring models (like jackson, gson, or json-io) for converting your models for App Search indexing.

Do I need to write an own converter for our models to match the field conventions of Elastic App Search?

Yes, the documents formed up for App Search APIs must comply with the restrictions you mentioned, whether that's through a custom converter or annotating conversions with Spring+JSON capabilities.

I hope this helps! I've let the team know about your interest in a Java client for App Search and am filing an enhancement request internally.

[mirko.romstadt]

Hi Scrilling,

Thank you a lot for your detailed response, I really appreciate that! I will write converters for our models and use Unirest for the API calls. I can still switch to an official Java client once it is available and maybe also get rid of my converters if you support camelCase eventually

Have a great day!
Mirko

[mirko.romstadt]

Hey Scrilling,

Just FYI, I now created a very simple component class called ElasticAppSearchOperations that mimics the ElasticsearchOperations methods index() and delete(). There I am doing the Unirest calls and the mapping of my documents to a json that follows the Elastic App Search API conventions. For the mapping I am using the Jackson ObjectMapper that comes with the Spring Boot framework. The only thing I had to do is set the PropertyNamingStrategy to snake case.

For anybody who is interested, this is the ElasticAppSearchOperations (requires Spring Boot, Unirest and Lombok):

@Component
@Slf4j
public class ElasticAppSearchOperations {
    
    @Value("${elasticappsearch.protocol}")
    private String protocol;
    
    @Value("${elasticappsearch.host}")
    private String host;
    
    @Value("${elasticappsearch.port}")
    private int port;
    
    @Value("${elasticappsearch.apikey}")
    private String apikey;
    
    @Autowired
    ObjectMapper objectMapper;
    
    public void delete(String id, String engineName) {
        
        try {
            
            var elasticAppSearchURL = new URL(protocol, host, port, "/api/as/v1/engines/" + engineName + "/documents");
            
            var body = new JSONArray();
            body.put(id);
            
            Unirest.delete(elasticAppSearchURL.toString()).headers(getHeaders()).body(body).asJson().ifFailure(
                    jsonNodeHttpResponse -> log.error(
                            "Could not delete. Elastic App Search response status: {}, message: {}",
                            jsonNodeHttpResponse.getStatus(), jsonNodeHttpResponse.getBody()));
            
        } catch (MalformedURLException e) {
            
            log.error(e.getMessage(), e);
        }
    }
    
    public void index(Object document, String engineName) {
        
        try {
            
            objectMapper.setPropertyNamingStrategy(PropertyNamingStrategies.SNAKE_CASE);
            
            var elasticAppSearchURL = new URL(protocol, host, port, "/api/as/v1/engines/" + engineName + "/documents");
            
            var body = new JSONArray();
            body.put(new JSONObject(objectMapper.writeValueAsString(document)));
            
            Unirest.post(elasticAppSearchURL.toString()).headers(getHeaders()).body(body).asJson().ifFailure(
                    jsonNodeHttpResponse -> log.error(
                            "Could not index. Elastic App Search response status: {}, message: {}",
                            jsonNodeHttpResponse.getStatus(), jsonNodeHttpResponse.getBody()));
            
        } catch (MalformedURLException | JsonProcessingException e) {
            
            log.error(e.getMessage(), e);
        }
    }
    
    private Map<String, String> getHeaders() {
        
        Map<String, String> headers = new HashMap<>();
        
        headers.put("Content-Type", "application/json");
        headers.put("Authorization", "Bearer " + apikey);
        
        return headers;
    }
    
}

Don't forget to add these lines to your application.properties:

elasticappsearch.protocol=http
elasticappsearch.host=localhost
elasticappsearch.port=3002
elasticappsearch.apikey=

Then you can inject the component via @Autowire and call the methods:

public class SomeClass {

    @Autowired
    ElasticAppSearchOperations elasticAppSearchOperations;

    public void someMethod(Object yourModel) {

            elasticAppSearchOperations.index(yourModel, "engine-name");
            elasticAppSearchOperations.delete(yourModel.getId(), "engine-name");
    }

This is not perfect, but does the job for me now.

Hope that helps!
Mirko

[Scrilling]

This is great, thanks for sharing back your code Mirko!