Search elasticsearch with java client using JSON query

[anisen (Anirban Sen)]

Hi, I am using ElasticSearch 5.1.1

I am using the below java code to fetch data from ES :

SearchResponse response = client.prepareSearch()
.setIndices("myindex01")
.setTypes("iis-log")
.setQuery(QueryBuilders.termQuery("status", "404"))
.execute().actionGet();

But, I want to use the following JSON quey instead:

String jsonQuery = "{"query": {"term" : { "status": "404" }}}";

What code changes I need to do?

[wenpos (Fanfan)]

is javax.ws.rs.client package to publish a es url available

[anisen (Anirban Sen)]

It's an elasticsearch Transport client. You need to initialize it like below:

import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;

Settings settings = Settings.builder()
.put("cluster.name", "my_cluster").build();
TransportClient client = new PreBuiltTransportClient(settings)
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getLocalHost(),9300));

[wenpos (Fanfan)]

yeah, this is ES-java TransportClient API. is this what you want?

[anisen (Anirban Sen)]

Yes. I have used elasticsearch Transport client. But, my question was different.

I want to use the JSON query while fetching the ES db using java client. Any suggestion on that?

[wenpos (Fanfan)]

It means you want build a query JSON first? you could try this :

import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;

//{"query": {"term" : { "status": "404" }}}"
XContentBuilder query = XContentFactory.jsonBuilder();
query.startObject()
.startObject("query").startObject("term").field("status", "404").endObject()
.endObject();
System.out.println(query.string());
//{"query":{"term":{"status":"404"}}}
client.prepareSearch("index").setQuery(query).execute().actionGet();

[anisen (Anirban Sen)]

I don't want to build a json query.
I have already some json queries. I just want to use them directly.

I was using this with ES 1.4.2 and it was working:

SearchResponse response= client.prepareSearch(indicesToSearch).setSource(query)
.execute().actionGet();
// query = Query string in JSON format

But this code is no more working with ES 5.1.1. Hence looking for solutions.

[wenpos (Fanfan)]

The API has changed in 5.x like this：

//public SearchRequestBuilder setSource(SearchSourceBuilder source) {...}

SearchSourceBuilder source = new SearchSourceBuilder()
.query(termQuery("field1", "test"))
.highlighter(highlight().field("field1", 100, 0).order("score").preTags("").postTags(""));
SearchResponse searchResponse = client().prepareSearch("test").setSource(source).get();

[anisen (Anirban Sen)]

So, can't we use a JSON query string anymore in ES 5.X ?

[wenpos (Fanfan)]

currently I have not seen the relative APIs

[anisen (Anirban Sen)]

Ok..

I was trying it like this:

String jsonQuery = "{"query": {"term" : { "status": "404" }}}";
//QueryBuilder query = ?? (code to convert json string to queryBuilder object)

SearchResponse response =
client.prepareSearch()
.setIndices("myindex")
.setTypes("log-type")
.setSearchType(SearchType.QUERY_AND_FETCH)
.setQuery(query) //query is of type QueryBuilder
.execute().actionGet();

But not able to covert the json-string to QueryBuilder object properly. Can you help me on this?

[jprante (Jörg Prante)]

You can use the following code fragment:

String content = "{\"query\":{\"match_all\":{}}}";

SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
SearchModule searchModule = new SearchModule(Settings.EMPTY, false, Collections.emptyList());
try (XContentParser parser = XContentFactory.xContent(content).createParser(content)) {
    QueryParseContext context = new QueryParseContext(searchModule.getQueryParserRegistry(), parser,
            new ParseFieldMatcher(true));
    searchSourceBuilder.parseXContent(context, searchModule.getAggregatorParsers(),
            searchModule.getSuggesters(), searchModule.getSearchExtRegistry());
}

SearchRequestBuilder searchRequestBuilder = new SearchRequestBuilder(client(), SearchAction.INSTANCE);
SearchResponse searchResponse = searchRequestBuilder.setSource(searchSourceBuilder)
        .execute().actionGet();

[wenpos (Fanfan)]

will this works?
TermQueryBuilder query = QueryBuilders.termQuery("status", "404");

[anisen (Anirban Sen)]

Hi,

I used this code. I think it's fetching records for all records in ES DB. How to specify the index here?

[schulm (Martin Schulze)]

Hi Jörg,
this snippet doesn't work in ES 5.2.0. The Createparser method needs an additional NamedXContentRegistry. Also getQueryParserRegistry() doesn't exist anymore and the parseFieldMatcher is deprecated and will possibly break in the future.
I think it should be possible to query a json query with the offical api without touching the ES internals.

[jprante (Jörg Prante)]

Here is an update for current ES API on master (untested)

String content = "{\"query\":{\"match_all\":{}}}";
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
SearchModule searchModule = new SearchModule(Settings.EMPTY, false, Collections.emptyList());
try (XContentParser parser = XContentFactory.xContent(XContentType.JSON)
                .createParser(NamedXContentRegistry.EMPTY, content)) {
    searchSourceBuilder.parseXContent(new QueryParseContext(parser));
}
SearchRequestBuilder searchRequestBuilder = new SearchRequestBuilder(client, SearchAction.INSTANCE);
SearchResponse searchResponse = searchRequestBuilder.setSource(searchSourceBuilder)
            .execute().actionGet();

The idea is to peek into the source code and see how RestSearchAction works. RestSearchAction must always translate an external JSON string into ES SearchRequest Java API. Unwrapping the code paths around RestSearchAction, and following a pattern where XContentParser runs over the JSON string and passes the result to the SearchSourceBuilder method that accepts XContent should do the trick. The example is of course a simplification and can not handle all real-world queries.

Your mileage may vary if you need to deal also with custom query/aggregations plugins, they have to be registered via SearchModule, and plugins might bring in other complications. For instance, "ext": { ... } won't work, see SearchExtBuilder. The ES developers have added some data binding features to XContent API, but I assume NamedXContentRegistry.EMPTY is safe as long as only ordinary JSON-encoded queries are parsed.

[cbuescher (Christoph)]

@jprante

createParser(NamedXContentRegistry.EMPTY, content)

Unfortunately NamedXContentRegistry.EMPTY doesn't work for parsing QueryBuilders without registering the built-in queries first, but they should be available through searchModule.getNamedXContents(). So to create the parser in the above example you can use

try (XContentParser parser = XContentFactory.xContent(XContentType.JSON)
                        .createParser(new NamedXContentRegistry(searchModule.getNamedXContents()), content)) {
            searchSourceBuilder.parseXContent(new QueryParseContext(parser));
}

[javanna (Luca Cavanna)]

I don't understand what the need is for using the java api, yet sending json requests that are manually handled. The java api helps sending request and composing them using its own objects. I would recommend to move over to the low level rest client for these cases where json is sent instead.

Query parsing was moved to the coordinating node with 5.0 and the java api must now send parsed objects. I wouldn't recommend doing the parsing on the client side, that is a work-around that may not work in the future. The on thing that should work though is using WrapperQueryBuilder, which allows to provide a query (not the whole search request, only the query part of it) as a string, and its parsing is actually performed on the data nodes rather than on the coordinating node.