Can't get highlighted results using Java High Level Rest Client

I am using Java 1.8 and Elasticsearch 7.12.0.
Hello, I am trying to get Highlighter to work but all i'm getting is the whole document in which word is available and I should get only the snippets of the highlighted words from that documents.

My Java code looks like this.

	@Override
	public SearchResponse getHighlights(String value) throws IOException {
		SearchRequest searchRequest = new SearchRequest();// 1
		searchRequest.indices("indexx");

		// creating match query
		MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("Content", value);
		// creating searchSourceBuilder
		SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
		// creating HighlightBuilder
		HighlightBuilder highlightBuilder = new HighlightBuilder();
		// Create a field highlighter for the Content field
		HighlightBuilder.Field highlightContent = new HighlightBuilder.Field("Content");
		// set field highlighter type
		highlightContent.highlighterType("unified");
		highlightBuilder.field(highlightContent);

		searchSourceBuilder.query(matchQueryBuilder);

		searchRequest.source(searchSourceBuilder);
		
		return client.search(searchRequest, RequestOptions.DEFAULT);
	}

I think I'm not passing the highlightBuilder into searchRequest. I am trying to figure it out how to pass it.

Hi!!

Try:

searchSourceBuilder.highlighter(highlightBuilder);

I tried this, and when I run the query I get all the documents available in my index. I think I need to pass the matchQueryBuilder as well and I also tried searchSourceBuilder.highlighter(highlightBuilder).query(matchQueryBuilder);
To which I get the documents in which the keyword is present, I just need the highlighted snippet of the matching word.

On my postman I get the whole document as output

{
    "internalResponse": {
        "numReducePhases": 1,
        "fragment": true
    },
    "scrollId": null,
    "totalShards": 1,
    "successfulShards": 1,
    "skippedShards": 0,
    "shardFailures": [],
    "clusters": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "fragment": true
    },
    "hits": {
        "hits": [
            {
                "score": 3.4118958,
                "id": "qKtq24ABRnv1ozaZXmfP",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "highlightFields": {
                    "Content": {
                        "name": "Content",
                        "fragments": [
                            {
                                "fragment": true
                            },
                            {
                                "fragment": true
                            },
                            {
                                "fragment": true
                            }
                        ],
                        "fragment": true
                    }
                },
                "sortValues": [],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "indexx",
                "clusterAlias": null,
                "sourceAsMap": {
                    "@timestamp": "2022-05-19 13:53:42.93",
                    "Content": "India’s honourable Prime Minister Shri Narendra Modi bought himself a new car, and we are more excited about it than he can ever be. Now, people like the Prime Minister or Presidents may or may not have many friends, but they definitely have a lot of enemies, and this new toy from Mercedes does its best to sabotage all of their plans. This is not just any bulletproof car, and it raises the bar higher than ever for safety standards pertaining to maximum security. Here are ten interesting facts about the Mercedes-Maybach S650 Guard that will blow your mind.\nMercedes-Maybach S650 Guard\nIt costs more than Rs.12 Crores, but the actual price is not revealed.\nVR10-Level Protection is the highest ever protection in a production vehicle.\nCan withstand an explosive force of up to 15kg of TNT from a distance of just 2 metres.\nBody shell and four-inch-thick windows get polycarbonate coating on the inside capable of withstanding armour-piercing bullets.\n6.0-litre Twin-Turbo V12 produces 523 hp of power and 830 Nm of torque and a top speed electronically limited to 190 km/h.\nIt has a heavily-armoured underbody to protect its occupants from explosions caused by land mines.\nIt runs on Michelin PAX run-flat tires that allow the vehicle to run as much as 30 kms after going flat.\nIt gets a separate oxygen tank in case of a gas attack. The fuel tank is made of similar material used by Boeing for its AH-64 Apache helicopters and receives a special coating to instantly seal holes after a bullet hit.Get your car serviced at GoMechanic.\nThe rest of the car remains a regular Maybach with the pinnacle of luxury ranging from powered massager seats, multi-zone climate control and more.\nSpecial Protection Group (SPG) is responsible for protecting the country’s head of state. The SPG identifies the security requirements and then decides if the dignified person they are protecting requires a vehicle upgrade or not. The SPG orders two identical cars, and the second vehicle is used as a decoy.\nVisually it is almost indistinguishable from a regular Maybach unless you start to observe the details, which is a neat trick to deceive the uninitiated. PM Narendra Modi was first seen arriving in this tank of a car at the Hyderabad House when he met Russian President Vladimir Putin on December 6. Previously, it was the Range Rover Vogue and Toyota Land Cruiser that was safeguarding the Prime Minister since he assumed office while he had a customised Scorpio and Bolero while he was the Chief Minister of Gujarat.\n\n",
                    "Name": "Maybach S-650 Guard"
                },
                "innerHits": null,
                "fields": {},
                "sourceAsString": "{\"@timestamp\":\"2022-05-19 13:53:42.93\",\"Content\":\"India’s honourable Prime Minister Shri Narendra Modi bought himself a new car, and we are more excited about it than he can ever be. Now, people like the Prime Minister or Presidents may or may not have many friends, but they definitely have a lot of enemies, and this new toy from Mercedes does its best to sabotage all of their plans. This is not just any bulletproof car, and it raises the bar higher than ever for safety standards pertaining to maximum security. Here are ten interesting facts about the Mercedes-Maybach S650 Guard that will blow your mind.\\nMercedes-Maybach S650 Guard\\nIt costs more than Rs.12 Crores, but the actual price is not revealed.\\nVR10-Level Protection is the highest ever protection in a production vehicle.\\nCan withstand an explosive force of up to 15kg of TNT from a distance of just 2 metres.\\nBody shell and four-inch-thick windows get polycarbonate coating on the inside capable of withstanding armour-piercing bullets.\\n6.0-litre Twin-Turbo V12 produces 523 hp of power and 830 Nm of torque and a top speed electronically limited to 190 km/h.\\nIt has a heavily-armoured underbody to protect its occupants from explosions caused by land mines.\\nIt runs on Michelin PAX run-flat tires that allow the vehicle to run as much as 30 kms after going flat.\\nIt gets a separate oxygen tank in case of a gas attack. The fuel tank is made of similar material used by Boeing for its AH-64 Apache helicopters and receives a special coating to instantly seal holes after a bullet hit.Get your car serviced at GoMechanic.\\nThe rest of the car remains a regular Maybach with the pinnacle of luxury ranging from powered massager seats, multi-zone climate control and more.\\nSpecial Protection Group (SPG) is responsible for protecting the country’s head of state. The SPG identifies the security requirements and then decides if the dignified person they are protecting requires a vehicle upgrade or not. The SPG orders two identical cars, and the second vehicle is used as a decoy.\\nVisually it is almost indistinguishable from a regular Maybach unless you start to observe the details, which is a neat trick to deceive the uninitiated. PM Narendra Modi was first seen arriving in this tank of a car at the Hyderabad House when he met Russian President Vladimir Putin on December 6. Previously, it was the Range Rover Vogue and Toyota Land Cruiser that was safeguarding the Prime Minister since he assumed office while he had a customised Scorpio and Bolero while he was the Chief Minister of Gujarat.\\n\\n\",\"Name\":\"Maybach S-650 Guard\"}",
                "sourceRef": {
                    "fragment": true
                },
                "rawSortValues": [],
                "fragment": false
            }
        ],
        "totalHits": {
            "value": 1,
            "relation": "EQUAL_TO"
        },
        "maxScore": 3.4118958,
        "sortFields": null,
        "collapseField": null,
        "collapseValues": null,
        "fragment": true
    },
    "failedShards": 0,
    "timedOut": false,
    "numReducePhases": 1,
    "terminatedEarly": null,
    "aggregations": null,
    "suggest": null,
    "took": {
        "seconds": 0,
        "hours": 0,
        "minutes": 0,
        "millis": 7,
        "days": 0,
        "stringRep": "7ms",
        "microsFrac": 7000.0,
        "millisFrac": 7.0,
        "micros": 7000,
        "nanos": 7000000,
        "daysFrac": 8.101851851851852E-8,
        "hoursFrac": 1.9444444444444444E-6,
        "secondsFrac": 0.007,
        "minutesFrac": 1.1666666666666667E-4
    },
    "profileResults": {},
    "fragment": false
}

Heey?

can you send a doc or mapping as an example?

Sure!
This is the mapping of my index named indexx

{
  "indexx" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Content" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "timestamp" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

I think I understand.
Were you just using "searchSourceBuilder.highlighter(highlightBuilder)"?
I used your code and got this result:

{
   "took":425,
   "timed_out":false,
   "_shards":{
      "total":1,
      "successful":1,
      "skipped":0,
      "failed":0
   },
   "hits":{
      "total":{
         "value":1,
         "relation":"eq"
      },
      "max_score":0.2879492,
      "hits":[
         {
            "_index":"ngram_custom_example",
            "_type":"_doc",
            "_id":"PRS18IABHFVsAeBm-4jT",
            "_score":0.2879492,
            "_ignored":[
               "Content.keyword"
            ],
            "_source":{
               "@timestamp":"2022-05-19 13:53:42.93",
               "Content":"India’s honourable Prime Minister Shri Narendra Modi bought himself a new car, and we are more excited about it than he can ever be.... long text",
               "Name":"Maybach S-650 Guard"
            },
            "highlight":{
               "Content":[
                  "India’s <em>honourable</em> Prime Minister Shri Narendra Modi bought himself a new car, and we are more excited"
               ]
            }
         }
      ]
   }
}

Code Java:

RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new 
HttpHost("localhost", 9200)));

SearchRequest searchRequest = new SearchRequest();// 1
searchRequest.indices("your_index");

// creating match query
MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("Content", "honourable");
// creating searchSourceBuilder
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
// creating HighlightBuilder
HighlightBuilder highlightBuilder = new HighlightBuilder();
// Create a field highlighter for the Content field
HighlightBuilder.Field highlightContent = new HighlightBuilder.Field("Content");
// set field highlighter type
highlightContent.highlighterType("unified");
highlightBuilder.field(highlightContent);

searchSourceBuilder.query(matchQueryBuilder);
searchSourceBuilder.highlighter(highlightBuilder);
searchRequest.source(searchSourceBuilder);

SearchResponse result = client.search(searchRequest, RequestOptions.DEFAULT);

I have a question, what is var and getClient() here? because im getting The method getClient() is undefined for the type EsImpl<Session>.
I'm using RestHighLevelClient.

I edited my answer. I use Java 11.

Oh! ok, I already tried this and i'm getting this response on postman

{
    "internalResponse": {
        "numReducePhases": 1,
        "fragment": true
    },
    "scrollId": null,
    "totalShards": 1,
    "successfulShards": 1,
    "skippedShards": 0,
    "shardFailures": [],
    "clusters": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "fragment": true
    },
    "hits": {
        "hits": [
            {
                "score": 2.4564419,
                "id": "qKtq24ABRnv1ozaZXmfP",
                "type": "_doc",
                "nestedIdentity": null,
                "version": -1,
                "seqNo": -2,
                "primaryTerm": 0,
                "highlightFields": {
                    "Content": {
                        "name": "Content",
                        "fragments": [
                            {
                                "fragment": true
                            }
                        ],
                        "fragment": true
                    }
                },
                "sortValues": [],
                "matchedQueries": [],
                "explanation": null,
                "shard": null,
                "index": "indexx",
                "clusterAlias": null,
                "sourceAsMap": {
                    "@timestamp": "2022-05-19 13:53:42.93",
                    "Content": "India’s honourable Prime Minister Shri Narendra Modi bought himself a new car, and we are more excited about it than he can ever be. Now, p..................long text............ Chief Minister of Gujarat.\n\n",
                    "Name": "Maybach S-650 Guard"
                },
                "innerHits": null,
                "fields": {},
                "sourceAsString": "{\"@timestamp\":\"2022-05-19 13:53:42.93\",\"Content\":\"India’s honourable Prime Minister Shri Narendra Modi bought himself a new car, and we are more excited about it than he can ever be. Now, p..................long text............ Chief Minister of Gujarat.\\n\\n\",\"Name\":\"Maybach S-650 Guard\"}",
                "rawSortValues": [],
                "sourceRef": {
                    "fragment": true
                },
                "fragment": false
            }
        ],
        "totalHits": {
            "value": 1,
            "relation": "EQUAL_TO"
        },
        "maxScore": 2.4564419,
        "sortFields": null,
        "collapseField": null,
        "collapseValues": null,
        "fragment": true
    },
    "terminatedEarly": null,
    "numReducePhases": 1,
    "timedOut": false,
    "failedShards": 0,
    "profileResults": {},
    "suggest": null,
    "aggregations": null,
    "took": {
        "hours": 0,
        "minutes": 0,
        "millis": 1,
        "seconds": 0,
        "days": 0,
        "stringRep": "1ms",
        "nanos": 1000000,
        "micros": 1000,
        "microsFrac": 1000.0,
        "millisFrac": 1.0,
        "secondsFrac": 0.001,
        "minutesFrac": 1.6666666666666667E-5,
        "daysFrac": 1.1574074074074074E-8,
        "hoursFrac": 2.7777777777777776E-7
    },
    "fragment": false
}

Note: I have created client as RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(new HttpHost("localhost", 9200, "http"))); which is working fine in all other methods.
Java code :

	@Override
	public SearchResponse getHighlights(String value) throws IOException {
		SearchRequest searchRequest = new SearchRequest();// 1
		searchRequest.indices("indexx");

		// creating match query
		MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("Content", value);
		// creating searchSourceBuilder
		SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
		// creating HighlightBuilder
		HighlightBuilder highlightBuilder = new HighlightBuilder();
		// Create a field highlighter for the Content field
		HighlightBuilder.Field highlightContent = new HighlightBuilder.Field("Content");
		// set field highlighter type
		highlightContent.highlighterType("unified");
		highlightBuilder.field(highlightContent);

		searchSourceBuilder.query(matchQueryBuilder);
		searchSourceBuilder.highlighter(highlightBuilder);
		searchRequest.source(searchSourceBuilder);

		return client.search(searchRequest, RequestOptions.DEFAULT);

	}

Im getting this at frontend

clusterAlias: null
explanation: null
fields: {}
fragment: false
highlightFields:
Content:
fragment: true
fragments: (3) [{…}, {…}, {…}]
name: "Content"
[[Prototype]]: Object
[[Prototype]]: Object
id: "1p3e9IABHhH3X-PY_aPi"
index: "indexx"
innerHits: null
matchedQueries: []
nestedIdentity: null
primaryTerm: 0
rawSortValues: []
score: 0.45207185
seqNo: -2
shard: null
sortValues: []
sourceAsMap: {Content: 'Tesla Motors was formed to develop an electric spo…itial public offering raised some $226 million.\n\n', timestamp: '2022-05-24 12:31:34.343', Name: 'Origin of Tesla'}
sourceAsString: "{\"Content\":\"Tesla Motors was formed to develop an electric sports car. Eberhard was Tesla’s chief executive officer (CEO) and Tarpenning its chief financial officer (CFO). Funding for the company was obtained from a variety of sources, most notably PayPal cofounder Elon Musk, who contributed more than $30 million to the new venture and served as chairman of the company, beginning in 2004.\\nIn 2008 Tesla Motors released its first car, the completely electric Roadster. In company tests, it achieved 245 miles (394 km) on a single charge, a range unprecedented for a production electric car. Additional tests showed that its performance was comparable to that of many gasoline-powered sports cars: the Roadster could accelerate from 0 to 60 miles (96 km) per hour in less than four seconds and could reach a top speed of 125 miles (200 km) per hour. The lightweight car body was made of carbon fibre. The Roadster produced no tailpipe emissions, as it did not use an internal-combustion engine. Tesla Motors found that the car attained efficiency ratings that were equivalent to a gasoline mileage of 135 miles per gallon (57 km per litre). The vehicle’s electric motor was powered by lithium-ion cells—often used in laptop-computer batteries—that could be recharged from a standard electrical outlet. Despite a federal tax credit of $7,500 for purchasing an electric vehicle, the Roadster’s cost of $109,000 made it a luxury item.\\nIn late 2007 Eberhard resigned as CEO and president of technology and joined the advisory board of the company. It was announced in 2008 that he had left the company, though he remained a shareholder. Tarpenning, who was vice president of electrical engineering, supervising the development of electronic and software systems for the Roadster, also left the company in 2008. Musk took over as CEO. In 2010 Tesla’s initial public offering raised some $226 million.\\n\\n\",\"timestamp\":\"2022-05-24 12:31:34.343\",\"Name\":\"Origin of Tesla\"}"
sourceRef: {fragment: true}
type: "_doc"
version: -1
[[Prototype]]: Object
length: 1
[[Prototype]]: Array(0)

Please note in the fragments section I'm getting 3 arrays as the word is available in the document 3 times, and when I open the array I get this

Content:
fragment: true
fragments: Array(3)
0:
fragment: true
[[Prototype]]: Object
1: {fragment: true}
2: {fragment: true}
length: 3
[[Prototype]]: Array(0)
name: "Content"
[[Prototype]]: Object
[[Prototype]]: Object

This output is from frontend - Angular 13.

I'm sending an image of how I'm getting the response.

Note that highlightFields is a map. I noticed that you return the SearchReponse, I normally use a dto to customize the response.

Can you give a hint on how to use dto to customize the response?

This is a simple example with Java 11 and Lombok lib:

DTO

@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class ResponseDTO {

  @JsonProperty("Name")
  private String name;
  @JsonProperty("Content")
  private String content;
  @Builder.Default
  private Map<String, List<String>> highlightsMap = new HashMap<>();
}

Method

 private static List<ResponseDTO> search() throws IOException {
    ObjectMapper objectMapper = new ObjectMapper();
    objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
    List<ResponseDTO> responseDTO = new ArrayList<>();

    SearchRequest searchRequest = new SearchRequest();// 1
    searchRequest.indices("idx_highlighter");

    // creating match query
    MatchQueryBuilder matchQueryBuilder = new MatchQueryBuilder("Content", "honourable excited");
    // creating searchSourceBuilder
    SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    // creating HighlightBuilder
    HighlightBuilder highlightBuilder = new HighlightBuilder();
    // Create a field javaapi.client.highlighter for the Content field
    HighlightBuilder.Field highlightContent = new HighlightBuilder.Field("Content");
    // set field javaapi.client.highlighter type
    highlightContent.highlighterType("unified");
    highlightBuilder.field(highlightContent);

    searchSourceBuilder.query(matchQueryBuilder);
    searchSourceBuilder.highlighter(highlightBuilder);
    searchRequest.source(searchSourceBuilder);

    SearchResponse result = getClient().search(searchRequest, RequestOptions.DEFAULT);

    for (var hit : result.getHits().getHits()) {
      ResponseDTO value = objectMapper.readValue(hit.getSourceAsString(), ResponseDTO.class);
      Map<String, HighlightField> highlightFieldsMap = hit.getHighlightFields();
      if (!highlightFieldsMap.isEmpty()) {
        for (Map.Entry<String, HighlightField> entry : highlightFieldsMap.entrySet()) {
          value.getHighlightsMap()
              .put(entry.getKey(), Arrays.stream(entry.getValue().getFragments()).map(Text::toString).collect(Collectors.toList()));
        }
      }
      responseDTO.add(value);
    }

    return responseDTO;
  }

Thank you for helping me, let me check this.

This worked! Thank you for your responses.

1 Like

What if I want to send id with the data as well?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.