What You'll Learn From This Read
- How to add the
semantic_textfield type in an index before ingestion. - Index data against this field for efficient search and retrieval.
- Extract chunks and perform a completion task to generate human-friendly summaries.
What's Included
- Wikipedia Extractor Script: A Python notebook to extract content from Wikipedia articles for indexing.
Prerequisites
- AWS Bedrock: Access and Secret Key
- Tasks: You need to create two tasks - Embedding and Completion Task.
Objective
By the end of this read, you’ll be able to:
- Index and query data using the
semantic_textfield type. - Utilize Elastic Playground to explore the features interactively with AWS Bedrock(used here), OpenAI, Gemini, and Azure OpenAI.
Getting Started
Detailed Tutorial Link: Semantic Text with Amazon Bedrock
The tutorial covers queries to:
- Set up
embedding and completion tasksusing different models tailored to your requirements. - Make use of
Elastic Cloud v8.15.0+for quick setup and access to the Playground feature.
Environment Setup
Use Elastic Cloud v8.15.0+ for Playground access.
Quick Testing
1. Create an Index
Define a semantic_text field and associate it with an embedding task.
PUT semantic-text-bedrock-3
{
"mappings": {
"properties": {
"super_body": {
"type": "semantic_text",
/* Change this model_id based on your embedding model name */
"inference_id": "my-inference-endpoint-v3"
}
}
}
}
2. Ingest a Sample Document that we can extract from the Wikipedia Extractor script
POST semantic-text-bedrock-3/_doc
{
"super_body": "Introduction to Food is the prime necessity of life. The food we eat is digested and assimilated in the body, providing energy for maintenance, growth, and work. Over time, humans have cultivated various grains, fruits, vegetables, nuts, and oilseeds, and have reared animals for food. The optimal selection of foods for health has been discovered through trial and error. Today, there is a wealth of information available on the nutritional value of food and dietary requirements.Scope and Importance:Nutrition and dietetics is an essential and expanding field with a wide range of opportunities. Employment in this field can be found in hospitals, health and recreation clubs, canteens, nursing care facilities, and governmental departments. Dietitians and nutritionists are also involved in food services like restaurants and catering services, where they plan menus and oversee food preparation. In the healthcare sector, dietitians create diet plans for patients and monitor the results of dietary interventions. Research and development play a crucial role in ensuring food quality and studying the effects of various diets on human health. Nutritionists are also key to food manufacturing, product development, and ensuring the nutritional value of food products. Concepts and Significance: Our understanding of human nutrition has advanced greatly and plays a crucial role in preventing diseases. Conditions like pellagra, beriberi, and rickets, once common, can now be prevented with proper nutrition. While overt nutritional deficiencies are rare in affluent populations, undernutrition can still be found among vulnerable groups like the elderly, children, and those on restricted diets. In impoverished populations, deficiencies in protein, vitamins, and minerals are more widespread, leading to conditions such as kwashiorkor, vitamin A deficiency, and anemia. Functions of Food: Food provides energy, supports growth, repairs damaged tissue, and regulates bodily processes. It also plays a central role in social, cultural, and religious life, symbolizing love, friendship, and community during gatherings. Psychologically, food satisfies emotional needs, offering a sense of security, love, and comfort. It is often associated with positive feelings and memories, making it a key part of celebrations and family life. Classification of Food:Food can be classified into three broad categories. Energy-yielding foods are rich in carbohydrates and fats, providing the energy needed for daily activities and bodily functions. Body-building foods, such as those rich in protein, help in the growth and repair of body tissues. Protective foods, rich in vitamins, minerals, and proteins, regulate bodily processes and maintain overall health. These foods help in maintaining body temperature, muscle contraction, water balance, and more. Balanced Diet:A balanced diet provides the necessary types and amounts of food to maintain health and support normal growth. It includes all the nutrients the body needs, such as carbohydrates, proteins, fats, vitamins, and minerals, to support the body’s functions and prevent diseases. A proper diet, combined with regular physical activity, is key to achieving a long and healthy life. It helps control body weight, maintains heart rate and blood pressure, improves muscle performance, and contributes to mental and physical well-being. This text has been fully formatted into paragraphs without any lists."
}
3. Query the Field
Run a semantic query to find related content:
GET semantic-text-bedrock-3/_search
{
"query": {
"semantic": {
"field": "super_body",
"query": "what is food?"
}
}
}
Or, increase the complexity of the query to get the chunks:
GET semantic-text-bedrock-3/_search
{
"_source": false,
"retriever": {
"standard": {
"query": {
"nested": {
"path": "super_body.inference.chunks",
"query": {
"knn": {
"field": "super_body.inference.chunks.embeddings",
"query_vector_builder": {
"text_embedding": {
/* Change this model_id based on your embedding model name */
"model_id": "my-inference-endpoint-v3",
"model_text": "what is food?"
}
}
}
},
"inner_hits": {
"size": 1,
"name": "semantic-text-bedrock-3.super_body",
"_source": "*.text"
}
}
}
}
}
}
4. Enhance with Completion Tasks
Retrieve chunks and pass them into a completion model for an abstract summary. This is manual process, so we need to copy the chunk from the above query and pass it into the completion model.
# bedrock-completion-v3 - Completion Task/Model
POST _inference/completion/bedrock-completion-v3
{
"input": """
Answer the question:\n
_what is food?_
, based on the following context: \n
and development play a crucial role in ensuring food quality and studying the effects of various diets on human health. Nutritionists are also key to food manufacturing, product development, and ensuring the nutritional value of food products. Concepts and Significance: Our understanding of human nutrition has advanced greatly and plays a crucial role in preventing diseases. Conditions like pellagra, beriberi, and rickets, once common, can now be prevented with proper nutrition. While overt nutritional deficiencies are rare in affluent populations, undernutrition can still be found among vulnerable groups like the elderly, children, and those on restricted diets. In impoverished populations, deficiencies in protein, vitamins, and minerals are more widespread, leading to conditions such as kwashiorkor, vitamin A deficiency, and anemia. Functions of Food: Food provides energy, supports growth, repairs damaged tissue, and regulates bodily processes. It also plays a central role in social, cultural, and religious life, symbolizing love, friendship, and community during gatherings. Psychologically, food satisfies emotional needs, offering a sense of security, love, and comfort. It is often associated with positive feelings and memories, making it a key part of celebrations and family life. Classification of Food:Food can be classified into three broad categories. Energy-yielding foods are rich in carbohydrates and fats, providing the energy needed for daily activities and bodily functions. Body-building foods, such as those rich in protein, help in the growth and repair of body tissues. Protective foods, rich in vitamins, minerals, and proteins, regulate bodily processes and maintain overall health. These foods
"""
}
Example output:
{
"completion": [
{
"result": """Based on the context provided, the answer to the question "What is food?" can be summarized as follows:
Food is a vital component that plays a crucial role in ensuring human health and well-being. It is a multifaceted concept with the following key aspects:
1. Nutritional Significance: Food provides the necessary nutrients, such as carbohydrates, fats, proteins, vitamins, and minerals, that are essential for growth, development, tissue repair, and the regulation of bodily processes. Proper nutrition is crucial in preventing various nutritional deficiency diseases.
2. Functional Aspects: Food serves as a source of energy for daily activities and bodily functions. It supports the growth and repair of body tissues, and it also plays a central role in regulating various physiological processes.
3. Social, Cultural, and Psychological Significance: Food holds a significant place in social, cultural, and religious life, often symbolizing love, friendship, and community during gatherings. It also satisfies emotional needs, providing a sense of security, comfort, and positive associations.
4. Classification: Food can be broadly classified into three categories: energy-yielding foods (rich in carbohydrates and fats), body-building foods (rich in proteins), and protective foods (rich in vitamins, minerals, and proteins).
In summary, food is a multifaceted concept that encompasses the nutritional, functional, social, cultural, and psychological aspects of human sustenance and well-being."""
}
]
}
Resources
- Wikipedia Extractor: Python Notebook
Using the Playground Feature
Now, that all the heavy-lifting and the nuances is performed above, we can perform the same using the Playground feature in a few clicks.
- Navigate to Enterprise Search/Elastic Search > Playground.
- Connect AWS Bedrock with Access and Secret Keys.
- Experiment with your data and tune the queries.