# Dec 17th, 2021: \[EN\] Getting Started with the Elasticsearch Python Client **URL:** \ **Category:** Advent Calendar\ **Created:** [December 17, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-17th-2021-en-getting-started-with-the-elasticsearch-python-client/290535 "2021-12-17T08:00:52Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Claire\_Burn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/claire_burn/32/97835_2.png) [@Claire\_Burn](https://discuss.elastic.co/u/Claire_Burn)\ **Post date:** [December 17, 2021, 8:00am UTC](https://discuss.elastic.co/t/dec-17th-2021-en-getting-started-with-the-elasticsearch-python-client/290535/1 "2021-12-17T08:00:52Z")
# Getting Started With The Elasticsearch Python Client ## An Introduction To Elasticsearch With Python Elastic provides its own client for most languages, including of course - Python! There are two official Elasticsearch clients for the Python language: - **[elasticsearch-dsl](https://elasticsearch-dsl.readthedocs.io/en/latest/)**: A very user-friendly Python client. It is written in Python, and is built on top of the **elasticsearch-py** client, in order to abstract some of the lower-level functionality. - **[elasticsearch-py](https://elasticsearch-py.readthedocs.io/en/master/)**: This is a lower-level elasticsearch-py client, which gives the user more fine-grained control of their API calls and query management. We’re going to take a deeper look at **elasticsearch-py** in this post. Both clients are open source and very extensible, so if you’re a keen Python developer and see a feature you need that’s missing, don’t hesitate to get involved! ## Installing The Elasticsearch Python Client The Elasticsearch client package can be installed with the **pip** package manager: `$ python -m pip install elasticsearch` The Elasticsearch Python client also supports asynchronous coding patterns, which you can make use of by installing the client with the **async** extra: `$ python -m pip install elasticsearch[async]` ## Getting A Connection To Elasticsearch From Python The Elasticsearch client class takes many configurable parameters, from `http_auth` arguments, to certificates. There are many and varied ways to authenticate to Elasticsearch, depending on whether you are attempting to connect to an Elastic Cloud instance, or a self-hosted one. By far the simplest way of connecting is the `http_auth` argument, which simply takes a tuple of **(username, password)**. For more on **authenticating** to Elasticsearch using the elastic-search-py client, see the [documentation here](https://www.elastic.co/guide/en/elasticsearch/client/python-api/master/connecting.html). **An example of an Elasticsearch connection object in Python:** ```auto from elasticsearch import Elasticsearch def setup_es_conn() -> Elasticsearch: """Sets up a connection to the Elasticsearch Cluster.""" return Elasticsearch( 'https://elasticsearch.url:port', http_auth=('elastic','yourpassword'), timeout=60, max_retries=2, retry_on_timeout=True, ) ``` ## Querying Lots Of Data Using `helpers.scan` The `helpers` classes of the **elasticsearch-py** client provide some simple and helpful (as the name might suggest) abstraction functionality over the raw Elasticsearch APIs, such as the **bulk and scroll APIs**. The `helpers.scan` function is a simple abstraction on top of the **scroll API** - a simple iterator that yields all hits returned by underlying scroll requests. The `helpers.scan` function takes the following as required arguments: - An **Elasticsearch connection object**. - An index name to query against as a **string**. - A query as a **dict** or **json** object. **An example of querying all data in an index using `helpers.scan`:** ```auto def scan_es(): elasticsearch_conn = setup_es_conn() results = elasticsearch.helpers.scan( elasticsearch_conn, index="test_index", doc_type="my_document", preserve_order=True, query={"query": {"match_all": {}}}, ) for item in results: # parse... ``` For more on the **Scan helper** , see [the documentation here](https://elasticsearch-py.readthedocs.io/en/master/helpers.html?highlight=scan#scan). ## Bulk Insertion To Elasticsearch Using `helpers.bulk` The **bulk API** can be used to insert and query data in bulk from Elasticsearch, and so is extremely useful for handling lots of data efficiently. Using the **bulk API** with the Python client to insert data can be done by using a simple list comprehension or generator to iterate over your data, like so: ```auto from datetime import datetime from elasticsearch import Elasticsearch, ElasticsearchException, helpers def es_insert(): """Runs a query and parses results.""" elasticsearch_conn = setup_es_conn() try: documents = [ { "_index": "example_data", "_op_type": "create", "_source": { "@timestamp": datetime.now().isoformat(), "file": {"path": file_obj["filename"]} }, } for file_obj in [{"filename": "/tmp/example.txt"}] ] helpers.bulk(es, documents) except ElasticsearchException as err: ... ``` One handy tip to note: Elasticsearch **Data Streams** support only the **create** action within the `_op_type` field. Actions are one of **index** , **create** , **delete** , and **update** , and if using a regular index, the recommended `_op_type` is the **index** action, which is the default if no user input is provided for this field. For more information on the Bulk API, see [Elastic’s documentation here](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). For more information on Helpers, see [the documentation here](https://elasticsearch-py.readthedocs.io/en/master/helpers.html). Happy data wrangling, everyone - thanks for reading! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 14, 2022, 8:01am UTC](https://discuss.elastic.co/t/dec-17th-2021-en-getting-started-with-the-elasticsearch-python-client/290535/2 "2022-01-14T08:01:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.