# How to send data post data to jobs api elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [May 20, 2019, 7:44pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893 "2019-05-20T19:44:22Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ryan\_Boch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_boch/32/46500_2.png) [@Ryan\_Boch](https://discuss.elastic.co/u/Ryan_Boch)\
**Post date:** [May 20, 2019, 7:44pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893/1 "2019-05-20T19:44:22Z")

</div>

I cannot figure out how to send json data via python for [posting data to jobs](https://elasticsearch-py.readthedocs.io/en/master/xpack.html#elasticsearch.client.xpack.ml.MlClient.post_data). The [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-post-data.html#ml-post-data) is not clear on the format the json file needs to be in. I've tried many different options, but I continue getting different errors.

Here is an example of a json doc saved as file\_name.json:

```auto
[{"myid": "id1", "client": "client1", "submit_date": 1514764857},{"my_id": "id2", "client": "client_2", "submit_date": 1514764857}]

```

```auto
from elasticsearch import Elasticsearch
from elasticsearch.client.xpack import MlClient

es = elastic_connection()
es_ml = MlClient(es)

def post_training_data(directory='Training Data', file_name='file_name.json'):
        with open(os.path.join(directory, file_name), mode='r') as train_file:
            train_data = json.load(train_file)
            es_ml.post_data(job_id=job_id, body=train_data)

post_training_data()

```

```auto
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "..\train_model.py", line 218, in post_training_data
    self.es_ml.post_data(job_id=self.job_id, body=train_data)
  File "..\inc_anamoly\lib\site-packages\elasticsearch\client\utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "..\inc_anamoly\lib\site-packages\elasticsearch\client\xpack\ml.py", line 81, in post_data
    body=self._bulk_body(body))
AttributeError: 'MlClient' object has no attribute '_bulk_body'

```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 22, 2019, 4:26pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893/2 "2019-05-22T16:26:18Z")

</div>

You found and filed a bug: [https://github.com/elastic/elasticsearch-py/issues/959](https://github.com/elastic/elasticsearch-py/issues/959)

Thanks for being a great community member! 🍻

---

<div class="post-metadata">

**Author:** ![Ryan\_Boch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_boch/32/46500_2.png) [@Ryan\_Boch](https://discuss.elastic.co/u/Ryan_Boch)\
**Post date:** [May 23, 2019, 1:57pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893/3 "2019-05-23T13:57:26Z")

</div>

I'm still not clear on exactly how to get post data to jobs api to work via the python elasticsearch client. The documentation says to send the data as so: "A sequence of one or more JSON documents containing the data to be analyzed. Only whitespace characters are permitted in between the documents."

How is this possible in python? The json library only allows multiple json docs to be serialized in a comma separated list format.

After this fix is applied, the model accepts the documents and shows the correct number processed but the results do not seem accurate. Very few anomalies are detected even when sending 5 months of data (about 300,000 json docs).

---

<div class="post-metadata">

**Author:** ![Ryan\_Boch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ryan_boch/32/46500_2.png) [@Ryan\_Boch](https://discuss.elastic.co/u/Ryan_Boch)\
**Post date:** [May 23, 2019, 2:15pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893/4 "2019-05-23T14:15:14Z")

</div>

When I sent the json docs as one string no whitespace separating I got this error:

```auto
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "..\train_model.py", line 289, in post_training_data
    self.es_ml.post_data(job_id=self.job_id, body=myjsons)
  File "..\inc_anamoly\lib\site-packages\elasticsearch\client\utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "..\inc_anamoly\lib\site-packages\elasticsearch\client\xpack\ml.py", line 81, in post_data
    body=self.client._bulk_body(body))
  File "..\inc_anamoly\lib\site-packages\elasticsearch\transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "..\inc_anamoly\lib\site-packages\elasticsearch\connection\http_urllib3.py", line 186, in perform_request
    self._raise_error(response.status, raw_data)
  File "..\inc_anamoly\lib\site-packages\elasticsearch\connection\base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, 'parse_exception', 'The input JSON data is malformed.')

```

---

<div class="post-metadata">

**Author:** ![blaklaybul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blaklaybul/32/59221_2.png) [@blaklaybul](https://discuss.elastic.co/u/blaklaybul)\
**Post date:** [May 23, 2019, 6:30pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893/5 "2019-05-23T18:30:03Z")

</div>

Hi,

The `_bulk_body` method will serialize the data to the proper format expected by ML. After applying [this fix](https://github.com/elastic/elasticsearch-py/pull/962), passing a list of dicts as the `body` parameter should work:

```auto
In [56]: data
Out[56]:
[{'client': 'client1', 'my_id': 'id1', 'submit_date': 1514764857},
 {'client': 'client2', 'my_id': 'id2', 'submit_date': 1514764857}]

In [57]: es.xpack.ml.open_job('test')
Out[57]: {'opened': True}

In [58]: es.xpack.ml.post_data('test', body=data)
Out[58]:
{'bucket_count': 0,
 'earliest_record_timestamp': 1514764857,
 'empty_bucket_count': 0,
 'input_bytes': 120,
 'input_field_count': 4,
 'input_record_count': 2,
 'invalid_date_count': 0,
 'job_id': 'test',
 'last_data_time': 1558635579074,
 'latest_record_timestamp': 1514764857,
 'missing_field_count': 2,
 'out_of_order_timestamp_count': 0,
 'processed_field_count': 0,
 'processed_record_count': 2,
 'sparse_bucket_count': 0}

```

As for the results of your job, there could be any number of reasons why you are seeing those results. What type of analysis are you trying to do? and what job configuration did you create?

Also, if you haven't already, take a look at our getting started material for creating ML jobs [here](https://www.elastic.co/guide/en/elastic-stack-overview/current/ml-getting-started.html).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2019, 6:30pm UTC](https://discuss.elastic.co/t/how-to-send-data-post-data-to-jobs-api-elasticsearch/181893/6 "2019-06-20T18:30:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.