# Elasticsearch-Hadoop formatting multi resouce writes issue

**URL:** <https://discuss.elastic.co/t/elasticsearch-hadoop-formatting-multi-resouce-writes-issue/122503>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [March 5, 2018, 12:08pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-formatting-multi-resouce-writes-issue/122503 "2018-03-05T12:08:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![petroslamb](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@petroslamb](https://discuss.elastic.co/u/petroslamb)\
**Post date:** [March 5, 2018, 12:08pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-formatting-multi-resouce-writes-issue/122503/1 "2018-03-05T12:08:31Z")

</div>

I am interfacing Elasticsearch with Spark, using the Elasticsearch-Hadoop plugin and I am having difficulty writing a dataframe with a `timestamp` type column to Elasticsearch.

The problem is when I try to write using dynamic/multi resource formatting to create a daily index.

From the [relevant documentation](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#cfg-multi-writes-format) I get the impression that this is possible, however, the python example below fails to run unless I change my dataframe type to `date`.

```
import pyspark
conf = pyspark.SparkConf()
conf.set('spark.jars', 'elasticsearch-spark-20_2.11-6.1.2.jar')
conf.set('es.nodes', '127.0.0.1:9200')
conf.set('es.read.metadata', 'true')
conf.set('es.nodes.wan.only', 'true')
from datetime import datetime, timedelta
now = datetime.now()
before = now - timedelta(days=1)
after = now + timedelta(days=1)
cols = ['idz', 'name', 'time']
vals = [(0,'maria', before), (1, 'lolis', after)]  
time_df = spark.createDataFrame(vals, cols)

```

When I try to write, I use the following:

```
time_df.write.mode('append').format(
    'org.elasticsearch.spark.sql'
).options(
    **{'es.write.operation': 'index' }
).save('xxx-{time|yyyy.MM.dd}/1')

```

Unfortunatelly this renders an error:

> .... Caused by: java.lang.IllegalArgumentException: Invalid format:  
> "2018-03-04 12:36:12.949897" is malformed at " 12:36:12.949897" at  
> org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945)

On the other hand this works perfectly fine if I use dates when I create my dataframe:

```
cols = ['idz', 'name', 'time']
vals = [(0,'maria', before.date()), (1, 'lolis', after.date())]  
time_df = spark.createDataFrame(vals, cols)

```

Is it possible to format a dataframe `timestamp` to be written to daily indexes with this method, without also keeping a `date` column around? How about monthly indexes?

> Pyspark version:  
> spark version 2.2.1  
> Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0\_151

> Elasticsearch version   
> number "6.2.2" build\_hash "10b1edd"  
> build\_date "2018-02-16T19:01:30.685723Z" build\_snapshot false  
> lucene\_version "7.2.1" minimum\_wire\_compatibility\_version "5.6.0"  
> minimum\_index\_compatibility\_version "5.0.0"

Mirror question on SO:

> <https://stackoverflow.com/questions/49110054/elasticsearch-hadoop-formatting-multi-resource-writes-issue>

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [March 16, 2018, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-formatting-multi-resouce-writes-issue/122503/2 "2018-03-16T23:59:23Z")

</div>

> "2018-03-04 12:36:12.949897" is malformed at " 12:36:12.949897"

Instead of `"2018-03-04 12:36:12.949897"` try using the format of `"2018-03-04T12:36:12"`. ES-Hadoop has to do quite of a bit of type related gymnastics because it supports many integrations that do not have schemas attached, especially when it comes to dates and timestamps. I suspect the timestamp format for python is just different enough that it trips it up.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2018, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-hadoop-formatting-multi-resouce-writes-issue/122503/3 "2018-04-13T23:59:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.