# Raw twitter river date format **URL:** \ **Category:** Elasticsearch\ **Created:** [September 13, 2015, 7:44pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201 "2015-09-13T19:44:50Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Chinch\_Pokli](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Chinch\_Pokli](https://discuss.elastic.co/u/Chinch_Pokli)\ **Post date:** [September 13, 2015, 7:44pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/1 "2015-09-13T19:44:50Z")
Hi, I am trying to index the raw Twitter data. Twitter returns the date in the format "Sun Sep 13 19:12:56 +0000 2015" and for that I have PUT mapping for my index with date as format "EEE MMM dd HH:mm:ss +ZZZZZ yyyy". But I am getting the error: ``` at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:621) at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:549) at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:235) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:406) ... 12 more Caused by: java.lang.IllegalArgumentException: Invalid format: "Sun Sep 13 19:12:56 +0000 2015" is malformed at "0000 2015" at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754) at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:615) ... 15 more [2015-09-13 15:12:56,866][WARN][river.twitter] There was failures while executing bulk ``` I also tried "EEE MMM dd HH:mm:ss ZZZZZ yyyy", "date", "dateOptionalTime" but nothing works. Been stuck on this for a while now! I'd appreciate any help. Thanks! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 13, 2015, 9:35pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/2 "2015-09-13T21:35:12Z")
Don't use the river as they will no longer be supported in the very near future, switch to using Logstash with the twitter input. ---
**Author:** ![Chinch\_Pokli](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Chinch\_Pokli](https://discuss.elastic.co/u/Chinch_Pokli)\ **Post date:** [September 14, 2015, 1:06pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/3 "2015-09-14T13:06:05Z")
Regardless of whether I use the river or Logstash, the question is still the same - what should be the mapping to successfully ingest the date returned by Twitter. In case of choosing the truncated data as is default, this is taken care of. However, when I want the entire stream ("raw" or "full\_tweet" option), the date is ingested as a string. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [September 14, 2015, 1:32pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/4 "2015-09-14T13:32:31Z")
I successfully used the river before and it uses this elasticsearch mapping: ``` { "my_twitter_river": { "mappings": { "status": { "properties": { "created_at": { "type": "date", "format": "dateOptionalTime" }.... ``` The river uses Twitter4J. Twitter4J returns a java.util.Date [1] so perhaps the question should be more about how Twitter4J parses whatever Twitter provides in its API? [1] [https://github.com/elastic/elasticsearch-river-twitter/blob/master/src/main/java/org/elasticsearch/river/twitter/TwitterRiver.java#L640](https://github.com/elastic/elasticsearch-river-twitter/blob/master/src/main/java/org/elasticsearch/river/twitter/TwitterRiver.java#L640) ---
**Author:** ![Chinch\_Pokli](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Chinch\_Pokli](https://discuss.elastic.co/u/Chinch_Pokli)\ **Post date:** [September 14, 2015, 4:20pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/5 "2015-09-14T16:20:47Z")
Thanks Mark. I tried "dateOptionalTime", "date\_optional\_time", "EEE MMM dd HH:mm:ss ZZZZZ yyyy", "EEE MMM dd HH:mm:ss +ZZZZ yyyy", "date" but nothing works!! I also looked at the code for Twitter4J and seems like they use "EEE MMM dd HH:mm:ss z yyyy" but even that didn't work! I've been trying all these for 2 days now but nothing has worked so far. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [September 14, 2015, 4:54pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/6 "2015-09-14T16:54:35Z")
I think the difference is Twitter4J uses in-built Java date parsing while we use joda which supports timezones differently [1] If you are not using the Twitter river/Twitter4J/java date parsing call stack and working with raw Twitter records you run into this discrepancy over timezones. [1] [http://stackoverflow.com/questions/4498274/why-joda-datetimeformatter-cannot-parse-timezone-names-z](http://stackoverflow.com/questions/4498274/why-joda-datetimeformatter-cannot-parse-timezone-names-z) ---
**Author:** ![Chinch\_Pokli](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Chinch\_Pokli](https://discuss.elastic.co/u/Chinch_Pokli)\ **Post date:** [September 14, 2015, 5:53pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/7 "2015-09-14T17:53:17Z")
Ok so it seem to be indexing now after I used the mapping as "EEE MMM dd HH:mm:ss +0000 yyyy". Now one question I have is - will I need to search in the same format? That would be useless, right? I tried searching for "2015-09-14" in the field but it gave me an error ``` "error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[5mltrJcdSMmx-qQon8_m7A][twittertool_untruncated][0]: SearchParseException[[twittertool_untruncated][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{\n \"query\": {\n \"match\": {\n \"created_at\": \"2015-09-14\"\n }\n }\n}\n]]]; nested: ElasticsearchParseException[failed to parse date field [2015-09-14], tried both date format [EEE MMM dd HH:mm:ss +0000 yyyy], and timestamp number]; nested: IllegalArgumentException[Invalid format: \"2015-09-14\"]; } ``` ---
**Author:** ![kontoudi](https://avatars.discourse-cdn.com/v4/letter/k/2bfe46/32.png) [@kontoudi](https://discuss.elastic.co/u/kontoudi)\ **Post date:** [September 22, 2015, 8:29am UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/8 "2015-09-22T08:29:50Z")
I have the same problem as you and nothing works. I tried the mapping you posted but it does not work either. Can you post you entire mapping so I can check that we are doing the same thing? Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:48pm UTC](https://discuss.elastic.co/t/raw-twitter-river-date-format/29201/9 "2017-07-05T23:48:50Z")