Controlling date field in attachment type

Hi,

Am using attachment mapper plugin and having issues with the date formats
for few html pages.

The exception am getting when I index a html file as an attachment is

org.elasticsearch.index.mapper.MapperParsingException: Failed to parse
[content.date]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.attachment.AttachmentMapper.parse(AttachmentMapper.java:327)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:587)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:486)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:430)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:297)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:211)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to
parse date field [Wed Jun 05 13:36:17 PDT 2013], tried both date format
[dateOptionalTime], and timestamp number
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:432)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:361)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:182)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Wed Jun 05
13:36:17 PDT 2013"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:426)
... 15 more

The mapping looks like this (curl localhost:9200/examples/files/_mapping)

{
"files": {
"properties": {
"content": {
"fields": {
"author": {
"type": "string"
},
"content": {
"include_in_all": true,
"store": "yes",
"term_vector": "with_positions_offsets",
"type": "string"
},
"content_type": {
"type": "string"
},
"date": {
"format": "dateOptionalTime",
"type": "date"
},
"keywords": {
"type": "string"
},
"name": {
"type": "string"
},
"title": {
"type": "string"
}
},
"path": "full",
"type": "attachment"
}
}
}
}

I tried setting date format explicitly and mapping after this change looks
like this

{
"files": {
"properties": {
"content": {
"fields": {
"author": {
"type": "string"
},
"content": {
"include_in_all": true,
"store": "yes",
"term_vector": "with_positions_offsets",
"type": "string"
},
"content_type": {
"type": "string"
},
"date": {

  •                    "format": "EEE MMM dd HH:mm:ss z yyyy",*
                      "type": "date"
                  },
                  "keywords": {
                      "type": "string"
                  },
                  "name": {
                      "type": "string"
                  },
                  "title": {
                      "type": "string"
                  }
              },
              "path": "full",
              "type": "attachment"
          }
      }
    
    }
    }

Even then its throwing up the same above exception.

  1. I checked if the format give is wrong, but doesn't seem like. Why is it
    not honouring the format given in mapping ?
  2. Is there a way to change the type of the date field in attachment type
    to string (given that I dont have any querying needs on that field)

Appreciate your thoughts on handling this.

-katta

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

We get somehow the same question on the french mailing list with another field for which the explicit mapping seems to be overwrote by mapper attachment.
I'll give a look into this in the next days.

Thanks!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 25 juin 2013 à 12:00, Srivatsa Katta vatsa.katta@gmail.com a écrit :

Hi,

Am using attachment mapper plugin and having issues with the date formats for few html pages.

The exception am getting when I index a html file as an attachment is

org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [content.date]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at org.elasticsearch.index.mapper.attachment.AttachmentMapper.parse(AttachmentMapper.java:327)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:587)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:486)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:430)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:297)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:211)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [Wed Jun 05 13:36:17 PDT 2013], tried both date format [dateOptionalTime], and timestamp number
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:432)
at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:361)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:182)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Wed Jun 05 13:36:17 PDT 2013"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:426)
... 15 more

The mapping looks like this (curl localhost:9200/examples/files/_mapping)

{
"files": {
"properties": {
"content": {
"fields": {
"author": {
"type": "string"
},
"content": {
"include_in_all": true,
"store": "yes",
"term_vector": "with_positions_offsets",
"type": "string"
},
"content_type": {
"type": "string"
},
"date": {
"format": "dateOptionalTime",
"type": "date"
},
"keywords": {
"type": "string"
},
"name": {
"type": "string"
},
"title": {
"type": "string"
}
},
"path": "full",
"type": "attachment"
}
}
}
}

I tried setting date format explicitly and mapping after this change looks like this

{
"files": {
"properties": {
"content": {
"fields": {
"author": {
"type": "string"
},
"content": {
"include_in_all": true,
"store": "yes",
"term_vector": "with_positions_offsets",
"type": "string"
},
"content_type": {
"type": "string"
},
"date": {
"format": "EEE MMM dd HH:mm:ss z yyyy",
"type": "date"
},
"keywords": {
"type": "string"
},
"name": {
"type": "string"
},
"title": {
"type": "string"
}
},
"path": "full",
"type": "attachment"
}
}
}
}

Even then its throwing up the same above exception.

  1. I checked if the format give is wrong, but doesn't seem like. Why is it not honouring the format given in mapping ?
  2. Is there a way to change the type of the date field in attachment type to string (given that I dont have any querying needs on that field)

Appreciate your thoughts on handling this.

-katta

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks David,

That will really help.

-katta

On Tue, Jun 25, 2013 at 9:38 PM, David Pilato david@pilato.fr wrote:

We get somehow the same question on the french mailing list with another
field for which the explicit mapping seems to be overwrote by mapper
attachment.
I'll give a look into this in the next days.

Thanks!

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 25 juin 2013 à 12:00, Srivatsa Katta vatsa.katta@gmail.com a écrit :

Hi,

Am using attachment mapper plugin and having issues with the date formats
for few html pages.

The exception am getting when I index a html file as an attachment is

org.elasticsearch.index.mapper.MapperParsingException: Failed to parse
[content.date]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.attachment.AttachmentMapper.parse(AttachmentMapper.java:327)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:587)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:486)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:430)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:297)
at
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:211)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed
to parse date field [Wed Jun 05 13:36:17 PDT 2013], tried both date format
[dateOptionalTime], and timestamp number
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:432)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:361)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:182)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Wed Jun 05
13:36:17 PDT 2013"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:426)
... 15 more

The mapping looks like this (curl localhost:9200/examples/files/_mapping)

{
"files": {
"properties": {
"content": {
"fields": {
"author": {
"type": "string"
},
"content": {
"include_in_all": true,
"store": "yes",
"term_vector": "with_positions_offsets",
"type": "string"
},
"content_type": {
"type": "string"
},
"date": {
"format": "dateOptionalTime",
"type": "date"
},
"keywords": {
"type": "string"
},
"name": {
"type": "string"
},
"title": {
"type": "string"
}
},
"path": "full",
"type": "attachment"
}
}
}
}

I tried setting date format explicitly and mapping after this change looks
like this

{
"files": {
"properties": {
"content": {
"fields": {
"author": {
"type": "string"
},
"content": {
"include_in_all": true,
"store": "yes",
"term_vector": "with_positions_offsets",
"type": "string"
},
"content_type": {
"type": "string"
},
"date": {

  •                    "format": "EEE MMM dd HH:mm:ss z yyyy",*
                      "type": "date"
                  },
                  "keywords": {
                      "type": "string"
                  },
                  "name": {
                      "type": "string"
                  },
                  "title": {
                      "type": "string"
                  }
              },
              "path": "full",
              "type": "attachment"
          }
      }
    
    }
    }

Even then its throwing up the same above exception.

  1. I checked if the format give is wrong, but doesn't seem like. Why is it
    not honouring the format given in mapping ?
  2. Is there a way to change the type of the date field in attachment type
    to string (given that I dont have any querying needs on that field)

Appreciate your thoughts on handling this.

-katta

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.