I have created an Elasticsearch instanced where we are place log data from a web site that includes information about when people book appointments, and I am having problems with a field that holds the start time of the appointment.
The first problem is, occasionally we are getting data parsed through logstash without a start time and this is causing this error:
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [appointmentDetails.startDateTime] //snip// Caused by: java.lang.IllegalArgumentException: Invalid format: ""
I understand that this is because we are passing nothing to the field that is expecting a timestamp, I think that I can put the ignore_malform option in the index so it won't fail. However I am creating a new index on a daily basis so I think I need to create a template, but I can't quite figure out how to edit the existing logstash template, or if that is even advisable. Should I be creating a new template?
The second problem (and I can happily create a separate discussion point if need be), is I am getting
Courier Fetch: 5 of 5 shards failed. when searching in Kibana and this is correlating with the following errors in the elasticsearch logs:
org.elasticsearch.transport.RemoteTransportException: [ElasticServer][<IPAddress>:9300][indices:data/read/search[phase/fetch/id]] Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [appointmentDetails.startDateTime] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.
I am not sure what is happening here, are the two related?
Any pointers here would be greatly appreciated.