Marvel: sudden errors with index_stats type

poohsen · May 12, 2014, 10:13am

Hi,

I'm sort of new to ES and currently evaluating how well it will work for
storing and querying our log data.
I installed Marvel and it worked fine for a while but now I'm suddenly
getting weird errors and the index stats are gone from Kibana:

[2014-05-09 13:35:43,035][ERROR][marvel.agent.exporter ] [Bushmaster] create failure (index:[.marvel-2014.05.09] type: [index_stats]): MapperParsingException[failed to parse [index]]; nested: MapperParsingException[failed to parse date field [.marvel-2014.05.08], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: ".marvel-2014.05.08"];
[2014-05-09 13:35:43,037][ERROR][marvel.agent.exporter ] [Bushmaster] create failure (index:[.marvel-2014.05.09] type: [index_stats]): MapperParsingException[failed to parse [index]]; nested: MapperParsingException[failed to parse date field [.marvel-2014.05.09], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: ".marvel-2014.05.09"];

I created a gist detailing the steps I took so far trying to understand/debug what's going on. While debugging, I realized that I'm unable to recreate a marvel index with the same mapping that it "normally" has (either created by marvel or by ES automapping). I detailed this in the gist, too:

gist.github.com

https://gist.github.com/poohsen/a3d3bb319010bf0c5648

gistfile1.txt

all of a sudden, at 2:00 in the night, I started getting these errors:

[2014-05-09 13:35:43,032][DEBUG][action.bulk              ] [Bushmaster] [.marvel-2014.05.09][0] failed to execute bulk item (index) index {[.marvel-2014.05.09][index_stats][zToUMDvuSxS0wVAPKB236A], source[{"@timestamp":"2014-05-09T11:35:43.024Z","cluster_name":"elasticsearch","index":".marvel-2014.05.09","_source_node":{"id":"62b9fpf0TTC8D3VH8NWgFg","name":"Bushmaster","transport_address":"inet[79.99.186.108/79.99.186.108:9300]","ip":"79.99.186.108","host":"79.99.186.108","ip_port":"79.99.186.108:9300","master_node":true,"data_node":true,"master":true},"primaries":{"docs":{"count":3368,"deleted":0},"store":{"size_in_bytes":16965332,"throttle_time_in_millis":73833},"indexing":{"index_total":3369,"index_time_in_millis":7681,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":73,"query_time_in_millis":179,"query_current":0,"fetch_total":71,"fetch_time_in_millis":23,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":83,"total_time_in_millis":138969,"total_docs":142155,"total_size_in_bytes":460876713},"refresh":{"total":901,"total_time_in_millis":89815},"flush":{"total":3,"total_time_in_millis":975},"warmer":{"current":0,"total":832,"total_time_in_millis":365},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":1,"memory_in_bytes":112423},"translog":{"operations":701,"size_in_bytes":1172594}},"total":{"docs":{"count":3368,"deleted":0},"store":{"size_in_bytes":16965332,"throttle_time_in_millis":73833},"indexing":{"index_total":3369,"index_time_in_millis":7681,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":73,"query_time_in_millis":179,"query_current":0,"fetch_total":71,"fetch_time_in_millis":23,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":83,"total_time_in_millis":138969,"total_docs":142155,"total_size_in_bytes":460876713},"refresh":{"total":901,"total_time_in_millis":89815},"flush":{"total":3,"total_time_in_millis":975},"warmer":{"current":0,"total":832,"total_time_in_millis":365},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":1,"memory_in_bytes":112423},"translog":{"operations":701,"size_in_bytes":1172594}}}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [index]
	at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418)
	at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616)
	at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:469)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
	at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:363)

This file has been truncated. show original

Since I didn't temper with Marvel indices to begin with, I don't really see how it suddenly could've gone wrong other than because of a bug.

Any hints are appreciated.

Regards,

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/61d54877-c232-4a50-ae5d-301316fcc19a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.