MapperParsingException with logstash


(Rob Bos) #1

I'm sending Windows logs, using nxlog, to a Logstash server with
Elasticsearch, and a field is failing to autodetect correctly, if I'm
reading Google searches correctly. ES is generating errors like "Failed to
parse [@fields.ErrorCode]" [1]. I'm struggling writing a template [2] to
set that field to 'string', but I don't even know for sure if ES is reading
the file, or if I've written it right. I would like a sanity check and
maybe some advice.

I've confirmed through ps that ES is getting path.conf set to
/etc/elasticsearch, and I've placed the template file as
templates/template_1.json. That ES did not error out when I had problems
with the JSON syntax suggests that it's either failing silently or not
reading the file at all, so I don't know what's up with that.

[1]: 2013-09-10 16:16:08,065][DEBUG][action.index ] [Stacy X] [
logstash-2013.09.10][1], node[bzY72RjbSFCTMPMPBvHWtQ], [P], s[STARTED]:
Failed to execute [index {[logstash-2013.09.10][eventlog][utbYay9iRcybQ-
V5EIflRQ], source[{"@source":"tcp://142.58.129.166:52691/","@tags":[],
"@fields":{"Keywords":-9223090561878065151,"ProviderGuid":
"{126CDB97-D346-4894-8A34-658DA5EEA1B6}","Version":0,"Task":0,"OpcodeValue":
2,"ThreadID":8416,"Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":
"SYSTEM","AccountType":"User","Opcode":"Stop","SnapshotPath":
"\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5","ErrorCode":"0x0",
"TotalDirectories":"49311","TotalFiles":"248624","FilesScoped":"102079",
"FilesResident":"26182","FilesCachedFirstPass":"21253",
"FilesMissedSecondPass":"18946","eventlog_severity":"info",
"eventlog_severity_code":2,"eventlog_channel":"Application",
"eventlog_program":"Microsoft-Windows-System-Restore","nxlog_input":
"eventlog","eventlog_id":8301,"eventlog_record_number":24878,"eventlog_pid":
5264},"@timestamp":"2013-09-10T22:02:28.000Z","@source_host":
"lib4013-2.lib.sfu.ca","@source_path":"/","@message":"Scoping completed for
shadowcopy \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5.","@type":
"eventlog"}]}]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [
@fields.ErrorCode]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:320)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue
(ObjectMapper.java:587)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(
ObjectMapper.java:459)
at org.elasticsearch.index.mapper.object.ObjectMapper.
serializeObject(ObjectMapper.java:507)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(
ObjectMapper.java:449)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:486)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:430)
at org.elasticsearch.index.shard.service.InternalIndexShard.
prepareCreate(InternalIndexShard.java:297)
at org.elasticsearch.action.index.TransportIndexAction.
shardOperationOnPrimary(TransportIndexAction.java:211)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction.
performOnPrimary(TransportShardReplicationOperationAction.java:533)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(
TransportShardReplicationOperationAction.java:431)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source
)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: "0x0"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.
longValue(AbstractXContentParser.java:72)
at org.elasticsearch.index.mapper.core.LongFieldMapper.
innerParseCreateField(LongFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.
parseCreateField(NumberFieldMapper.java:182)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:307)
... 13 more

[2] http://pastebin.com/jAW6VBUK

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Martijn Van Groningen) #2

Did you add or change the index template after you create the index? Index
templates are taken into account when a new index is being created.
If you haven't done so create a new index and start indexing into that, the
index template should add ErrorCode as field. Optionally migrate your
data to this new index.

Also I recommend using the put & delete index template api over using
static files, using the apis you're more flexible when it comes to changing
templates.

On 11 September 2013 01:29, Rob Bos robertbos@gmail.com wrote:

I'm sending Windows logs, using nxlog, to a Logstash server with
Elasticsearch, and a field is failing to autodetect correctly, if I'm
reading Google searches correctly. ES is generating errors like "Failed to
parse [@fields.ErrorCode]" [1]. I'm struggling writing a template [2] to
set that field to 'string', but I don't even know for sure if ES is reading
the file, or if I've written it right. I would like a sanity check and
maybe some advice.

I've confirmed through ps that ES is getting path.conf set to
/etc/elasticsearch, and I've placed the template file as
templates/template_1.json. That ES did not error out when I had problems
with the JSON syntax suggests that it's either failing silently or not
reading the file at all, so I don't know what's up with that.

[1]: 2013-09-10 16:16:08,065][DEBUG][action.index ] [Stacy X]
[logstash-2013.09.10][1], node[bzY72RjbSFCTMPMPBvHWtQ], [P], s[STARTED]:
Failed to execute [index {[logstash-2013.09.10][eventlog][utbYay9iRcybQ-
V5EIflRQ], source[{"@source":"tcp://142.58.129.166:52691/","@tags":[],
"@fields":{"Keywords":-9223090561878065151,"ProviderGuid":
"{126CDB97-D346-4894-8A34-658DA5EEA1B6}","Version":0,"Task":0,
"OpcodeValue":2,"ThreadID":8416,"Domain":"NT AUTHORITY","AccountName":
"SYSTEM","UserID":"SYSTEM","AccountType":"User","Opcode":"Stop",
"SnapshotPath":"\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5",
"ErrorCode":"0x0","TotalDirectories":"49311","TotalFiles":"248624",
"FilesScoped":"102079","FilesResident":"26182","FilesCachedFirstPass":
"21253","FilesMissedSecondPass":"18946","eventlog_severity":"info",
"eventlog_severity_code":2,"eventlog_channel":"Application",
"eventlog_program":"Microsoft-Windows-System-Restore","nxlog_input":
"eventlog","eventlog_id":8301,"eventlog_record_number":24878,
"eventlog_pid":5264},"@timestamp":"2013-09-10T22:02:28.000Z",
"@source_host":"lib4013-2.lib.sfu.ca","@source_path":"/","@message":"Scoping
completed for shadowcopy
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5.","@type":"eventlog"
}]}]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [
@fields.ErrorCode]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:320)
at org.elasticsearch.index.mapper.object.ObjectMapper.
serializeValue(ObjectMapper.java:587)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(
ObjectMapper.java:459)
at org.elasticsearch.index.mapper.object.ObjectMapper.
serializeObject(ObjectMapper.java:507)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(
ObjectMapper.java:449)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:486)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:430)
at org.elasticsearch.index.shard.service.InternalIndexShard.
prepareCreate(InternalIndexShard.java:297)
at org.elasticsearch.action.index.TransportIndexAction.
shardOperationOnPrimary(TransportIndexAction.java:211)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction.
performOnPrimary(TransportShardReplicationOperationAction.java:533)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(
TransportShardReplicationOperationAction.java:431)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NumberFormatException: For input string: "0x0"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at org.elasticsearch.common.xcontent.support.
AbstractXContentParser.longValue(AbstractXContentParser.java:72)
at org.elasticsearch.index.mapper.core.LongFieldMapper.
innerParseCreateField(LongFieldMapper.java:281)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.
parseCreateField(NumberFieldMapper.java:182)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:307)
... 13 more

[2] http://pastebin.com/jAW6VBUK

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #3