Get MapperParsingException failed to parse in 0.90.10


(Stefan Sabolowitsch) #1

Hi all,
get MapperParsingException failed to parse in 0.90.10

[2014-02-11 16:05:09,402][DEBUG][action.bulk ] [Thunderbolt]
[logstash-2014.02.11][4] failed to execute bulk item (index) index
{[logstash-2014.02.11][suricata][deuCC2bkRvehNSA62tuuHw],
source[{"tags":["suricata"],"@version":1,"@timestamp":"2014-02-11T16:05:07.540+01:00","host":"ipd1.felten-group.com","file":{"filename":"/SpamResolverNG/SpamResolverNG.dll","magic":"data","state":"CLOSED","stored":false,"size":115},"message":"{"time":"02\/11\/2014-15:05:07.540410","event_type":"file","src_ip":"84.39.152.31","src_port":80,"dest_ip":"192.168.100.120","dest_port":3255,"proto":"TCP","http":{"url":"\/SpamResolverNG\/SpamResolverNG.dll?DoNewRequest","hostname":"resolver1.altn.ctmail.com","http_refer":"","http_user_agent":"Mozilla\/4.0
(compatible; Win32; Commtouch Http
Client)"},"file":{"filename":"\/SpamResolverNG\/SpamResolverNG.dll","magic":"data","state":"CLOSED","stored":false,"size":115}}","type":"suricata","received_at":"2014-02-11
16:05:07
+0100","event_type":"file","src_ip":"84.39.152.31","src_port":80,"proto":"TCP","http":{"url":"/SpamResolverNG/SpamResolverNG.dll?DoNewRequest","hostname":"resolver1.altn.ctmail.com","http_refer":"","http_user_agent":"Mozilla/4.0
(compatible; Win32; Commtouch Http
Client)"},"dst_ip":"192.168.100.120","dst_port":3255,"geoip":{"ip":"84.39.152.31","country_code2":"DE","country_code3":"DEU","country_name":"Germany","continent_code":"EU","latitude":51.0,"longitude":9.0,"timezone":"Europe/Berlin","location":[9.0,51.0]}}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[file]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:416)
at
org.elasticsearch.index.mapper.multifield.MultiFieldMapper.parse(MultiFieldMapper.java:204)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:514)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:456)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:516)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:460)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:353)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:156)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException: unknown
property [filename]
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:310)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:261)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:405)

and this is my mapping:

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s",
"analysis" : {
"analyzer" : {
"default" : {
"type" : "standard",
"stopwords" : "none"
}
}
}
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "multi_field",
"fields" : {
"{name}" : {"type": "string", "index" : "analyzed",
"omit_norms" : true },
"raw" : {"type": "string", "index" : "not_analyzed",
"ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"ipver":{"type":"long"},
"protocol":{"type":"long"},
"size":{"type":"long"},
"sp":{"type":"long"},
"stored":{"type":"boolean"},
"@timestamp":{"type":"date", "format":"dateOptionalTime"},
"dp":{"type":"long"},
"rcvd":{"type":"long"},
"sent":{"type":"long"},
"sid":{"type":"long"},
"policy_id":{"type":"long"},
"size":{"type":"long"},
"ids_priority":{"type":"long"},
"duration":{"type":"long"},
"src_port":{"type":"long"},
"src_xlated_port":{type: "long"},
"dst_port":{"type":"long"},
"dst_xlated_port":{type: "long"},
"TTL":{"type":"long"},
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

Any idea ?
thanks for any help here.
Stefan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0f01d48-9131-4e9a-9df5-e50e91e55ea2%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #2

What is your current mapping? Use the GetMapping API.

The file field is an inner object, but you do not have one defined in your
mapping. Very likely you already have indexed a document with the file
field as another type.

--
Ivan

On Tue, Feb 11, 2014 at 7:12 AM, Stefan Sabolowitsch <
sabolowitschst@in-trier.de> wrote:

Hi all,
get MapperParsingException failed to parse in 0.90.10

[2014-02-11 16:05:09,402][DEBUG][action.bulk ] [Thunderbolt]
[logstash-2014.02.11][4] failed to execute bulk item (index) index
{[logstash-2014.02.11][suricata][deuCC2bkRvehNSA62tuuHw],
source[{"tags":["suricata"],"@version":1,"@timestamp":"2014-02-11T16:05:07.540+01:00","host":"
ipd1.felten-group.com
","file":{"filename":"/SpamResolverNG/SpamResolverNG.dll","magic":"data","state":"CLOSED","stored":false,"size":115},"message":"{"time":"02\/11\/2014-15:05:07.540410","event_type":"file","src_ip":"84.39.152.31","src_port":80,"dest_ip":"192.168.100.120","dest_port":3255,"proto":"TCP","http":{"url":"\/SpamResolverNG\/SpamResolverNG.dll?DoNewRequest","hostname":"
resolver1.altn.ctmail.com","http_refer":"","http_user_agent":"Mozilla\/4.0
(compatible; Win32; Commtouch Http
Client)"},"file":{"filename":"\/SpamResolverNG\/SpamResolverNG.dll","magic":"data","state":"CLOSED","stored":false,"size":115}}","type":"suricata","received_at":"2014-02-11
16:05:07
+0100","event_type":"file","src_ip":"84.39.152.31","src_port":80,"proto":"TCP","http":{"url":"/SpamResolverNG/SpamResolverNG.dll?DoNewRequest","hostname":"
resolver1.altn.ctmail.com","http_refer":"","http_user_agent":"Mozilla/4.0
(compatible; Win32; Commtouch Http
Client)"},"dst_ip":"192.168.100.120","dst_port":3255,"geoip":{"ip":"84.39.152.31","country_code2":"DE","country_code3":"DEU","country_name":"Germany","continent_code":"EU","latitude":51.0,"longitude":9.0,"timezone":"Europe/Berlin","location":[9.0,51.0]}}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[file]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:416)
at
org.elasticsearch.index.mapper.multifield.MultiFieldMapper.parse(MultiFieldMapper.java:204)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:514)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:456)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:516)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:460)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:353)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:156)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: org.elasticsearch.ElasticSearchIllegalArgumentException:
unknown property [filename]
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:310)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:261)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:405)

and this is my mapping:

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s",
"analysis" : {
"analyzer" : {
"default" : {
"type" : "standard",
"stopwords" : "none"
}
}
}
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "multi_field",
"fields" : {
"{name}" : {"type": "string", "index" : "analyzed",
"omit_norms" : true },
"raw" : {"type": "string", "index" : "not_analyzed",
"ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"ipver":{"type":"long"},
"protocol":{"type":"long"},
"size":{"type":"long"},
"sp":{"type":"long"},
"stored":{"type":"boolean"},
"@timestamp":{"type":"date", "format":"dateOptionalTime"},
"dp":{"type":"long"},
"rcvd":{"type":"long"},
"sent":{"type":"long"},
"sid":{"type":"long"},
"policy_id":{"type":"long"},
"size":{"type":"long"},
"ids_priority":{"type":"long"},
"duration":{"type":"long"},
"src_port":{"type":"long"},
"src_xlated_port":{type: "long"},
"dst_port":{"type":"long"},
"dst_xlated_port":{type: "long"},
"TTL":{"type":"long"},
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

Any idea ?
thanks for any help here.
Stefan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e0f01d48-9131-4e9a-9df5-e50e91e55ea2%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDDcPo_yzH%3DUfi4JOM8jgORvVQrMQ1wzV2aH2etSo_x%3Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Stefan Sabolowitsch) #3

Hi Ivan,
thanks for your answer, i use as an indexer logstash.

this is my current mapping:

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s",
"analysis" : {
"analyzer" : {
"default" : {
"type" : "standard",
"stopwords" : "none"
}
}
}
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "multi_field",
"fields" : {
"{name}" : {"type": "string", "index" : "analyzed",
"omit_norms" : true },
"raw" : {"type": "string", "index" : "not_analyzed",
"ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"ipver":{"type":"long"},
"protocol":{"type":"long"},
"size":{"type":"long"},
"sp":{"type":"long"},
"stored":{"type":"boolean"},
"@timestamp":{"type":"date", "format":"dateOptionalTime"},
"dp":{"type":"long"},
"rcvd":{"type":"long"},
"sent":{"type":"long"},
"sid":{"type":"long"},
"policy_id":{"type":"long"},
"size":{"type":"long"},
"ids_priority":{"type":"long"},
"duration":{"type":"long"},
"src_port":{"type":"long"},
"src_xlated_port":{type: "long"},
"dst_port":{"type":"long"},
"dst_xlated_port":{type: "long"},
"TTL":{"type":"long"},
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d0e5d29-4747-4dfa-9b79-9469e4814d91%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #4

That is your template. Use the Get Mapping API to find out what actually is
in effect.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-get-mapping.html

On Tue, Feb 11, 2014 at 12:17 PM, Stefan Sabolowitsch <
sabolowitschst@in-trier.de> wrote:

Hi Ivan,
thanks for your answer, i use as an indexer logstash.

this is my current mapping:

{
"template" : "logstash-",
"settings" : {
"index.refresh_interval" : "5s",
"analysis" : {
"analyzer" : {
"default" : {
"type" : "standard",
"stopwords" : "none"
}
}
}
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "multi_field",
"fields" : {
"{name}" : {"type": "string", "index" : "analyzed",
"omit_norms" : true },
"raw" : {"type": "string", "index" : "not_analyzed",
"ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"ipver":{"type":"long"},
"protocol":{"type":"long"},
"size":{"type":"long"},
"sp":{"type":"long"},
"stored":{"type":"boolean"},
"@timestamp":{"type":"date", "format":"dateOptionalTime"},
"dp":{"type":"long"},
"rcvd":{"type":"long"},
"sent":{"type":"long"},
"sid":{"type":"long"},
"policy_id":{"type":"long"},
"size":{"type":"long"},
"ids_priority":{"type":"long"},
"duration":{"type":"long"},
"src_port":{"type":"long"},
"src_xlated_port":{type: "long"},
"dst_port":{"type":"long"},
"dst_xlated_port":{type: "long"},
"TTL":{"type":"long"},
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7d0e5d29-4747-4dfa-9b79-9469e4814d91%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCT-n3ELV2WCC6Y4zZmqbETxsnn-6RB_abcjp8CP5FEEw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Stefan Sabolowitsch) #5

Ah OK,

i get this with "curl -XGET 'http://localhost:9200/_all/_mapping'"

{"kibana-int":{"temp":{"properties":{"dashboard":{"type":"string"},"group":{
"type":"string"},"title":{"type":"string"},"user":{"type":"string"}}},
"dashboard":{"properties":{"dashboard":{"type":"string"},"group":{"type":
"string"},"title":{"type":"string"},"user":{"type":"string"}}}},
"logstash-2014.02.11":{"default":{"dynamic_templates":[{"string_fields":{
"mapping":{"type":"multi_field","fields":{"raw":{"index":"not_analyzed",
"ignore_above":256,"type":"string"},"{name}":{"index":"analyzed",
"omit_norms":true,"type":"string"}}},"match":"","match_mapping_type":
"string"}}],"properties":{"@timestamp":{"type":"date","format":
"dateOptionalTime"},"@version":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs"},"TTL":{"type":"long"},"dp"
:{"type":"long"},"dst_port":{"type":"long"},"dst_xlated_port":{"type":"long"
},"duration":{"type":"long"},"geoip":{"dynamic":"true","properties":{
"location":{"type":"geo_point"}}},"ids_priority":{"type":"long"},"ipver":{
"type":"long"},"policy_id":{"type":"long"},"protocol":{"type":"long"},"rcvd"
:{"type":"long"},"sent":{"type":"long"},"sid":{"type":"long"},"size":{"type"
:"long"},"sp":{"type":"long"},"src_port":{"type":"long"},"src_xlated_port":{
"type":"long"},"stored":{"type":"boolean"}}},"suricata":{"dynamic_templates"
:[{"string_fields":{"mapping":{"type":"multi_field","fields":{"raw":{"index"
:"not_analyzed","ignore_above":256,"type":"string"},"{name}":{"index":
"analyzed","omit_norms":true,"type":"string"}}},"match":"
",
"match_mapping_type":"string"}}],"properties":{"@timestamp":{"type":"date",
"format":"dateOptionalTime"},"@version":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs"},"TTL":{
"type":"long"},"alert":{"properties":{"action":{"type":"multi_field",
"fields":{"action":{"type":"string","norms":{"enabled":false}},"raw":{"type"
:"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"category":{"type":
"multi_field","fields":{"category":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},"gid":{
"type":"long"},"rev":{"type":"long"},"severity":{"type":"long"},"signature"
:{"type":"multi_field","fields":{"signature":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"signature_id":{"type":"long"}}},"dns":{"properties":{
"id":{"type":"long"},"rdata":{"type":"multi_field","fields":{"rdata":{"type"
:"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"rrname":{"type":"multi_field",
"fields":{"rrname":{"type":"string","norms":{"enabled":false}},"raw":{"type"
:"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"rrtype":{"type":
"multi_field","fields":{"rrtype":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},"ttl":{
"type":"long"},"type":{"type":"multi_field","fields":{"type":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}}}},"dp":{"type":"long"},"dst_ip"
:{"type":"multi_field","fields":{"dst_ip":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"dst_port":{"type":"long"},"dst_xlated_port":{"type":
"long"},"duration":{"type":"long"},"event_type":{"type":"multi_field",
"fields":{"event_type":{"type":"string","norms":{"enabled":false}},"raw":{
"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},"file":{
"type":"multi_field","fields":{"file":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"geoip":{"dynamic":"true","properties":{"area_code":{"type":"long"},
"city_name":{"type":"multi_field","fields":{"city_name":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"continent_code":{"type":"multi_field","fields":{
"continent_code":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"country_code2":{"type":
"multi_field","fields":{"country_code2":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"country_code3":{"type":"multi_field","fields":{"country_code3":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"country_name":{"type":
"multi_field","fields":{"country_name":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"dma_code":{"type":"long"},"ip":{"type":"multi_field","fields":{"ip":{"type"
:"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"latitude":{"type":"double"},
"location":{"type":"geo_point"},"longitude":{"type":"double"},"postal_code"
:{"type":"multi_field","fields":{"postal_code":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"real_region_name":{"type":"multi_field","fields":{
"real_region_name":{"type":"string","norms":{"enabled":false}},"raw":{"type"
:"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"region_name":{"type":
"multi_field","fields":{"region_name":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"timezone":{"type":"multi_field","fields":{"timezone":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}}}},"host":{"type":"multi_field","fields":{"host":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"http":{"properties":{
"hostname":{"type":"multi_field","fields":{"hostname":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"http_content_type":{"type":"multi_field","fields":{
"http_content_type":{"type":"string","norms":{"enabled":false}},"raw":{
"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"http_method":{"type":"multi_field","fields":{"http_method":{"type":"string"
,"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"http_refer":{"type":"multi_field","fields":{
"http_refer":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"http_user_agent":{
"type":"multi_field","fields":{"http_user_agent":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"length":{"type":"long"},"protocol":{"type":
"multi_field","fields":{"protocol":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"redirect":{"type":"multi_field","fields":{"redirect":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"status":{"type":"multi_field","fields":{"status":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"url":{"type":"multi_field",
"fields":{"url":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"xff":{"type":
"multi_field","fields":{"xff":{"type":"string","norms":{"enabled":false}},
"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}}}},
"ids_priority":{"type":"long"},"ipver":{"type":"long"},"message":{"type":
"multi_field","fields":{"message":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"policy_id":{"type":"long"},"proto":{"type":"multi_field","fields":{"proto"
:{"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"protocol":{"type":"long"},
"rcvd":{"type":"long"},"received_at":{"type":"multi_field","fields":{
"received_at":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"sent":{"type":"long"},
"sid":{"type":"long"},"size":{"type":"long"},"sp":{"type":"long"},"src_ip":{
"type":"multi_field","fields":{"src_ip":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"src_port":{"type":"long"},"src_xlated_port":{"type":"long"},"stored":{
"type":"boolean"},"tags":{"type":"multi_field","fields":{"tags":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"tls":{"properties":{
"fingerprint":{"type":"multi_field","fields":{"fingerprint":{"type":"string"
,"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"issuerdn":{"type":"multi_field","fields":{"issuerdn"
:{"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"subject":{"type":"multi_field"
,"fields":{"subject":{"type":"string","norms":{"enabled":false}},"raw":{
"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"version":{"type":"multi_field","fields":{"version":{"type":"string","norms"
:{"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}}}},"type":{"type":"multi_field","fields":{"type":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}}}}}}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30daa8bd-3eb8-436e-8c75-9bff454cb7a5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Ivan Brusic) #6

The actual mapping demonstrates the inconsistencies. Did the index already
contain documents when the mapping was applied.

You can use retrieve the mapping on a single field if desired
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-get-field-mapping.html

In your case:
http://localhost:9200/logstash-2014.02.11/suricata/_mapping/field/file

From your mapping:
{
"logstash-2014.02.11": {
....
"suricata": {
"properties": {
...
"file": {
"type": "multi_field",
"fields": {
"file": {
"type": "string",
"norms": {
"enabled": false
}
},
"raw": {
"type": "string",
"index": "not_analyzed",
"norms": {
"enabled": false
},
"index_options": "docs",
"include_in_all": false,
"ignore_above": 256
}
}
...

The file field is identified as a text multi-field. Your indexed document
however looks like:

{
...
"file": {
"filename": "/SpamResolverNG/SpamResolverNG.dll",
"magic": "data",
"state": "CLOSED",
"stored": false,
"size": 115
},
...
}

Here file is an inner object. Either some document was indexed earlier with
a text "file" field or there is something wrong with the mapping. The
dynamic templates should not apply since the match_mapping_type, but
perhaps there is an error somewhere (your mapping or the code). You should
create a reproducible error as described here:
http://www.elasticsearch.org/help

--
Ivan

On Tue, Feb 11, 2014 at 1:50 PM, Stefan Sabolowitsch <
sabolowitschst@in-trier.de> wrote:

Ah OK,

i get this with "curl -XGET 'http://localhost:9200/_all/_mapping'"

{"kibana-int":{"temp":{"properties":{"dashboard":{"type":"string"},"group"
:{"type":"string"},"title":{"type":"string"},"user":{"type":"string"}}},
"dashboard":{"properties":{"dashboard":{"type":"string"},"group":{"type":
"string"},"title":{"type":"string"},"user":{"type":"string"}}}},
"logstash-2014.02.11":{"default":{"dynamic_templates":[{"string_fields"
:{"mapping":{"type":"multi_field","fields":{"raw":{"index":"not_analyzed",
"ignore_above":256,"type":"string"},"{name}":{"index":"analyzed",
"omit_norms":true,"type":"string"}}},"match":"","match_mapping_type":
"string"}}],"properties":{"@timestamp":{"type":"date","format":
"dateOptionalTime"},"@version":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs"},"TTL":{"type":"long"},
"dp":{"type":"long"},"dst_port":{"type":"long"},"dst_xlated_port":{"type":
"long"},"duration":{"type":"long"},"geoip":{"dynamic":"true","properties"
:{"location":{"type":"geo_point"}}},"ids_priority":{"type":"long"},"ipver"
:{"type":"long"},"policy_id":{"type":"long"},"protocol":{"type":"long"},
"rcvd":{"type":"long"},"sent":{"type":"long"},"sid":{"type":"long"},"size"
:{"type":"long"},"sp":{"type":"long"},"src_port":{"type":"long"},
"src_xlated_port":{"type":"long"},"stored":{"type":"boolean"}}},"suricata"
:{"dynamic_templates":[{"string_fields":{"mapping":{"type":"multi_field",
"fields":{"raw":{"index":"not_analyzed","ignore_above":256,"type":"string"
},"{name}":{"index":"analyzed","omit_norms":true,"type":"string"}}},
"match":"
","match_mapping_type":"string"}}],"properties":{"@timestamp":{
"type":"date","format":"dateOptionalTime"},"@version":{"type":"string",
"index":"not_analyzed","norms":{"enabled":false},"index_options":"docs"},
"TTL":{"type":"long"},"alert":{"properties":{"action":{"type":
"multi_field","fields":{"action":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false
},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"category":{"type":"multi_field","fields":{"category":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"gid":{"type":"long"},"rev":{"type":"long"},
"severity":{"type":"long"},"signature":{"type":"multi_field","fields":{
"signature":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"signature_id":{"type"
:"long"}}},"dns":{"properties":{"id":{"type":"long"},"rdata":{"type":
"multi_field","fields":{"rdata":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false
},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"rrname":{"type":"multi_field","fields":{"rrname":{"type":"string","norms"
:{"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms"
:{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"rrtype":{"type":"multi_field","fields":{"rrtype":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"ttl":{"type":"long"},"type"
:{"type":"multi_field","fields":{"type":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}}}},"dp":{"type":"long"},"dst_ip":{"type":
"multi_field","fields":{"dst_ip":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false
},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"dst_port":{"type":"long"},"dst_xlated_port":{"type":"long"},"duration":{
"type":"long"},"event_type":{"type":"multi_field","fields":{"event_type":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"file":{"type":"multi_field",
"fields":{"file":{"type":"string","norms":{"enabled":false}},"raw":{"type"
:"string","index":"not_analyzed","norms":{"enabled":false},"index_options"
:"docs","include_in_all":false,"ignore_above":256}}},"geoip":{"dynamic":
"true","properties":{"area_code":{"type":"long"},"city_name":{"type":
"multi_field","fields":{"city_name":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256
}}},"continent_code":{"type":"multi_field","fields":{"continent_code":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"country_code2":{"type":
"multi_field","fields":{"country_code2":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"country_code3":{"type":"multi_field","fields":{
"country_code3":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"country_name":{"type"
:"multi_field","fields":{"country_name":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"dma_code":{"type":"long"},"ip":{"type":
"multi_field","fields":{"ip":{"type":"string","norms":{"enabled":false}},
"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"latitude":{"type":"double"},"location":{"type":"geo_point"},"longitude":{
"type":"double"},"postal_code":{"type":"multi_field","fields":{
"postal_code":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"real_region_name":{
"type":"multi_field","fields":{"real_region_name":{"type":"string","norms"
:{"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms"
:{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"region_name":{"type":"multi_field","fields":{
"region_name":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"timezone":{"type":
"multi_field","fields":{"timezone":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256
}}}}},"host":{"type":"multi_field","fields":{"host":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"http":{"properties":{"hostname":{"type":
"multi_field","fields":{"hostname":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256
}}},"http_content_type":{"type":"multi_field","fields":{
"http_content_type":{"type":"string","norms":{"enabled":false}},"raw":{
"type":"string","index":"not_analyzed","norms":{"enabled":false},
"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"http_method":{"type":"multi_field","fields":{"http_method":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"http_refer":{"type":
"multi_field","fields":{"http_refer":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256
}}},"http_user_agent":{"type":"multi_field","fields":{"http_user_agent":{
"type":"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"length":{"type":"long"},
"protocol":{"type":"multi_field","fields":{"protocol":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"redirect":{"type":"multi_field","fields":{
"redirect":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"status":{"type":
"multi_field","fields":{"status":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false
},"index_options":"docs","include_in_all":false,"ignore_above":256}}},
"url":{"type":"multi_field","fields":{"url":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"xff":{"type":"multi_field","fields":{"xff":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}}}},"ids_priority":{"type":
"long"},"ipver":{"type":"long"},"message":{"type":"multi_field","fields":{
"message":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"policy_id":{"type":
"long"},"proto":{"type":"multi_field","fields":{"proto":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"protocol":{"type":"long"},"rcvd":{"type":"long"},
"received_at":{"type":"multi_field","fields":{"received_at":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"sent":{"type":"long"},"sid"
:{"type":"long"},"size":{"type":"long"},"sp":{"type":"long"},"src_ip":{
"type":"multi_field","fields":{"src_ip":{"type":"string","norms":{
"enabled":false}},"raw":{"type":"string","index":"not_analyzed","norms":{
"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"src_port":{"type":"long"},"src_xlated_port":{"type"
:"long"},"stored":{"type":"boolean"},"tags":{"type":"multi_field","fields"
:{"tags":{"type":"string","norms":{"enabled":false}},"raw":{"type":
"string","index":"not_analyzed","norms":{"enabled":false},"index_options":
"docs","include_in_all":false,"ignore_above":256}}},"tls":{"properties":{
"fingerprint":{"type":"multi_field","fields":{"fingerprint":{"type":
"string","norms":{"enabled":false}},"raw":{"type":"string","index":
"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}},"issuerdn":{"type":
"multi_field","fields":{"issuerdn":{"type":"string","norms":{"enabled":
false}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":
false},"index_options":"docs","include_in_all":false,"ignore_above":256
}}},"subject":{"type":"multi_field","fields":{"subject":{"type":"string",
"norms":{"enabled":false}},"raw":{"type":"string","index":"not_analyzed",
"norms":{"enabled":false},"index_options":"docs","include_in_all":false,
"ignore_above":256}}},"version":{"type":"multi_field","fields":{"version"
:{"type":"string","norms":{"enabled":false}},"raw":{"type":"string",
"index":"not_analyzed","norms":{"enabled":false},"index_options":"docs",
"include_in_all":false,"ignore_above":256}}}}},"type":{"type":
"multi_field","fields":{"type":{"type":"string","norms":{"enabled":false
}},"raw":{"type":"string","index":"not_analyzed","norms":{"enabled":false
},"index_options":"docs","include_in_all":false,"ignore_above":256}}}}}}}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/30daa8bd-3eb8-436e-8c75-9bff454cb7a5%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBE4%3DxsPLD6b7NBgbo%3D4L9t1Fx0qRZ7Hn4sUkHXQOeD6A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Stefan Sabolowitsch) #7

Ivan,
thank you for your help and good explanations.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d752362-0fc9-4bdd-81a7-624ed35428c7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8