Elasticsearch Templates: How do I handle field properties?


(tom rkba) #1

I am running Logstash and one of my logs has various properties.

I have successfully applied this template:

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '{
"template": "logstash*",
"settings": {
"index.query.default_field": "@message",
"index.cache.field.type": "soft",
"index.store.compress.stored": true
},
"mappings": {
"default": {
"_all": { "enabled": false },
"properties": {
"@message": { "type": "string", "index": "analyzed" },
"@source": { "type": "string", "index": "not_analyzed" },
"@source_host": { "type": "string", "index": "not_analyzed" },
"@source_path": { "type": "string", "index": "not_analyzed" },
"@tags": { "type": "string", "index": "not_analyzed" },
"@timestamp": { "type": "string", "index": "not_analyzed" },
"@type": { "type": "string", "index": "not_analyzed" }
}
}
}
}
'

However, one of the logs being sent to ES via Logstash includes field
properties. This is the grok filter from the Logstash conf:

pattern => "%{IP:vipIP} %{IP:queryingServerIP} - -
[%{HTTPDATE:timestamp}] "%{WORD:httpmethod} %{URIPATHPARAM:request}
%{WORD:httpprotocol}/%{NUMBER:httpversion:float}"
%{NUMBER:httpresponsecode:int} %{NUMBER:bytes:int}
%{NUMBER:responsetime:float}"
add_field => [ "teamname", "Team1" ]
add_field => [ "sourcetype", "access_log" ]

This translates into these mappings on the ES server:

"metadata" : {
"templates" : { },
"indices" : {
"logstash-2013.10.18" : {
"state" : "open",
"settings" : {
"index.number_of_shards" : "3",
"index.number_of_replicas" : "2",
"index.version.created" : "900599"
},
"mappings" : {
"access_log" : {
"properties" : {
"@fields" : {
"properties" : {
"vipIP" : {
"type" : "string"
},
"timestamp" : {
"type" : "string"
},
"httpversion" : {
"type" : "double"
},
"responsetime" : {
"type" : "double"
},
"bytes" : {
"type" : "long"
},
"teamname" : {
"type" : "string"
},
"request" : {
"type" : "string"
},
"queryingServerIP" : {
"type" : "string"
},
"httpmethod" : {
"type" : "string"
},
"httpprotocol" : {
"type" : "string"
},
"httpresponsecode" : {
"type" : "long"
},
"sourcetype" : {
"type" : "string"
}
}
},
"@timestamp" : {
"format" : "dateOptionalTime",
"type" : "date"
},
"@message" : {
"type" : "string"
},
"@source" : {
"type" : "string"
},
"@type" : {
"type" : "string"
},
"@tags" : {
"type" : "string"
},
"@source_host" : {
"type" : "string"
},
"@source_path" : {
"type" : "string"
}
}
},
[snip]

Do I need to handle the field properties in the template? Or will the
generic one I showed at the beginning of the post be adequate? If I do
need to do something with them, what do I need to add?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Luca Cavanna) #2

Hi,
you don't necessarily need to add all the fields (or objects, which are
fields anyway) to your mappings, unless you are not happy with the defaults.
If you don't specify the @fields one in the mapping the type of its
children will be auto-detected based on their json type. Also, the strings
fields will be "analyzed".

On Thursday, October 31, 2013 3:04:03 PM UTC+1, tom rkba wrote:

I am running Logstash and one of my logs has various properties.

I have successfully applied this template:

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '{
"template": "logstash*",
"settings": {
"index.query.default_field": "@message",
"index.cache.field.type": "soft",
"index.store.compress.stored": true
},
"mappings": {
"default": {
"_all": { "enabled": false },
"properties": {
"@message": { "type": "string", "index": "analyzed" },
"@source": { "type": "string", "index": "not_analyzed" },
"@source_host": { "type": "string", "index": "not_analyzed" },
"@source_path": { "type": "string", "index": "not_analyzed" },
"@tags": { "type": "string", "index": "not_analyzed" },
"@timestamp": { "type": "string", "index": "not_analyzed" },
"@type": { "type": "string", "index": "not_analyzed" }
}
}
}
}
'

However, one of the logs being sent to ES via Logstash includes field
properties. This is the grok filter from the Logstash conf:

pattern => "%{IP:vipIP} %{IP:queryingServerIP} - -
[%{HTTPDATE:timestamp}] "%{WORD:httpmethod} %{URIPATHPARAM:request}
%{WORD:httpprotocol}/%{NUMBER:httpversion:float}"
%{NUMBER:httpresponsecode:int} %{NUMBER:bytes:int}
%{NUMBER:responsetime:float}"
add_field => [ "teamname", "Team1" ]
add_field => [ "sourcetype", "access_log" ]

This translates into these mappings on the ES server:

"metadata" : {
"templates" : { },
"indices" : {
"logstash-2013.10.18" : {
"state" : "open",
"settings" : {
"index.number_of_shards" : "3",
"index.number_of_replicas" : "2",
"index.version.created" : "900599"
},
"mappings" : {
"access_log" : {
"properties" : {
"@fields" : {
"properties" : {
"vipIP" : {
"type" : "string"
},
"timestamp" : {
"type" : "string"
},
"httpversion" : {
"type" : "double"
},
"responsetime" : {
"type" : "double"
},
"bytes" : {
"type" : "long"
},
"teamname" : {
"type" : "string"
},
"request" : {
"type" : "string"
},
"queryingServerIP" : {
"type" : "string"
},
"httpmethod" : {
"type" : "string"
},
"httpprotocol" : {
"type" : "string"
},
"httpresponsecode" : {
"type" : "long"
},
"sourcetype" : {
"type" : "string"
}
}
},
"@timestamp" : {
"format" : "dateOptionalTime",
"type" : "date"
},
"@message" : {
"type" : "string"
},
"@source" : {
"type" : "string"
},
"@type" : {
"type" : "string"
},
"@tags" : {
"type" : "string"
},
"@source_host" : {
"type" : "string"
},
"@source_path" : {
"type" : "string"
}
}
},
[snip]

Do I need to handle the field properties in the template? Or will the
generic one I showed at the beginning of the post be adequate? If I do
need to do something with them, what do I need to add?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #3