# Logstash into Elasticsearch Mapping Issues
**URL:** https://discuss.elastic.co/t/logstash-into-elasticsearch-mapping-issues/20058
**Category:** Elasticsearch
**Created:** [October 3, 2014, 1:02am UTC](https://discuss.elastic.co/t/logstash-into-elasticsearch-mapping-issues/20058 "2014-10-03T01:02:17Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@elorion](https://discuss.elastic.co/u/elorion)
#### Post date: [October 3, 2014, 1:02am UTC](https://discuss.elastic.co/t/logstash-into-elasticsearch-mapping-issues/20058/1 "2014-10-03T01:02:17Z")
Anyone have an idea what to do in a situation where I am using the output
function in logstash to send it to an Elasticsearch cluster via protocol
http and using a JSON template....and the mappings in the JSON template
aren't being used in the elasticsearch cluster.
logstash.conf
input {
tcp {
port =\> 5170
type =\> "sourcefire"
}
}
filter {
```
mutate{
split => ["message", "|"]
add_field => {
"event" => "%{message[5]}"
"eventSource" => "%{message[1]}"
}
}
kv {
include_keys => ["dhost", "dst", "dpt", "shost", "src", "spt", "rt"]
}
mutate {
rename => ["dhost", "destinationHost"]
rename => ["dst", "destinationAddress"]
rename => ["dpt", "destinationPort"]
rename => ["shost", "sourceHost"]
rename => ["src", "sourceAddress"]
rename => ["spt", "sourcePort"]
}
date {
match => ["rt","UNIX_MS"]
target => "eventDate"
}
geoip {
add_tag => ["sourceGeo"]
source => "src"
database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
}
geoip {
add_tag => ["destinationGeo"]
source => "src"
database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
}
```
}
output {
if [type] == "sourcefire" {
elasticsearch {
cluster =\> "XXX-cluster"
flush\_size =\> 1
manage\_template =\> true
template =\> "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-sourcefire.json"
}
}
}
JSON Template
{
"template": "logstash-\*",
"settings": {
"index.refresh\_interval": "5s"
},
"mappings": {
"Sourcefire": {
"\_all": {
"enabled": true
},
"properties": {
"@timestamp": {
"type": "date",
"format": "basicDateTimeNoMillis"
},
"@version": {
"type": "string",
"index": "not\_analyzed"
},
"geoip": {
"type": "object",
"dynamic": true,
"path": "full",
"properties": {
"location": {
"type": "geo\_point"
}
}
},
"event": {
"type": "string",
"index": "not\_analyzed"
},
"eventDate": {
"type": "date",
"format": "basicDateTimeNoMillis"
},
"destinationAddress": {
"type": "ip"
},
"destinationHost": {
"type": "string",
"index": "not\_analyzed"
},
"destinationPort": {
"type": "integer",
"index": "not\_analyzed"
},
"sourceAddress": {
"type": "ip"
},
"sourceHost": {
"type": "string",
"index": "not\_analyzed"
},
"sourcePort": {
"type": "integer",
"index": "not\_analyzed"
}
}
}
}
}
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/86a4ddaf-554b-4e78-b8ac-18af1373f0cc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/86a4ddaf-554b-4e78-b8ac-18af1373f0cc%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)
#### Post date: [October 6, 2014, 8:44pm UTC](https://discuss.elastic.co/t/logstash-into-elasticsearch-mapping-issues/20058/2 "2014-10-06T20:44:18Z")
I haven't ever let logstash set the default mappings. Instead, whenever a
logstash-style index is created, I let Elasticsearch set the default
mappings from its template. That way, it works even if I replace logstash
with something else.
For example, with my $ES\_CONFIG/templates/automap.json file is the following
{
"automap" : {
"template" : "logstash-\*",
"settings" : {
"index.mapping.ignore\_malformed" : true
},
"mappings" : {
"_default_" : {
"numeric\_detection" : true,
"\_all" : { "enabled" : false },
"properties" : {
"message" : { "type" : "string" },
"host" : { "type" : "string" },
"UUID" : { "type" : "string", "index" : "not\_analyzed" },
"logdate" : { "type" : "string", "index" : "no" }
}
}
}
}
}
And since logstash stores the entire message within the message field and I
never modify that particular field, the \_all field is disabled and
Elasticsearch is told to use the message field as the default within a
Kibana query via the following Java option when starting Elasticsearch as
part of the ELK stack:
-Des.index.query.default\_field=message
I hope this helps!
Brian
On Thursday, October 2, 2014 9:02:17 PM UTC-4, [elo...@gmail.com](mailto:elo...@gmail.com) wrote:
> Anyone have an idea what to do in a situation where I am using the output
> function in logstash to send it to an Elasticsearch cluster via protocol
> http and using a JSON template....and the mappings in the JSON template
> aren't being used in the elasticsearch cluster.
>
> logstash.conf
>
> input {
> tcp {
> port =\> 5170
> type =\> "sourcefire"
> }
> }
>
> filter {
>
> ```
> mutate{
> split => ["message", "|"]
> add_field => {
> "event" => "%{message[5]}"
> "eventSource" => "%{message[1]}"
> }
> }
>
> kv {
> include_keys => ["dhost", "dst", "dpt", "shost", "src", "spt", "rt"]
> }
>
> mutate {
> rename => ["dhost", "destinationHost"]
> rename => ["dst", "destinationAddress"]
> rename => ["dpt", "destinationPort"]
> rename => ["shost", "sourceHost"]
> rename => ["src", "sourceAddress"]
> rename => ["spt", "sourcePort"]
> }
>
> date {
> match => ["rt","UNIX_MS"]
> target => "eventDate"
> }
>
> geoip {
> add_tag => ["sourceGeo"]
> source => "src"
> database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
> }
>
> geoip {
> add_tag => ["destinationGeo"]
> source => "src"
> database => "/opt/logstash/vendor/geoip/GeoLiteCity.dat"
> }
>
> ```
>
> }
>
> output {
> if [type] == "sourcefire" {
> elasticsearch {
> cluster =\> "XXX-cluster"
> flush\_size =\> 1
> manage\_template =\> true
> template =\> "/opt/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-sourcefire.json"
> }
> }
> }
>
> JSON Template
>
> {
> "template": "logstash-\*",
> "settings": {
> "index.refresh\_interval": "5s"
> },
> "mappings": {
> "Sourcefire": {
> "\_all": {
> "enabled": true
> },
> "properties": {
> "@timestamp": {
> "type": "date",
> "format": "basicDateTimeNoMillis"
> },
> "@version": {
> "type": "string",
> "index": "not\_analyzed"
> },
> "geoip": {
> "type": "object",
> "dynamic": true,
> "path": "full",
> "properties": {
> "location": {
> "type": "geo\_point"
> }
> }
> },
> "event": {
> "type": "string",
> "index": "not\_analyzed"
> },
> "eventDate": {
> "type": "date",
> "format": "basicDateTimeNoMillis"
> },
> "destinationAddress": {
> "type": "ip"
> },
> "destinationHost": {
> "type": "string",
> "index": "not\_analyzed"
> },
> "destinationPort": {
> "type": "integer",
> "index": "not\_analyzed"
> },
> "sourceAddress": {
> "type": "ip"
> },
> "sourceHost": {
> "type": "string",
> "index": "not\_analyzed"
> },
> "sourcePort": {
> "type": "integer",
> "index": "not\_analyzed"
> }
> }
> }
> }
> }
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ed3eba42-7142-4b9a-8334-8463f519c9bc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ed3eba42-7142-4b9a-8334-8463f519c9bc%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)
#### Post date: [October 6, 2014, 10:15pm UTC](https://discuss.elastic.co/t/logstash-into-elasticsearch-mapping-issues/20058/3 "2014-10-06T22:15:17Z")
I also have the following Logstash output configuration:
output {
# For testing only
stdout { codec =\> rubydebug }
# Elasticsearch via HTTP REST
elasticsearch {
protocol =\> "http"
codec =\> json
manage\_template =\> false
```
# Or whatever target ES host is required:
host => "localhost"
# Or whatever _type is desired: Usually the environment name
# e.g. qa, devtest, prod, and so on:
index_type => "sample"
```
}
}
Brian
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e05143d2-a2fd-4365-932b-b4603b08165c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e05143d2-a2fd-4365-932b-b4603b08165c%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:58am UTC](https://discuss.elastic.co/t/logstash-into-elasticsearch-mapping-issues/20058/4 "2017-07-06T00:58:02Z")