# Configuring Elastic Search Mappings and Logstash **URL:** \ **Category:** Elasticsearch\ **Created:** [July 15, 2016, 3:26am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586 "2016-07-15T03:26:02Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![eyeris](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@eyeris](https://discuss.elastic.co/u/eyeris)\ **Post date:** [July 15, 2016, 3:26am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/1 "2016-07-15T03:26:02Z")
Hi, I am trying to put a `.csv` file in to logstash and then get the index to kibana. When a dynamic mapping is given and run logstash. It works fine and Kibana shows the index. Following is the `config` file and the dynamic mapping created by `logstash`. ``` `input { file { path => "D:\Projects\A\Installations\logstash\logstash-2.3.4\bin\code.txt" start_position => beginning } } filter { csv { columns => [ "A", "B", "C", "D" ] separator => "," } mutate{ convert => { "B" => "integer" "C" => "integer" "D" => "integer" } } } output { elasticsearch { hosts=>["localhost:9200"] index => "report" document_id => "%{A}" } stdout { codec => rubydebug } } ``` The dynamic mapping at the elasticsearche's side. ``` "report" : { "mappings" : { "logs" : { "properties" : { "@timestamp" : { "type" : "date", "format" : "strict_date_optional_time||epoch_millis" }, "@version" : { "type" : "string" }, "A" : { "type" : "string" }, "B" : { "type" : "long" }, "C" : { "type" : "long" }, "D" : { "type" : "long" }, "host" : { "type" : "string" }, "message" : { "type" : "string" }, "path" : { "type" : "string" } } } } } } ``` However, once I create custom mapping (shown below) and try to upload the documents, Its not accepted by elastic search. Following is the mapping I have created. ``` curl -XPUT 'http://localhost:9200/test_coverage/' -d '{ "settings" : { "index" : { "number_of_shards" : 3, "number_of_replicas" : 2 }} , "mappings": { "logs": { "properties" : { "A": {"type": "string","index": "not_analyzed"}, "B": {"type": "integer"}, "C": {"type": "integer"}, "D": {"type": "integer"} } } } }' ``` I have following questions, 1. Do I need to add meta fields (@timestamp, @version..etc) to the custom mapping I am creating in elasticsearch? 2. Dynamic mapping actually identifies `long` for fields but `integer` is adequate. Cant I force it to use `integer` s ? 3. Once I delete a document in the log file, it seems like that change is not reflected in the elasticsearche's index. Is there any way to configure it through the logstash configuration file or have to manually remove the doument via an external script. I have following versions of the elk stack and I am working on Windows 7 64 bit. Kibana 4.5.2 Logstash 2.3.4 Elastic Search 2.3.4 Thank You! ---
**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Post date:** [July 15, 2016, 5:46am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/2 "2016-07-15T05:46:26Z")
1/ Looks like that you haven't specify index pattern in your mapping template, it should be something like below ```auto "template": "report*", "settings": { }, "mappings" : { } ``` Any index named `report` or `report-abc...` will use this template 2/ You can use dynamic template to set data type for fields, but those field names must have something in common to match like ```auto any_field1 any_field2 any_field3 ``` then you can use the following dynamic mappings to set those fields as integer ```auto "dynamic_templates": [ { "integer_field": { "mapping": { "type": "integer" }, "match_mapping_type": "string", "match": "any_*" } }, ``` 3/ I'm not sure if you are able to delete just certain documents in an ES index. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 15, 2016, 6:21am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/3 "2016-07-15T06:21:11Z")
> I'm not sure if you are able to delete just certain documents in an ES index. Sure you can. However, deleting (or changing) lines in a log file monitored by Logstash won't cause the corresponding documents in ES to be touched. ---
**Author:** ![eyeris](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@eyeris](https://discuss.elastic.co/u/eyeris)\ **Post date:** [July 15, 2016, 8:50am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/4 "2016-07-15T08:50:44Z")
Thanks. But how can I make the deletes reflected in the documents in ES index? that is one of the problems I am currently having. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 15, 2016, 10:04am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/5 "2016-07-15T10:04:14Z")
What kind of files are you monitoring? How are they updated? ---
**Author:** ![eyeris](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@eyeris](https://discuss.elastic.co/u/eyeris)\ **Post date:** [July 18, 2016, 3:29am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/6 "2016-07-18T03:29:25Z")
Hi The files are `.csv` file and its being updated by a script. Actually the script's output becomes the `.csv` file ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 18, 2016, 9:44am UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/7 "2016-07-18T09:44:39Z")
The challenge here is that Logstash with few exceptions is stateless, i.e. it doesn't track what it has processed. Specifically, it has no support for detecting what has been deleted, but _if_ you know the ids of the documents in ES that have been deleted you can use the elasticsearch output to delete them. You'll need to write glue that diffs the old and new CSV files and emits a list of documents that have disappeared. If the data has a natural key that you can pass to ES as the documents id you can use that key to delete the documents. Otherwise you can generate an id based on other fields (check out the fingerprint filter). ---
**Author:** ![eyeris](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@eyeris](https://discuss.elastic.co/u/eyeris)\ **Post date:** [July 18, 2016, 12:41pm UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/8 "2016-07-18T12:41:39Z")
Hi, Actually I have resolved that challenge using following code. I am taking the field `A` as the key. This configuration works in one of following scenarios only. 1. when the initial field (Fields B) has value `688` and the change it from `688` to `68` =\> **works.** 2. (Fields B) has value `6` and the change it from `6` to `800` =\> **Doesnt Work.** 3. Adding a new raw to the file =\> **Doesnt Work.** Scenario 2 and 3 produce following error, ![](https://us1.discourse-cdn.com/elastic/original/2X/a/a171e1d49f01dd631efdeb0b89fe3486c999afea.PNG) `“exception”=>#` Which I strongly believe is not having a proper mapping in the elasticsearche's side. The custom mapping I apply still doesn't work and I am currently working with a dynamic mapping. Here is the output configuration of the logstash. > ``` > output { > elasticsearch { > hosts=>["localhost:9200"] > index => "report" > document_id => "%{A}" > } > stdout { codec => rubydebug } > } > > ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:34pm UTC](https://discuss.elastic.co/t/configuring-elastic-search-mappings-and-logstash/55586/9 "2017-07-05T22:34:13Z")