# How to transform fields but dont insert them into elasticsearch

**URL:** https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504
**Category:** Logstash
**Created:** [July 9, 2019, 9:39am UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504 "2019-07-09T09:39:38Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![Josep\_Soler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josep_soler/32/47016_2.png) [@Josep\_Soler](https://discuss.elastic.co/u/Josep_Soler)
#### Post date: [July 9, 2019, 9:39am UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504/1 "2019-07-09T09:39:38Z")

</div>

I would like to insert the document name into de index value via variable but i need to lowercase it, so i have to create a new field, also i would like to change the @timestamp to get %{+YYYY.MM.dd} variable. The problem is that i don't want to insert "@timestamp" and "indexName" fields into elastic so i try to delete them but doesn't work. Sorry for my english.

```
input {
   file { 
     path => "/etc/logstash/data/data*.json"
     start_position => "beginning"
     sincedb_path => "/dev/null"
     codec => "json"
     file_completed_action => "delete"
   }
}

filter {
  mutate {
    copy => { "name" => "indexName" }
  }
  mutate {
    lowercase => ["indexName"]
  }
  date {
    match => ["serverTimestamp","ISO8601"]
    target => "@timestamp"
  }
  mutate {
    remove_field => ["path", "host","@version", "@timestamp", "indexName"]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "data_%{indexName}_%{+YYYY.MM.dd}"
  }
}

```

Error :

`ERROR] 2019-07-09 11:28:24.609 [[main]>worker5] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"data_iot_%{indexName}_", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x786acfe3>], :response=>{"index"=>{"_index"=>"data_%{indexName}_", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [data_%{indexName}_], must be lowercase", "index_uuid"=>"_na_", "index"=>"data_%{indexName}_"}}}}`

---

<div class="post-metadata">

### Author: ![Shaoranlaos](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Shaoranlaos](https://discuss.elastic.co/u/Shaoranlaos)
#### Post date: [July 9, 2019, 11:30am UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504/2 "2019-07-09T11:30:20Z")

</div>

There is a special field `@metadata` that is not send to elasticsearch from the output.  
So you can add your addtional fields below that one and they won't be send to elasticsearch.  
e.g.

```auto
...
filter {
  mutate {
    copy => { "name" => "[@metadata][indexName]" }
  }
  mutate {
    lowercase => ["[@metadata][indexName]" ]
  }
  date {
    match => ["serverTimestamp","ISO8601"]
    target => "[@metadata][@timestamp]"
  }
  mutate {
    remove_field => ["path", "host","@version", "@timestamp"]
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "data_%{[@metadata][indexName]}_%{+YYYY.MM.dd}"
  }
}

```

i will have to search myself if you can format another field as date (will come back to you later, if someone else knows how, feel free to elaborate)

---

<div class="post-metadata">

### Author: ![Josep\_Soler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josep_soler/32/47016_2.png) [@Josep\_Soler](https://discuss.elastic.co/u/Josep_Soler)
#### Post date: [July 9, 2019, 11:41am UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504/3 "2019-07-09T11:41:18Z")

</div>

**Shaoranlaos**  
Thank you very much, I will try it this way, at least it will work for now.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2019, 12:49pm UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504/4 "2019-07-09T12:49:15Z")

</div>

> [@Josep\_Soler](#):
>
> mutate { remove\_field =\> ["path", "host","@version", "@timestamp", "indexName"] }

You are deleting [indexName] before the event gets to the output. So the sprintf reference to %{indexName} does not get substituted.

Also, you are using %{+YYYY.MM.dd} in the index name, which is a reference to [@timestamp,] but you have also deleted that, so that does not get substituted.

Lastly, if I recall correctly, [@timestamp] is not optional for elasticsearch. I believe it will fail to index events that do not have that field.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 6, 2019, 12:49pm UTC](https://discuss.elastic.co/t/how-to-transform-fields-but-dont-insert-them-into-elasticsearch/189504/5 "2019-08-06T12:49:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
