# Elasticsearch message format

**URL:** <https://discuss.elastic.co/t/elasticsearch-message-format/73856>\
**Category:** Logstash\
**Created:** [February 3, 2017, 1:17pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856 "2017-02-03T13:17:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Oleksii\_Mazilov](https://avatars.discourse-cdn.com/v4/letter/o/9de053/32.png) [@Oleksii\_Mazilov](https://discuss.elastic.co/u/Oleksii_Mazilov)\
**Post date:** [February 3, 2017, 1:17pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856/1 "2017-02-03T13:17:03Z")

</div>

I need to remove several fields before pushing data to Elasticsearch. I can't use **mutate** filter because in **aggregate** filter I'm using field which I need to remove  
Conf example:

```
input {
	jdbc {
		bla_bla_bla
	}
}

filter {
	date {
		match => ["timestamp", "YYYYMMddHHmmss", "YYYY-MM-dd HH:mm:ss", "ISO8601"]
		target => "timestamp"
	}
	if [event_name] == "submitted" {
		aggregate {
			task_id => "%{step_id}"
			code => "map['step_duration'] = event.get('timestamp')"
			map_action => "create"
		}
	}
	if [event_name] == "process_end" {
		aggregate {
			add_field => { "step_name" => "step_ololo" }
			task_id => "%{step_id}"
			code => "event.set('step_duration', event.get('timestamp') - map['step_duration'])"
			map_action => "update"
			end_of_task => true
			timeout => 3600
		}
	}
}

output {
	if ([step_name]) {
		elasticsearch {
			ssl_certificate_verification => "false"
			ssl => "false"
			index => "events-%{+YYYY.MM.dd}"
			hosts => ["localhost:9200"]
			user => "ololo_user"
			password => "ololo_pass"
		}
	}
}

```

Need to remove **event\_name** field

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 3, 2017, 1:26pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856/2 "2017-02-03T13:26:11Z")

</div>

Can't you just remove the field _after_ the aggregate filter?

---

<div class="post-metadata">

**Author:** ![Oleksii\_Mazilov](https://avatars.discourse-cdn.com/v4/letter/o/9de053/32.png) [@Oleksii\_Mazilov](https://discuss.elastic.co/u/Oleksii_Mazilov)\
**Post date:** [February 3, 2017, 3:20pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856/3 "2017-02-03T15:20:35Z")

</div>

Doesn't work:

```
input {
	jdbc {
		bla_bla_bla
	}
}

filter {
	date {
		match => ["timestamp", "YYYYMMddHHmmss", "YYYY-MM-dd HH:mm:ss", "ISO8601"]
		target => "timestamp"
	}
	if [event_name] == "submitted" {
		aggregate {
			task_id => "%{step_id}"
			code => "map['step_duration'] = event.get('timestamp')"
			map_action => "create"
		}
	}
	if [event_name] == "process_end" {
		aggregate {
			add_field => { "step_name" => "step_ololo" }
			task_id => "%{step_id}"
			code => "event.set('step_duration', event.get('timestamp') - map['step_duration'])"
			map_action => "update"
			end_of_task => true
			timeout => 3600
		}
	}
        mutate {
            remove_field => ["%{event_name}"]
        }
}

output {
	if ([step_name]) {
		elasticsearch {
			ssl_certificate_verification => "false"
			ssl => "false"
			index => "events-%{+YYYY.MM.dd}"
			hosts => ["localhost:9200"]
			user => "ololo_user"
			password => "ololo_pass"
		}
	}
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 3, 2017, 3:25pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856/4 "2017-02-03T15:25:04Z")

</div>

> ```
> remove_field => ["%{event_name}"]
> 
> ```

This will remove the field named according to the contents of the `event_name` field, i.e. if that field contains "foo" Logstash will attempt to remove the field `foo`. Do this instead:

```
 remove_field => ["event_name"]

```

---

<div class="post-metadata">

**Author:** ![Oleksii\_Mazilov](https://avatars.discourse-cdn.com/v4/letter/o/9de053/32.png) [@Oleksii\_Mazilov](https://discuss.elastic.co/u/Oleksii_Mazilov)\
**Post date:** [February 3, 2017, 3:37pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856/5 "2017-02-03T15:37:28Z")

</div>

Thanks a lot. Works now!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2017, 3:37pm UTC](https://discuss.elastic.co/t/elasticsearch-message-format/73856/6 "2017-03-03T15:37:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.