How to remove configuration option from document [SOLVED]

Elastic Stack Logstash
1

## when i config like this

input {
redis {
host => "127.0.0.1"
port => 6379
key => "logstash:test"
data_type => "list"
codec => "json"
type => "%{[type]}"
}
}

output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
document_id => "%{[id]}"
action => "%{[action]}"
index => "%{[index]}"
}
}

**

and i set key in redis like this :

**

lpush logstash:test '{"id" :"2","action": "index","name":"test","index":"blog","type":"1"}'

**

in elasticsearch

**

{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "blog",
"_type": "1",
"_id": "2",
"_score": 1,
"_source": {
"@timestamp": "2016-11-03T08:54:43.994Z",
"name": "test",
"@version": "1",
"action": "index",
"index": "blog",
"id": "2",
"type": "1"
}
}
]
}
}

but i don't want action ,index ,id ,and type in the field,i only want these as follows

{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "blog",
"_type": "1",
"_id": "2",
"_score": 1,
"_source": {
"@timestamp": "2016-11-03T08:54:43.994Z",
"name": "test",
"@version": "1"
}
}
]
}
}

how can i do? thanks

Regards,
Archer

2

you can move these fields to the @metadata field, they aren't pushed to elasticsearch:

filter {
  mutate {
    rename => {
      "action" => "[@metadata][action]"
      "index" => "[@metadata][index]"
      "id" => "[@metadata][id]"
      "type" => "[@metadata][type]"
    }
  }
}
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    document_id => "%{[@metadata][id]}"
    action => "%{[@metadata][action]}"
    index => "%{[@metadata][index]}"
  }
}
1 Like
3

Hi,jsvd, thanks for you help.but it seems that the field 'type' always be 'log'
as follows:
"_index": "blog",
"_type": "logs",
"_id": "1",
"_score": 1,
"_source": {
"@timestamp": "2016-11-04T05:27:17.364Z",
"name": "test21",
"@version": "1",
"content": "update"

the configuration

input {
  redis {
    host => "127.0.0.1"
    port => 6379
    key => "logstash:test"
    data_type => "list"
    codec => "json"
    type =>  "%{[type]}"
  }
}

filter {
  mutate {
    rename => {
      "action" => "[@metadata][action]"
      "index" => "[@metadata][index]"
      "id" => "[@metadata][id]"
      "type" => "[@metadata][type]"
    }
  }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    document_id => "%{[@metadata][id]}"
    action => "%{[@metadata][action]}"
    index => "%{[@metadata][index]}"
  }
}

btw,i use type => %{[@metadata][type]} instead of %{[type]},still can not work
so how can i accept the "type" ?

4

Using type => "%{[type]}" in an input doesn't make sense. What do you want the document type to be? Same for all events arriving via your redis input? Or set based on the JSON payload? In the latter case I suspect you can just drop the type option for your input.

5

hi,magnusbaeck,thanks for you help.

i want the document type to be what i set in the redis.

for example:
the JSON set in redis

'{"id" :"1","action": "index","name":"test21","index":"blog","type":"type-test","content":"update"}'

what I expect in document is

"_index": "blog",
"_type": "type-test",
"_id": "1",
"_score": 1,
"_source": {
"@timestamp": "2016-11-04T05:55:34.382Z",
"name": "test21",
"@version": "1",
"content": "update"

but now the "type" always be "logs".

and it seems that i drop the type option for my input does not work:confused:

6

Please show your current configuration and what a sample event looks like. Use a stdout { codec => rubydebug } so we can see exactly what the event looks like.

7

thanks, I use "document_type" in output instead of "type" in input,and it works.
But thanks a lot !