Using logstash plugin to change a field value

Elastic Stack Logstash
1

I have a ELK stack deployed in docker swarm and i need to improve some data created by the filebeat. I have a field with docker metadata that contains the node id (container.labels.com_docker_swarm_node_id) with a random docker id and i want to create another field with the hostname of the node. For instance, i have a docker id such as yrkr4jyjlru1kkdw9z8wycj7i in the node id field and everytime that logstash detects a field with that specific code i want to create another field by the name node_name with the value SWARMNODE1 for instance. My logstash config is as follows

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044
}
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
translate {
field => "container.labels.com_docker_swarm_node_id"
destination => "node_name"
dictionary => [
"yrkr4jyjlru1kkdw9z8wycj7i", "SWARMNODE1 ",
"lh7jk40yywxrsfw2xemghmpcz", "SWARMNODE2"
]
}
}

output {
elasticsearch {
hosts => ["http://es-master:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"
}
}

I executed the bin/logstash-plugin list command inside the container and i see both mutate and translate plugin in it, so i assume that both are active. Those plugins are shown with this names:
logstash-filter-mutate
logstash-filter-translate
What am i doing wrong?

2

Does your field name really contain periods, or did you mean [container][labels][com_docker_swarm_node_id]?

3

The exact name inside the table view of the registry in kibana it says container.labels.com_docker_swarm_node_id but with after your question i went to the JSON view and i see this

"container": {
...
  "labels": {
  ...
    "com_docker_swarm_node_id": "lh7jk40yywxrsfw2xemghmpcz",
...
  },
},

the ... are part of the text that i removed because is mostly sensitive data
I am very new in all of this, what's the difference between [container][labels][com_docker_swarm_node_id] and "container.labels.com_docker_swarm_node_id"? Does it have to do with the JSON format?

4

logstash and kibana use different syntax for the names of nested fields. In logstash you would use [container][labels][com_docker_swarm_node_id], whereas in kibana it would be called container.labels.com_docker_swarm_node_id

5

Thanks for the tip Badger! It's working now. I will dig more in the nested fields of logstash and kibana for future filters.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.