Using logstash plugin to change a field value

I have a ELK stack deployed in docker swarm and i need to improve some data created by the filebeat. I have a field with docker metadata that contains the node id (container.labels.com_docker_swarm_node_id) with a random docker id and i want to create another field with the hostname of the node. For instance, i have a docker id such as yrkr4jyjlru1kkdw9z8wycj7i in the node id field and everytime that logstash detects a field with that specific code i want to create another field by the name node_name with the value SWARMNODE1 for instance. My logstash config is as follows

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {
beats {
port => 5044

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
translate {
field => "container.labels.com_docker_swarm_node_id"
destination => "node_name"
dictionary => [
"yrkr4jyjlru1kkdw9z8wycj7i", "SWARMNODE1 ",
"lh7jk40yywxrsfw2xemghmpcz", "SWARMNODE2"

output {
elasticsearch {
hosts => ["http://es-master:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
#user => "elastic"
#password => "changeme"

I executed the bin/logstash-plugin list command inside the container and i see both mutate and translate plugin in it, so i assume that both are active. Those plugins are shown with this names:
What am i doing wrong?

Does your field name really contain periods, or did you mean [container][labels][com_docker_swarm_node_id]?

The exact name inside the table view of the registry in kibana it says container.labels.com_docker_swarm_node_id but with after your question i went to the JSON view and i see this

"container": {
  "labels": {
    "com_docker_swarm_node_id": "lh7jk40yywxrsfw2xemghmpcz",

the ... are part of the text that i removed because is mostly sensitive data
I am very new in all of this, what's the difference between [container][labels][com_docker_swarm_node_id] and "container.labels.com_docker_swarm_node_id"? Does it have to do with the JSON format?

logstash and kibana use different syntax for the names of nested fields. In logstash you would use [container][labels][com_docker_swarm_node_id], whereas in kibana it would be called container.labels.com_docker_swarm_node_id

Thanks for the tip Badger! It's working now. I will dig more in the nested fields of logstash and kibana for future filters.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.