Problems with mutate and python

[dabyd (David Mediano Montesinos)]

Hello,

I have prblems with mutate. I have problems separating the message into variables. For the variable "@campo1" the operation is correct, but for the other variables it does not work. it includes quotes or question marks.

I send my data by python:

while True:
pulsaciones = random.uniform(50, 130)
data = {'@campo1': pulsaciones, '@user':'Jhon Tr', 'hostname':socket.gethostname(), '@campo2':pulsaciones, '@campo3':pulsaciones }
data_string = json.dumps(data)
s.send(json.dumps(data))

My config.conf
input {
tcp {
# codec => json
host => "127.0.0.1"
port => 5959
}

}

filter {
mutate {
gsub => [
"message", '{', "",
"message", '}', ""
]
gsub => [
"message", "[\?#-]",".",
"message", "/","_"
]
}
kv {
source => "message"
trim_value => """
trim_key => ""\ ()"
field_split => ","
value_split => ":"
remove_field => ["message"]
}
mutate {
convert => { "@campo1" => "integer" }
convert => { "@campo2" => "integer" }
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
user => elastic
password => changeme
}

    stdout {
            codec => rubydebug
    }

}

And i see that in logstash

{
""@campo2" => "90.56460061568626",
"@timestamp" => 2017-12-29T12:08:53.893Z,
""@campo3" => "90.56460061568626",
"port" => 50550,
"@version" => "1",
"host" => "127.0.0.1",
""@user" => "Jhon Tr",
"@campo1" => 90,
""hostname" => "osboxes"
}

And i see that in kibana

[magnusbaeck (Magnus Bäck)]

Use a json filter or json codec to process JSON data. With a tcp input like this you might have to use a json_lines codec and change the Python code to end each JSON blob with a newline character.