Hi Community!
I hope that you're doing great! I've been involved in a project, and we have to process data with a python script and then insert it with Logstash into Elasticsearch. Having that said, I use the exec plugin to execute the script, and then, I print the result in the script, such as: print(response), so Logstash can detect and insert the data into Elasticsearch.
The problem is that this script is being logged in a monitor, and the output is sensitive data that shouldn't log.
Is there any way to pass the value through Python to Logstash output without printing in console, such as executing the script and just using return response?
An example of my Logstash pipeline would be the following:
input {
exec {
command => "/usr/bin/python3 /usr/share/logstash/pipeline/example.py"
interval => 60
codec => "json"
}
}
output {
elasticsearch {
index => "example"
hosts => [ "https://elasticsearch:9200" ]
user => "${ELASTIC_USERNAME}"
password => "${ELASTIC_PASSWORD}"
ssl => true
ssl_certificate_verification => true
cacert => '/certs/ca.crt'
codec => json_lines
}
stdout {}
}
and a small example of a python script would be like:
def hello_world():
hello_world = { "first_message": "hello world!" }
print(hello_world)
if "__main__" == __name__:
hello_world()
The ideal way to do this would be to use return hello_world or something else to don't log the message in the console.