journal of the quest for the holy-working-HTTPS-input, day 1: innocent youth
Hi!
I'm trying to configure logstash to receive post json data from client-side JavaScript with SSL enable but i'm a bit new to all these concepts
Basically, i'm trying to find a way to send json to ES from a client-sided JS script from a SSL secured website (therefore, chrome require JS to only communicate with secure services which is not the case of my ES service). My current context does not allow me to use the x-pack security to enable SSL on ES (which, to my understanding, is required to enable SSL on ES). So i'm trying to enable SSL on a logstash http input which will then send the documents to my ES.
Maybe it's a potentially working architecture, maybe not (i'm open to propositions).
Here's my logstash http input + logstash output configuration:
input:
http {
type => "monitoring-js"
host => "0.0.0.0"
port => "8081"
ssl => true
keystore => "/tmp/SSLcerts/keystore.jks"
keystore_password => "generic"
tags => ["reach"]
response_headers => {
"Access-Control-Allow-Origin" => "*"
"Content-Type" => "text/plain"
"Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
}
}
output:
if "reach" in [tags] {
elasticsearch {
hosts => [ "0.0.0.0:9200" ]
index => "reach-%{+YYYY.MM.dd}"
}
}
please note: those aren't the actual ips, keystore path and password.
When i start logstash with these configs, i do not seem to have problems.
Here is my JS ajax post call:
$.ajax(url, {
method: 'POST',
contentType: "application/json",
data:{
action:'watched',
userID:userID,
email:email,
objectID:objectID,
filename:filename,
watched:total,
duration:videoElem.duration,
path:path,
content:content,
"@timestamp":timestamp,
space:space
}
});
url is 'https://site:8081' and all other data are JS vars containing strings, ints or floats. I manually generated a java keystore from an existing key and cert.
When running the script, i get this in the chrome console:
XHR failed loading: OPTIONS "https://site:8081/".
Perhaps it's something with the response header config because it's the last thing i added to my configs while solving my last issues. Maybe i could remove the response from the http input?
journal of the quest for the holy-working-HTTPS-input, day 3: early struggles
EDIT/UPDATE:
I've read a bit more about the subject of CORS to get an actual understanding of what my response header config is doing. Here's the new response header config:
response_headers => {
"Access-Control-Allow-Origin" => "*" # will restrain this eventually...
"Content-Type" => "application/json"
"Access-Control-Allow-Headers" => "content-type"
"Access-Control-Allow-Methods" => "POST, OPTIONS"
}
here's what my browser is failing to send (copied as curl from chrome):
curl "https://site:8081/" -X OPTIONS -H "Access-Control-Request-Method: POST" -H "Origin: https://site" -H "Referer: https://site/index.php?r=space^%^2Fspace^&sguid=7a0662ee-1179-48b5-beec-aa556426a56d" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" -H "Access-Control-Request-Headers: content-type" --compressed
and here's my error message in console:
XHR failed loading: OPTIONS "https://site:8081/".
I must admit that i'm a bit puzzled at the moment...
journal of the quest for the holy-working-HTTPS-input, day 5: the dim-lighted tunnel
RE-EDIT:
Since i was sending the request on the unload event in my JavaScript, the page was unloaded before the request could be sent. I simply disabled the async parameter in the creating on my ajax post request (default was true) and now the browser must wait till the request is sent before unloading the page completely.
$.ajax(url, {
method: 'POST',
contentType: "application/json",
async:false, <-HERE
data:{
foo:foo,
bar:bar,
lots of stuff...
}
});
This might not be the most efficient way to do it, but it works (i'm still open to better propositions tho...)
Now everything is going well and ES is able to receive the information. All of my json info from the data field of my request end up in the single "message" field in my ES document but i guess i can fix that by using grok patterns in my config or by reworking a bit how i add the informations to my ajax request.
Beside this, the only "issue" left is that ES also creates a document for the OPTION request automatically sent by jquery's ajax post call since it's required by my actual usage of the post request (CORS-stuff). So for a single post call, i end up with two documents in ES: one with all the wanted info in the message field and another with no "interesting" information with lots of technical-CORS-handshaking-like-stuff.
Any advice on how to make logstash/ES ignore such requests without blocking the CORS handshake-thingy.
And also how to properly put my informations in my post request so that logstash will properly create the field without putting everything in the "message"?
ES's documentation is very good and complete but i'm actually surprised of how little resources i've managed to find for my use case. Am-i the only one trying to figure out a JS-logstash SSL secured pipeline? Am-i absolutely lost?
journal of the quest for the holy-working-HTTPS-input, day 5.5: Halls of the fallen
REEEEEEEEEEEEEEEE-EDIT:
after nearly losing any trace of sanity while navigating many dim-lighted forum posts and various ancient docs, i stumbled on a working configuration to filter out OPTIONS requests from my output. The legendary scripture goes like this:
output {
if "holy" in [tags] {
if "OPTIONS" not in [headers][request_method]{ <-THIS
elasticsearch {
hosts => [ "myKingdom:9200" ]
index => "holy_grail-%{+YYYY.MM.dd}"
}
}
}
}
i have a feeling my struggles will soon come to an end. Trough the hall of the fallen i shall find the rest of the holiest logstash config. I now only need to assign the datas from the message field to their righteous place to unlock the secrets of the holy-working-HTTPS-input.