The quest for the holy-working-HTTPS-input (halloween edition)

journal of the quest for the holy-working-HTTPS-input, day 1: innocent youth

I'm trying to configure logstash to receive post json data from client-side JavaScript with SSL enable but i'm a bit new to all these concepts

Basically, i'm trying to find a way to send json to ES from a client-sided JS script from a SSL secured website (therefore, chrome require JS to only communicate with secure services which is not the case of my ES service). My current context does not allow me to use the x-pack security to enable SSL on ES (which, to my understanding, is required to enable SSL on ES). So i'm trying to enable SSL on a logstash http input which will then send the documents to my ES.

Maybe it's a potentially working architecture, maybe not (i'm open to propositions).

Here's my logstash http input + logstash output configuration:

http {
type => "monitoring-js"
	host => ""
	port => "8081"
	ssl => true
	keystore => "/tmp/SSLcerts/keystore.jks"
	keystore_password => "generic"
	tags => ["reach"]
    response_headers => {
		"Access-Control-Allow-Origin" => "*"
		"Content-Type" => "text/plain"
		"Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"

if "reach" in [tags] {
	elasticsearch {
	hosts => [ "" ]
	index => "reach-%{+YYYY.MM.dd}"

please note: those aren't the actual ips, keystore path and password.

When i start logstash with these configs, i do not seem to have problems.

Here is my JS ajax post call:

$.ajax(url, {
            	method: 'POST',
            	contentType: "application/json",

url is 'https://site:8081' and all other data are JS vars containing strings, ints or floats. I manually generated a java keystore from an existing key and cert.

When running the script, i get this in the chrome console:
XHR failed loading: OPTIONS "https://site:8081/".

Perhaps it's something with the response header config because it's the last thing i added to my configs while solving my last issues. Maybe i could remove the response from the http input?

journal of the quest for the holy-working-HTTPS-input, day 3: early struggles

I've read a bit more about the subject of CORS to get an actual understanding of what my response header config is doing. Here's the new response header config:

response_headers => {
		"Access-Control-Allow-Origin" => "*"  # will restrain this eventually...
		"Content-Type" => "application/json"
		"Access-Control-Allow-Headers" => "content-type"
		"Access-Control-Allow-Methods" => "POST, OPTIONS"

here's what my browser is failing to send (copied as curl from chrome):

curl "https://site:8081/" -X OPTIONS -H "Access-Control-Request-Method: POST" -H "Origin: https://site" -H "Referer: https://site/index.php?r=space^%^2Fspace^&sguid=7a0662ee-1179-48b5-beec-aa556426a56d" -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" -H "Access-Control-Request-Headers: content-type" --compressed

and here's my error message in console:
XHR failed loading: OPTIONS "https://site:8081/".

I must admit that i'm a bit puzzled at the moment...

journal of the quest for the holy-working-HTTPS-input, day 5: the dim-lighted tunnel
Since i was sending the request on the unload event in my JavaScript, the page was unloaded before the request could be sent. I simply disabled the async parameter in the creating on my ajax post request (default was true) and now the browser must wait till the request is sent before unloading the page completely.

$.ajax(url, {
            	method: 'POST',
            	contentType: "application/json",
            	async:false, <-HERE
            		lots of stuff...

This might not be the most efficient way to do it, but it works (i'm still open to better propositions tho...)

Now everything is going well and ES is able to receive the information. All of my json info from the data field of my request end up in the single "message" field in my ES document but i guess i can fix that by using grok patterns in my config or by reworking a bit how i add the informations to my ajax request.

Beside this, the only "issue" left is that ES also creates a document for the OPTION request automatically sent by jquery's ajax post call since it's required by my actual usage of the post request (CORS-stuff). So for a single post call, i end up with two documents in ES: one with all the wanted info in the message field and another with no "interesting" information with lots of technical-CORS-handshaking-like-stuff.

Any advice on how to make logstash/ES ignore such requests without blocking the CORS handshake-thingy.
And also how to properly put my informations in my post request so that logstash will properly create the field without putting everything in the "message"?

ES's documentation is very good and complete but i'm actually surprised of how little resources i've managed to find for my use case. Am-i the only one trying to figure out a JS-logstash SSL secured pipeline? Am-i absolutely lost?

journal of the quest for the holy-working-HTTPS-input, day 5.5: Halls of the fallen
after nearly losing any trace of sanity while navigating many dim-lighted forum posts and various ancient docs, i stumbled on a working configuration to filter out OPTIONS requests from my output. The legendary scripture goes like this:

output {
if "holy" in [tags] {
	if "OPTIONS" not in [headers][request_method]{ <-THIS
		elasticsearch {
		hosts => [ "myKingdom:9200" ]
		index => "holy_grail-%{+YYYY.MM.dd}"

i have a feeling my struggles will soon come to an end. Trough the hall of the fallen i shall find the rest of the holiest logstash config. I now only need to assign the datas from the message field to their righteous place to unlock the secrets of the holy-working-HTTPS-input.

journal of the quest for the holy-working-HTTPS-input, day 6: The Library of the lost wisdom
I ran out of paper(7000 chars) in my previous journal so i will continue my story here...

In the halls i have found the entry to the mystic library of the lost wisdom, where abandonned contents of my ajax request's data field ended up being eaten by vermins and rotting all together in an unexploitable message field in my ES document. I must now find a way to split this pile of wisdom-full rubble into nice and pretty classified json fields.

 "message": "action=watched&userID=0000&"

On these ES documents figures the tag "_jsonparsefailure".

"tags": [

Perhaps this runic incantation will show me the path to the long-forgotten configuration that will allow me to put some order in this library and reach my goal: to return its long-lost knowledge to my king!

journal of the quest for the holy-working-HTTPS-input, day 6.5: Homecoming

After a few throws of madness i finally came back to my sense to realize i simply was holding this unintelligible script upside-down. Within a second, everything made sense and the whole pipeline was going to work!

I rushed trough my previous incantations and found the one spot where i forgot one of the most basic step of spell-casting! I fixed my previous javaScript to unleashed the power of JSON:

var data = JSON.stringify({

            $.ajax(url, {
            	method: 'POST',
            	contentType: "application/json",
            	data: data    		   		

With this scripture now encoded correctly, i spoke the spell to the saccaged library lying before me. Slowly, the piles of helf-rotting documents began to shift slowly to their organised place on the now-dust-free shelves of this seemingly-endless room.

Alast! I've struck light trough this madness and restored order to this god-forsaken place! I shall now return to my kingdom and announce my victory! Knowledge will be ours and a new golden era of prosperity will bring riches and plentyful harvests to my people!

As i came out of the lonely halls of the darkest forum, i suddently felt a shiver from the depts of my soul. An unspeakable feeling of doubt filling my head: Was this all true? The forests and damp caves; the shady tunnels and mead hall going beyond what my eyes could see. Was i a programmer stuck on a config file? Or a wizard sent on a holy quest? My steady pace came to a sudden stop as i felt the cold wind on my face and stared at the full moon in the empty sky.

Is my mind made of code or my code made of mind?
I scream, for i do not know.
But no sound came out.
And none ever will,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.