Sending JSON file to Elasticsearch via Logstash HTTP input plugin and Docker

[patchong]

I have a JSON file I want to send to Elasticseaerch via Logstash’s Http input plugin. I also have Docker installed.

I'm just confused as to why I can't see the data I see the data I've sent in Kibana/Elasticsearch

This is my curl statement and response

This is my logstash.conf file

This is Docker

And VirtualBox

[patchong]

@magnusbaeck @fbaligand

[fbaligand (Fabien Baligand)]

Some things :

• first, if you make a curl call with content referenced inside a file, you have to use @ char just before filename :
  curl -v -H "content-type: application/json" http://localhost:12346 -d @'/path/to/file.json'
• then I invite you in a first time to use this simple output to check what is logstash result (instead of elasticsearch) :

stdout {
		codec => rubydebug { metadata => true }
	} 

• then, are you sure your elasticsearch host is "elasticsearch:9200" and not "localhost:9200" ?
• finally, if you still have errors, check Logstash logs and provide it here.

[patchong]

This is my logstash.conf file now

And this is the curl command I'm using to send some dummy date

I was reading, do I have to create an index first so the data can be stored somewhere. Right now, my endpoint is just 31311 but is that specific enough for when I'm posting.

Also how can I get the Logstash log. I installed ELK using Docker

Thanks

[fbaligand (Fabien Baligand)]

Docker documentation for Logstash is here :
https://www.elastic.co/guide/en/logstash/current/_pulling_the_image.html

You can see there that logs are generated here :
/usr/share/logstash/logs

Concerning stdout result, you have to get Logstash process standard output.

Your http input configuration is enough for what you do.
However, I invite you to clearly indicate header "Content-Type" in your curl request, it is important to indicate if your content is "text/plain" or "application/json". It is interpreted by logstash http input.

No you don't have to create an index first, because when you index a document in elasticsearch, if document index is missing, elasticsearch creates it automaticaly.

That said, I invite you to create index template first (to indicate mapping to elasticsearch), as indicated in my previous comment.