# Http plugin of logstash

**URL:** <https://discuss.elastic.co/t/http-plugin-of-logstash/33257>\
**Category:** Logstash\
**Created:** [October 29, 2015, 2:35pm UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257 "2015-10-29T14:35:00Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 29, 2015, 2:35pm UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/1 "2015-10-29T14:35:00Z")

</div>

Hi,  
I am using ELK stack for Log Centralisation. I have written an ansible role (logstash-forwarder) which helps my application to send log to ELK using certificate. My ELK is currently supporting lumberjack. One of my application is hosted to Iron.io from where I want to send log to ELK. I have updated both elastic search and lostash version to 2.0. So Now I want to configure my input file to take http and tcp input. My objective is my iron.io app will send log to my ELK using http/tcp/udp and logstash will convert the data and log will be shown in Kibana.

My input file:

input {  
lumberjack {  
port =\> 5000  
type =\> "logs"  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
}  
http {  
port =\> 8080  
user =\> myuser  
password =\> "\*\*\*\*\*\*"  
}  
}

Do I need to bring any change in my output file?  
My Output file:  
input { stdin { } }  
output {  
elasticsearch { hosts =\> localhost }  
stdout { codec =\> rubydebug }  
}

Please also let me know how will I conduct the test (sending log from local)

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 29, 2015, 7:55pm UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/2 "2015-10-29T19:55:55Z")

</div>

> Do I need to bring any change in my output file?

No, not if you want to send all events received via HTTP to the two existing outputs.

> Please also let me know how will I conduct the test (sending log from local)

Use curl or some other HTTP client?

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 30, 2015, 2:48am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/3 "2015-10-30T02:48:14Z")

</div>

My input configuration:  
input {  
lumberjack {  
port =\> 5000  
type =\> "logs"  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
}  
http {  
port =\> 5555  
user =\> myuser  
password =\> "\*\*\*\*\*\*"  
}  
}

I tried to curl from my local using the below command:

curl -H "content-type: application/json" -XPUT '[http://elk.example.com:5555/twitter/tweet/1](http://elk.example.com:5555/twitter/tweet/1)' -d '{  
"post\_date" : "2015-10-29T14:12:12",  
"message" : "trying out Elasticsearch"  
"port =\> 5555  
"user" =\> "myuser"  
"password" =\> ""\*\*\*\*\*\*""  
}'

seems like it was successfully posted. But I could not find this in Kibana.. Please help me

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2015, 6:59am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/4 "2015-10-30T06:59:44Z")

</div>

I'm not sure what you're trying to accomplish. There are multiple problems here:

- Your curl command looks like an attempt to post directly to Elasticsearch, but you're actually posting to Logstash. What's the purpose of your http input?
- The payload you post isn't valid JSON (because of the `=>`).
- You're specifying the username and password as part of the payload, not it needs to be part of the HTTP header.

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 30, 2015, 7:16am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/5 "2015-10-30T07:16:15Z")

</div>

I have an Ubuntu server where I am running ELK stack (I have installed Elastics Search, Logstash & kibana in a single server). I am using logstash-forwarder in my some applications. They are properly sending log to logstash and I can see the log in Kibana.

My problem is, One of my application is run in Iron.io infrastructure. They can send log using http. So I enabled http plugin in my ELK server. I want to show the log in Kibana which is sent from Iron.Io application using http.

This is my input config file:

input {  
lumberjack {  
port =\> 5000  
type =\> "logs"  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
}  
http {  
port =\> 5555  
user =\> myuser  
password =\> "\*\*\*\*\*\*"  
}  
}

Please provide me a valid request format so that I can send log to my ELK server using curl. 5555 port is open for all in my elk server. So please at least let me know how can I send a curl request from my local to send log that will be shown in kibana. If I can send from local then I will be able to configure in iron.io application.

Objective:

I will send a log to elk using http from local or other machine -\> Logstash will process the http request -\> Will show it in Kibana

I want to keep the lumberjack plugin in my input file as it is being used by those servers who are sending log using logstash-forwarder. beside that I want add http functionality

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2015, 7:26am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/6 "2015-10-30T07:26:32Z")

</div>

Okay, so you actually need the http input. So:

- Fix your broken JSON payload.
- Skip the authentication requirements (i.e. drop the `user` and `password` options for the http input) until you've verified that things work. Then authenticate properly (pass `-u myuser: ***** ` to curl).

I suggest you leave Elasticsearch out of the equation for now and use a `stdout { codec => rubydebug }` output to see what the events look like. It's quite possible that you want to process them in some manner before passing them off to ES).

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 30, 2015, 10:11am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/7 "2015-10-30T10:11:27Z")

</div>

I followed the process that you told me to do. I have generated certificate in my ELK server that I am using to verify my client who is sending log using logstash-forwarder. But for the http plugin, In the input section, I have not provided any authentication.

My current Input file:

input {  
lumberjack {  
port =\> 5000  
type =\> "logs"  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
}  
http {  
port = 5555}  
}

and I am sending request using below command:  
curl -H "content-type: application/json" -XPUT '[http://elk.startjobs.net:5555](http://elk.startjobs.net:5555)' -d '{  
"post\_date" : "2015-10-29T15:20:34",  
"message" : "Nowshad Amin" }' -v

and I am getting the below response:

- Adding handle: conn: 0x7f886a813600
- Adding handle: send: 0
- Adding handle: recv: 0
- Curl\_addHandleToPipeline: length: 1
- 
  - Conn 0 (0x7f886a813600) send\_pipe: 1, recv\_pipe: 0

- About to connect() to [elk.startjobs.net](http://elk.startjobs.net) port 5555 (#0)
- Trying 52.91.230.20...
- Connected to [elk.startjobs.net](http://elk.startjobs.net) (52.91.230.20) port 5555 (#0)

> PUT / HTTP/1.1  
> User-Agent: curl/7.30.0  
> Host: [elk.startjobs.net:5555](http://elk.startjobs.net:5555)  
> Accept: _/_  
> content-type: application/json  
> Content-Length: 67

- upload completely sent off: 67 out of 67 bytes  
\< HTTP/1.1 401 Unauthorized  
\< Content-Type: text/plain  
\< WWW-Authenticate: Basic realm=""  
\< Content-Length: 0  
\<
- Connection #0 to host [elk.startjobs.net](http://elk.startjobs.net) left intact

Since I have not enabled any authentication why I am getting 404 error.

Can we have any video chat. I am really stuck on it

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2015, 10:25am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/8 "2015-10-30T10:25:21Z")

</div>

> Since I have not enabled any authentication why I am getting 404 [sic] error.

It works fine for me. Are you sure it's Logstash that's responding on that port on that machine?

> Can we have any video chat. I am really stuck on it

I'm just a volunteer and I don't offer private video chats.

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 30, 2015, 11:25am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/9 "2015-10-30T11:25:49Z")

</div>

I tried it from ELk server using command:  
curl -H "content-type: application/json" -XPUT '[http://elk.startjobs.net:5555](http://elk.startjobs.net:5555)' -d '{  
"post\_date" : "2015-10-29T15:20:34","message" : "Nowshad\_amin" }' -v  
Response:

- Rebuilt URL to: [http://elk.startjobs.net:5555/](http://elk.startjobs.net:5555/)
- Hostname was NOT found in DNS cache
- Trying 172.32.0.157...
- Connected to [elk.startjobs.net](http://elk.startjobs.net) (172.32.0.157) port 5555 (#0)

> PUT / HTTP/1.1  
> User-Agent: curl/7.35.0  
> Host: [elk.startjobs.net:5555](http://elk.startjobs.net:5555)  
> Accept: _/_  
> content-type: application/json  
> Content-Length: 66

- upload completely sent off: 66 out of 66 bytes  
\< HTTP/1.1 401 Unauthorized  
\< Content-Type: text/plain  
\< WWW-Authenticate: Basic realm=""  
\< Content-Length: 0  
\<
- Connection #0 to host [elk.startjobs.net](http://elk.startjobs.net) left intact

and also  
curl -H "content-type: application/json" -XPUT '127.0.0.1:5555' -d '{  
"post\_date" : "2015-10-29T15:20:34","message" : "Nowshad\_amin" }' -v  
Response:

- Rebuilt URL to: 127.0.0.1:5555/
- Hostname was NOT found in DNS cache
- Trying 127.0.0.1...
- Connected to 127.0.0.1 (127.0.0.1) port 5555 (#0)

> PUT / HTTP/1.1  
> User-Agent: curl/7.35.0  
> Host: 127.0.0.1:5555  
> Accept: _/_  
> content-type: application/json  
> Content-Length: 66

- upload completely sent off: 66 out of 66 bytes  
\< HTTP/1.1 401 Unauthorized  
\< Content-Type: text/plain  
\< WWW-Authenticate: Basic realm=""  
\< Content-Length: 0  
\<
- Connection #0 to host 127.0.0.1 left intact

I am also tried to connect from my local machine. All time getting the same response (401)

Please take this into your consideration (For lumberjack I am using keys, but for http I am not using any keys and no authentication in my input file)

input {  
lumberjack {  
port =\> 5000  
type =\> "logs"

> **ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"**  
> **ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"**  
> }  
> http {  
> port = 5555}  
> }

Thanks

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 31, 2015, 5:47am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/10 "2015-10-31T05:47:45Z")

</div>

I tried it from ELk server using command:  
curl -H "content-type: application/json" -XPUT '[http://elk.example.net:5555](http://elk.example.net:5555)' -d '{  
"post\_date" : "2015-10-29T15:20:34","message" : "Nowshad\_amin" }' -v  
Response:

- Rebuilt URL to: [http://elk.example.net:5555/](http://elk.example.net:5555/)
- Hostname was NOT found in DNS cache
- Trying 172.32.0.157...
- Connected to [elk.startjobs.net](http://elk.startjobs.net) (172.32.0.157) port 5555 (#0)  
PUT / HTTP/1.1  
User-Agent: curl/7.35.0  
Host: [elk.example.net:5555](http://elk.example.net:5555)  
Accept: /  
content-type: application/json  
Content-Length: 66

upload completely sent off: 66 out of 66 bytes  
\< HTTP/1.1 401 Unauthorized  
\< Content-Type: text/plain  
\< WWW-Authenticate: Basic realm=""  
\< Content-Length: 0  
\<  
Connection #0 to host [elk.example.net](http://elk.example.net) left intact

Please take this into your consideration (For lumberjack I am using keys, but for http I am not using any keys and no authentication in my input file). I can send log using lumberjack but getting 401 response while sending log using http

input {  
lumberjack {  
port =\> 5000  
type =\> "logs"  
ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt"  
ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key"  
}  
http {  
port =\> 5555  
}  
}

Any help will be highly appreciated

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 31, 2015, 10:31am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/11 "2015-10-31T10:31:59Z")

</div>

Can I use multiple Input plugin in a single input file. Since I have enabled certificate for my lumberjack input so I am assuming it is creating problem for tcp though I have not enabled any authentication for http and tcp. Everytime I am trying to curl to elk from elk server/external server usinh http and tcp I am getting 401 Unauthorized. Please help me with your expert opinion

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 31, 2015, 10:50am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/12 "2015-10-31T10:50:03Z")

</div>

> Can I use multiple Input plugin in a single input file.

Logstash handles multiple inputs just fine. You can have more than one input per configuration file and you can have multiple configuration files. The input can be of the same type or of different types. The one things you should be careful about is having two identically configured inputs, i.e. two inputs listening on the same port or reading from the same file.

> Since I have enabled certificate for my lumberjack input so I am assuming it is creating problem for tcp though I have not enabled any authentication for http and tcp.

As mentioned before, make sure you're _actually_ connecting to Logstash in your curl commands. For example, if you shut down Logstash, are subsequent connections refused? Could there be some kind of proxy that intercepts the connections?

It's obviously very easy to check the hypothesis of the lumberjack input disturbing the http input; just start a Logstash instance with only an http input (and a `stdout { codec => rubydebug }` output) and see how that behaves. You can also try different port numbers to see if they behave differently.

---

<div class="post-metadata">

**Author:** ![aniamin](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@aniamin](https://discuss.elastic.co/u/aniamin)\
**Post date:** [October 31, 2015, 6:15pm UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/13 "2015-10-31T18:15:53Z")

</div>

I have tried different port for different input and got error everytime. My ELK server is [elk.startjobs.net](http://elk.startjobs.net) which is running fine and collecting logs from Servers that are using logstash-forwarder.

I have setup Elasticsearch Logstash and Kibana in a same Ubuntu Server.

For Lumberjack we provide ssl\_certificate and ssl\_key, Can we anyhow provide these certificate and keys in http?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:24am UTC](https://discuss.elastic.co/t/http-plugin-of-logstash/33257/14 "2017-07-06T05:24:43Z")

</div>