# Push data into elasticsearch indices from logstash via http
**URL:** https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183
**Category:** Logstash
**Created:** [November 2, 2018, 2:32pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183 "2018-11-02T14:32:39Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@BoffinPanda](https://discuss.elastic.co/u/BoffinPanda)
#### Post date: [November 2, 2018, 2:32pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/1 "2018-11-02T14:32:39Z")
I am trying to push my data which I am collecting from various sensors into elasticsearch.
I am able to push my data directly into elasticsearch indices using the post method.
I am now attempting to push the data via logstash.
For this I have created a logstash conf as :
input {
http {
port =\> 9600
response\_headers =\> {
"Access-Control-Allow-Origin" =\> "\*"
"Content-Type" =\> "text/json"
"Access-Control-Allow-Headers" =\> "Origin, X-Requested-With, Content-Type, Accept"
}
}
}
output {
elasticsearch {
index =\> "panda-%{+YYYY.MM.dd}"
document\_type =\> "%{[@metadata][\_type]}"
document\_id =\> "%{[@metadata][\_id]}"
}
}
When I post data using postman I see that data is being pushed but the previous data is being lost and the doc count is only one. How to eliminate this problem to store all data?
The output is :
{
"took": 2,
"timed\_out": false,
"\_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max\_score": 1,
"hits": [
{
"\_index": "panda-2018.11.02",
"\_type": "%{[@metadata][\_type]}",
"\_id": "%{[@metadata][\_id]}",
"\_score": 1,
"\_source": {
"@version": "1",
"@timestamp": "2018-11-02T14:23:42.469Z",
"message": "",
"headers": {
"cache\_control": "no-cache",
"http\_host": "localhost:9600",
"request\_path": "/",
"http\_accept": "_/_",
"postman\_token": "61020876-8a8f-428b-8d99-af0ae669574a",
"connection": "keep-alive",
"accept\_encoding": "gzip, deflate",
"request\_method": "POST",
"test": "duper",
"content\_length": "0",
"http\_user\_agent": "PostmanRuntime/7.2.0",
"http\_version": "HTTP/1.1"
},
"host": "0:0:0:0:0:0:0:1"
}
}
]
}
}
Note: I have not created the elasticsearch index. I am letting logstash create it.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 2, 2018, 2:40pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/2 "2018-11-02T14:40:31Z")
> [@BoffinPanda](#):
>
> document\_type =\> "%{[@metadata][\_type]}"
> document\_id =\> "%{[@metadata][\_id]}"
As you do not seem to have these metadata fields set, all documents will be indexed using the same ID. Remove these lines and issue should be resolved.
---
### Author:  [@BoffinPanda](https://discuss.elastic.co/u/BoffinPanda)
#### Post date: [November 2, 2018, 2:42pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/3 "2018-11-02T14:42:32Z")
If I remove these, I will get a warning and I see no data at all!
---
### Author:  [@BoffinPanda](https://discuss.elastic.co/u/BoffinPanda)
#### Post date: [November 2, 2018, 2:44pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/4 "2018-11-02T14:44:01Z")
[WARN] 2018-11-02 20:13:25.882 [Ruby-0-Thread-8@[main]\>worker2: :1] elasticsearch - Could not index event to Elasticsearch. {:status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>"kenwin-2018.11.02", :\_type=\>"doc", :\_routing=\>nil}, #LogStash::Event:0x1ff44dcc], :response=\>{"index"=\>{"\_index"=\>"kenwin-2018.11.02", "\_type"=\>"doc", "\_id"=\>"YvXg1GYBjBuPNLEf-h8I", "status"=\>400, "error"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Rejecting mapping update to [kenwin-2018.11.02] as the final mapping would have more than 1 type: [%{[@metadata][\_type]}, doc]"}}}}
Upon removing those, I see no data at all!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 2, 2018, 2:48pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/5 "2018-11-02T14:48:14Z")
That is because you have already got documents created with the incorrect type `%{[@metadata][_type]}`. I would recommend deleting the index and let Logstash create a new one.
---
### Author:  [@BoffinPanda](https://discuss.elastic.co/u/BoffinPanda)
#### Post date: [November 2, 2018, 3:17pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/6 "2018-11-02T15:17:22Z")
It worked after deleting.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 30, 2018, 3:17pm UTC](https://discuss.elastic.co/t/push-data-into-elasticsearch-indices-from-logstash-via-http/155183/7 "2018-11-30T15:17:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.