# Trying to index from Logstash Kubernetes pod into an Elasticsearch Docker container
**URL:** https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991
**Category:** Elasticsearch
**Tags:** docker
**Created:** [January 29, 2024, 12:58pm UTC](https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991 "2024-01-29T12:58:12Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@rowish](https://discuss.elastic.co/u/rowish)
#### Post date: [January 29, 2024, 12:58pm UTC](https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991/1 "2024-01-29T12:58:12Z")
On the same cloud, I have an instance of Logstash 6.6.1 running within a certain Kubernetes pod that I am trying to make it index into an Elasticsearch 8.6.1 running on a Docker container hosted on an VM with IP address 10.85.173.33.
Testing a manual indexing from my laptop was a success:
```auto
$ curl -vvk -XPOST 'https://10.85.173.33:9200/_bulk?refresh&pretty' -H 'Content-Type: application/json' -u my_user:my_pass --data-binary "@my_doc.json"
Note: Unnecessary use of -X or --request, POST is already inferred.
* processing: https://10.85.173.33:9200/_bulk?refresh&pretty
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 10.85.173.33:9200...
* Connected to 10.85.173.33 (10.85.173.33) port 9200
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* using HTTP/1.x
* Server auth using Basic with user 'my_user'
> POST /_bulk?refresh&pretty HTTP/1.1
> Host: 10.85.173.33:9200
> Authorization: Basic ZWxhc3RpYzppdGVyZ28yMDIz
> User-Agent: curl/8.2.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 1428
>
} [1428 bytes data]
* schannel: remote party requests renegotiation
* schannel: renegotiating SSL/TLS connection
* schannel: SSL/TLS connection renegotiated
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 450
<
{ [450 bytes data]
100 1878 100 450 100 1428 1058 3359 --:--:-- --:--:-- --:--:-- 4429{
"took" : 77,
"errors" : false,
"items" : [
{
"index" : {
"_index" : "logstash-2024.01.29",
"_id" : "8881-IwBnfFXgAFtvWKK",
"_version" : 1,
"result" : "created",
"forced_refresh" : true,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 58552,
"_primary_term" : 1,
"status" : 201
}
}
]
}
* Connection #0 to host 10.85.173.33 left intact
```
I am trying with the following pipeline output output configuration:
```auto
output {
elasticsearch {
document_id => "%{document_id}"
hosts => ["https://10.85.173.33:9200"]
user => "my_user"
password => "my_pass"
ssl => true
ssl_certificate_verification => false
index => "logstash-%{+YYYY.MM.dd}"
manage_template => true
template => "/usr/share/logstash/templates/logstash.json"
template_name => "logstash"
template_overwrite => true
retry_on_conflict => 5
cacert => "/usr/share/logstash/config/ca.crt"
}
stdout { codec => rubydebug }
}
```
Nothing gets to Elasticsearch and no information is logged.
The Logstash log keeps on outputting:
`[2024-01-29T12:54:19,828][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.85.173.33:9200/_bulk"}`
I appreciate any hint of how can I make this work.
---
### Author:  [@Julien](https://discuss.elastic.co/u/Julien)
#### Post date: [January 29, 2024, 2:37pm UTC](https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991/2 "2024-01-29T14:37:10Z")
Per [elasticsearch output retry policy](https://www.elastic.co/guide/en/logstash/8.12/plugins-outputs-elasticsearch.html#_retry_policy) 400 errors are not retriable and the event will be sent to dead-letter queue if enabled and dismissed
The curl command with `@my_doc.json` works means that logstash is not sending the same bulk definition as this file content
To find out what document data is rejected to elasticsearch ; you will need to either :
- enable [dead letter queue](https://www.elastic.co/guide/en/logstash/8.12/dead-letter-queues.html) to check why the event is rejected by elasticsearch
- potentially easier : set elasticsearch output plugin log level in [debug log level](https://www.elastic.co/guide/en/logstash/8.12/logging.html) - this should contain the document and likely the response body of the 400 responses
- another option to check how the events are sent to the output plugin is to add a stdout or file output plugin
---
### Author:  [@rowish](https://discuss.elastic.co/u/rowish)
#### Post date: [January 30, 2024, 4:05pm UTC](https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991/3 "2024-01-30T16:05:34Z")
Thanks for suggestions, Julien. Indeed, Logstash 6.6.1 seems to add metadata fields that Elasticsearch 8.6.1 do not like.
`[2024-01-30T15:59:06,766][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.85.173.33:9200/_bulk", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}"}`
The question now is how do I get rid of that '\_type' field, since mutate filter don't seem to work.
---
### Author:  [@Julien](https://discuss.elastic.co/u/Julien)
#### Post date: [January 31, 2024, 10:04am UTC](https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991/4 "2024-01-31T10:04:42Z")
> The question now is how do I get rid of that '\_type' field, since mutate filter don't seem to work.
The value is added by the elasticsearch output plugin which run after all filters so this is not directly related to logstash version (using logstash 6.6.1 along with elasticsearch 8.x is not recommended/supported generally - you should be using latest logstash version in same major release per [compatibility matrix](https://www.elastic.co/support/matrix#matrix_compatibility) so currently that would be logstash `8.12.0` which will happen to also include more recent plugin versions which will really be what will solve your issue)
Current elasticsearch output plugin version documentation : [Elasticsearch output plugin | Logstash Reference [8.12] | Elastic](https://www.elastic.co/guide/en/logstash/8.12/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_type) -\> `for elasticsearch clusters 8.x: no value will be used;`
I assume you never updated elasticsearch output plugin so the behaviour is fully expected if you look at the same doc released at time of 6.6.0 - [Elasticsearch output plugin | Logstash Reference [6.6] | Elastic](https://www.elastic.co/guide/en/logstash/6.6/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_type) -\> `for elasticsearch clusters 6.x and above: the value of doc will be used`
Also check out how to update logstash plugins to test to latest elasticsearch ouptut plugin if you need logstash 6.6 for any reason : [Working with plugins | Logstash Reference [6.6] | Elastic](https://www.elastic.co/guide/en/logstash/6.6/working-with-plugins.html#updating-plugins) (you might still need to update the logstash core to be able to update some of the plugin to latest release depending on each plugin dependencies)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 28, 2024, 10:05am UTC](https://discuss.elastic.co/t/trying-to-index-from-logstash-kubernetes-pod-into-an-elasticsearch-docker-container/351991/5 "2024-02-28T10:05:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.