# Unable to ship data to logstash 6.8 via Python

**URL:** <https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677>\
**Category:** Logstash\
**Created:** [July 2, 2020, 4:36pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677 "2020-07-02T16:36:04Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Swetha\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swetha_b/32/71544_2.png) [@Swetha\_B](https://discuss.elastic.co/u/Swetha_B)\
**Post date:** [July 2, 2020, 4:36pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/1 "2020-07-02T16:36:04Z")

</div>

We have a Python script to collect data and ship to Logstash. Please find the python script and logstash config below.

```auto
import socket
import json
import logging
from datetime import datetime
import sys

print("starting to send data to Elastic search")
# Create TCP/IP socket
print("Creating TCP/IP socket")
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
message = []
try:
    # Connect to port where server is running
    server_address = ('<host>', 50504)
    sock.connect(server_address)
    data = {'@test' : 'test1', '@message': 'python test message', '@tags': ['python', 'test']}
    sock.sendall(json.dumps(data).encode())
    print("Sent")
except socket.error as e:
    sys.stderr.write(str(e))
finally:
    sock.close()

```

Logstash conf looks like the following:

```auto
 input {
   tcp {
       port => 50504
       type => "xxx"
       id => "yyy"
       codec => json
   }
 }
 
 filter {
     if [type] == "xxx" {
             json {
                 source => "message"
               }
   }
 
   date {
           match => ["time", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ", "ISO8601", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
        }
 
 }
 
 output {
              elasticsearch {
                                 hosts => ["<hosts>:4080"]
                                 index => "logstash-%{[type]}-s3-%{+YYYY.MM.dd}"
                                 id => "zzz"
                            }
       }

```

Getting the following error on logstash log:

```auto
[2020-07-02T16:08:12,176][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-xxx-s3-2020.07.02", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x119d0bf>], :response=>{"index"=>{"_index"=>"logstash-convergence-status-s3-2020.07.02", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information."}}}}

```

Wanted to check if python could be used for shipping logs to logstash from version 6.8 and above.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2020, 4:53pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/2 "2020-07-02T16:53:09Z")

</div>

> [@Swetha\_B](#):
>
> `reason"=>"[_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information."`

That appears to be version 7.0 or above, not 6.8.

You have an index template that includes a \_default\_ mapping, which was deprecated in 6.0 and results in an error in 7.0 or above. Update the template and remove the \_default\_ mapping.

---

<div class="post-metadata">

**Author:** ![Swetha\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swetha_b/32/71544_2.png) [@Swetha\_B](https://discuss.elastic.co/u/Swetha_B)\
**Post date:** [July 2, 2020, 5:05pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/3 "2020-07-02T17:05:13Z")

</div>

The logstash version is logstash-6.8.6-1.noarch and elasticsearch version is elasticsearch-7.7.1-1.x86\_64

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2020, 5:11pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/4 "2020-07-02T17:11:01Z")

</div>

> [@Swetha\_B](#):
>
> elasticsearch version is elasticsearch-7.7.1-1.x86\_64

OK, so update the template and remove the \_default\_ mapping.

---

<div class="post-metadata">

**Author:** ![Swetha\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swetha_b/32/71544_2.png) [@Swetha\_B](https://discuss.elastic.co/u/Swetha_B)\
**Post date:** [July 2, 2020, 6:00pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/5 "2020-07-02T18:00:04Z")

</div>

The template was already updated as part of ELK-7 upgrade. Following is the current elasticsearch template.

```auto
curl -XGET localhost:4080/_template/xxx?pretty
{
  "xxx" : {
    "order" : 0,
    "index_patterns" : [
      "xxx-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "2",
      }
    },
    "mappings" : {
      "properties" : {
        "hostname" : {
          "type" : "keyword"
        },
        "@timestamp" : {
          "format" : "strict_date_optional_time||epoch_millis",
          "type" : "date"
        },
        "port" : {
          "type" : "long"
        },
        "logdate" : {
          "format" : "strict_date_optional_time||epoch_millis",
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "type" : {
          "type" : "keyword"
        }
      }
    },
    "aliases" : { }
  }
}

```

Note:  
Logstash was upgraded to logstash-7.7.1-1.x86\_64 and was restarted. It was giving the same error.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2020, 6:03pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/6 "2020-07-02T18:03:05Z")

</div>

What do you get from

```
curl -XGET localhost:4080/_template/xxx?include_type_name
```

---

<div class="post-metadata">

**Author:** ![Swetha\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swetha_b/32/71544_2.png) [@Swetha\_B](https://discuss.elastic.co/u/Swetha_B)\
**Post date:** [July 2, 2020, 6:08pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/7 "2020-07-02T18:08:43Z")

</div>

```auto
{
  "xxx": {
    "order": 0,
    "index_patterns": [
      "xxx-*"
    ],
    "settings": {
      "index": {
        "number_of_shards": "2",
      }
    },
    "mappings": {
      "_doc": {
        "properties": {
          "hostname": {
            "type": "keyword"
          },
          "@timestamp": {
            "format": "strict_date_optional_time||epoch_millis",
            "type": "date"
          },
          "port": {
            "type": "long"
          },
          "logdate": {
            "format": "strict_date_optional_time||epoch_millis",
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "type": {
            "type": "keyword"
          }
        }
      }
    },
    "aliases": {}
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2020, 6:28pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/8 "2020-07-02T18:28:26Z")

</div>

Try setting

```
manage_template => false

```

on the elasticsearch output. It may be trying to load the [6.x template](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template-es6x.json), which includes \_default\_.

---

<div class="post-metadata">

**Author:** ![Swetha\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swetha_b/32/71544_2.png) [@Swetha\_B](https://discuss.elastic.co/u/Swetha_B)\
**Post date:** [July 2, 2020, 6:32pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/9 "2020-07-02T18:32:25Z")

</div>

Tried adding the above in logstash conf

```auto
output {
		    		 elasticsearch {
                                hosts => ["<host>:4080"]
                                index => "logstash-%{[type]}-s3-%{+YYYY.MM.dd}"
                                id => "zzz"
                                http_compression => true
                                manage_template => false
                           }
      }

```

Returning the same error in the logs

```auto
[2020-07-02T18:30:07,054][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-xxx-s3-2020.07.02", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x62492c0f>], :response=>{"index"=>{"_index"=>"logstash-xxx-s3-2020.07.02", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"[_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information."}}}}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2020, 7:03pm UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/10 "2020-07-02T19:03:03Z")

</div>

OK, check the output of

```
curl -XGET localhost:4080/logstash-xxx-s3-2020.07.02/_mapping?include_type name

```

Are there any other templates that would be getting applied? Check

```
curl -XGET localhost:4080/_template

```

Do you get an error for

```
curl -XGET localhost:4080/_index_template
```

---

<div class="post-metadata">

**Author:** ![Swetha\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swetha_b/32/71544_2.png) [@Swetha\_B](https://discuss.elastic.co/u/Swetha_B)\
**Post date:** [July 3, 2020, 4:32am UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/11 "2020-07-03T04:32:34Z")

</div>

logstash-xxx-s3-2020.07.02 index isn't getting created on elasticsearch end.

There was one overriding template which had the index pattern as logstash-\*

```auto
{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

```

After that template was deleted, the index got created and data started flowing in.

Thank you for the help @Badger

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 31, 2020, 4:46am UTC](https://discuss.elastic.co/t/unable-to-ship-data-to-logstash-6-8-via-python/239677/12 "2020-07-31T04:46:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.