Python -> ElasticSearch Data Stream.. i'm doing something wrong.. suggestions

Elastic Stack Elasticsearch
1

i'm trying to write a pretty basic python script to bulk insert some ip blacklists into an elasticsearch data stream.

this is my code, its basic for now:

    def bulkESSubmit(self):
        try:
            count=1
            es = Elasticsearch(['http://<HOST>:9200'], http_auth=('<user>', '<password>'), timeout=100, max_retries=2, retry_on_timeout=True)
            actions = []
            indexName="firehol-bad-ips"

            for item in tqdm(self.BadIP_Dict):
                commonTags=self.buildCommonTags(self.BadIP_Dict[item])
                commonTags=list(set(commonTags))
                tempDict = {
                    '_index': indexName,
                    '_op_type': "create",
                    '_source': {
                        '@timestamp': datetime.now().replace(microsecond=0).isoformat(),
                        'createdAt': datetime.now().replace(microsecond=0).isoformat(),
                        'ipaddress': item,
                        'sources': list(set(self.BadIP_Dict[item])),
                        'tags' : list(set(commonTags)),
                    },
                }
                actions.append(tempDict)
                tempDict.clear()
                count += 1
            helpers.bulk(es, actions)

i've got the ILM set up, i've got the index template set up.. when i try to insert i get this error:
"'reason': 'only write ops with an op_type of create are allowed in data streams'"

i've looked for examples online.. i've found a few:

but nothing seems to work.. what am i missing?
any suggestions would be greatly appreciated

2

Do you get an error if you don't declare the _op_type (and let it default to index)?

3

that doesnt work either.. I tried that too.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.