Hi,
Being new to the datastream subject, I tried setting up a datastream using the documentation here: https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html and index towards it.
However, My logstash (7.9.1) is not able to deliver any data, even though I can see the datastream index is created. What am I doing wrong?
The message in logstash logging:
[2020-09-30T15:42:30,681][WARN ][logstash.outputs.elasticsearch][main][fa72e9983cda258dabc20ca0c215f0dab291c9be0d9e9affbd052bfc5d89e274] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"7620e77ee0e88c1f10558a0fe06cd42fc0d28044", :_index=>"agl-api-ds", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x3347c5b>], :response=>{"index"=>{"_index"=>"agl-api-ds", "_type"=>"_doc", "_id"=>"7620e77ee0e88c1f10558a0fe06cd42fc0d28044", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}
logstash output (metadata&indexname are translated to 'agl-api' ):
output {
if "api-log" in [tags] or "access-log" in [tags] or "tls-proxy" in [tags] {
elasticsearch {
hosts => ["esserver1.servers.local:9200","esserver2.servers.local:9200"]
document_id => "%{[@metadata][fingerprint]}"
index => "%{[@metadata][indexbasename]}-ds"
sniffing => false
}
}
}
I see a hidden index created for the datastream:
I have this template fir the datastream:
The request is:
PUT _index_template/agl-api-datastream-tmpl
{
"version": 1,
"priority": 200,
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "agl-data-stream-policy"
},
"refresh_interval": "30s",
"number_of_shards": "5",
"number_of_replicas": "1"
}
},
"mappings": {
"_routing": {
"required": false
},
"_source": {
"excludes": [],
"includes": [],
"enabled": true
},
"dynamic": false,
"properties": {
"ms-host": {
"norms": false,
"type": "keyword"
},
"api-version": {
"norms": false,
"type": "keyword"
},
"agent": {
"dynamic": true,
"type": "object",
"enabled": true,
"properties": {
"hostname": {
"norms": false,
"type": "keyword"
},
"name": {
"norms": false,
"type": "text"
},
"version": {
"norms": false,
"type": "text"
}
}
},
"api-platform": {
"norms": false,
"type": "keyword"
},
"api-url": {
"norms": false,
"type": "text"
},
"ms-error-message": {
"norms": false,
"type": "text"
},
"api-nanoservice": {
"norms": false,
"type": "keyword"
},
"source": {
"norms": false,
"type": "text"
},
"platform": {
"norms": false,
"type": "keyword"
},
"ms-result-code": {
"norms": false,
"type": "text"
},
"api-brand": {
"norms": false,
"type": "keyword"
},
"transaction-id": {
"norms": false,
"type": "text"
},
"@version": {
"norms": false,
"type": "keyword"
},
"host": {
"norms": false,
"type": "keyword"
},
"ms-request-body": {
"norms": false,
"type": "text"
},
"thread-id": {
"type": "integer"
},
"additional-data": {
"norms": false,
"type": "text"
},
"log-message": {
"norms": false,
"type": "text"
},
"ms-url": {
"norms": false,
"type": "text"
},
"api-type": {
"norms": false,
"type": "keyword"
},
"app-id": {
"norms": false,
"type": "keyword"
},
"offset": {
"type": "long"
},
"event-type": {
"norms": false,
"type": "keyword"
},
"user-id": {
"norms": false,
"type": "text"
},
"client-ip": {
"type": "ip"
},
"prospector": {
"type": "object",
"properties": {
"type": {
"norms": false,
"type": "text"
}
}
},
"api-query-string": {
"norms": false,
"type": "text"
},
"message": {
"norms": false,
"type": "text"
},
"tags": {
"norms": false,
"type": "text"
},
"api-tenant": {
"norms": false,
"type": "keyword"
},
"@timestamp": {
"type": "date"
},
"ms-result": {
"norms": false,
"type": "keyword"
},
"log-level": {
"norms": false,
"type": "keyword"
},
"execution-time": {
"type": "integer"
},
"fields": {
"type": "object",
"properties": {
"environment": {
"norms": false,
"type": "keyword"
}
}
},
"http-verb": {
"norms": false,
"type": "keyword"
},
"session-id": {
"norms": false,
"type": "text"
},
"username": {
"norms": false,
"type": "text"
}
}
}
},
"index_patterns": [
"agl-api-ds"
],
"data_stream": {},
"composed_of": []
}
And there is this 1 lifecycle policy nicely connected to the hidden index.
So somehow everything looks to me as ok and yet I cannot get datainto it.
What am I doing wrong?
For the record, when I index to a different indexname (non-existing) that index is created and indeed filling with data, so the connection is OK. It's something with that datastream.
Any help is appreciated