# Datastream gets no data
**URL:** https://discuss.elastic.co/t/datastream-gets-no-data/250516
**Category:** Elasticsearch
**Created:** [September 30, 2020, 1:54pm UTC](https://discuss.elastic.co/t/datastream-gets-no-data/250516 "2020-09-30T13:54:37Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Tuckson](https://discuss.elastic.co/u/Tuckson)
#### Post date: [September 30, 2020, 1:54pm UTC](https://discuss.elastic.co/t/datastream-gets-no-data/250516/1 "2020-09-30T13:54:37Z")
Hi,
Being new to the datastream subject, I tried setting up a datastream using the documentation here: [https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-a-data-stream.html) and index towards it.
However, My logstash (7.9.1) is not able to deliver any data, even though I can see the datastream index is created. What am I doing wrong?
The message in logstash logging:
```auto
[2020-09-30T15:42:30,681][WARN][logstash.outputs.elasticsearch][main][fa72e9983cda258dabc20ca0c215f0dab291c9be0d9e9affbd052bfc5d89e274] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"7620e77ee0e88c1f10558a0fe06cd42fc0d28044", :_index=>"agl-api-ds", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"agl-api-ds", "_type"=>"_doc", "_id"=>"7620e77ee0e88c1f10558a0fe06cd42fc0d28044", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"only write ops with an op_type of create are allowed in data streams"}}}}
```
logstash output (metadata&indexname are translated to 'agl-api' ):
```auto
output {
if "api-log" in [tags] or "access-log" in [tags] or "tls-proxy" in [tags] {
elasticsearch {
hosts => ["esserver1.servers.local:9200","esserver2.servers.local:9200"]
document_id => "%{[@metadata][fingerprint]}"
index => "%{[@metadata][indexbasename]}-ds"
sniffing => false
}
}
}
```
I see a hidden index created for the datastream:
I have this template fir the datastream:
The request is:
```auto
PUT _index_template/agl-api-datastream-tmpl
{
"version": 1,
"priority": 200,
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "agl-data-stream-policy"
},
"refresh_interval": "30s",
"number_of_shards": "5",
"number_of_replicas": "1"
}
},
"mappings": {
"_routing": {
"required": false
},
"_source": {
"excludes": [],
"includes": [],
"enabled": true
},
"dynamic": false,
"properties": {
"ms-host": {
"norms": false,
"type": "keyword"
},
"api-version": {
"norms": false,
"type": "keyword"
},
"agent": {
"dynamic": true,
"type": "object",
"enabled": true,
"properties": {
"hostname": {
"norms": false,
"type": "keyword"
},
"name": {
"norms": false,
"type": "text"
},
"version": {
"norms": false,
"type": "text"
}
}
},
"api-platform": {
"norms": false,
"type": "keyword"
},
"api-url": {
"norms": false,
"type": "text"
},
"ms-error-message": {
"norms": false,
"type": "text"
},
"api-nanoservice": {
"norms": false,
"type": "keyword"
},
"source": {
"norms": false,
"type": "text"
},
"platform": {
"norms": false,
"type": "keyword"
},
"ms-result-code": {
"norms": false,
"type": "text"
},
"api-brand": {
"norms": false,
"type": "keyword"
},
"transaction-id": {
"norms": false,
"type": "text"
},
"@version": {
"norms": false,
"type": "keyword"
},
"host": {
"norms": false,
"type": "keyword"
},
"ms-request-body": {
"norms": false,
"type": "text"
},
"thread-id": {
"type": "integer"
},
"additional-data": {
"norms": false,
"type": "text"
},
"log-message": {
"norms": false,
"type": "text"
},
"ms-url": {
"norms": false,
"type": "text"
},
"api-type": {
"norms": false,
"type": "keyword"
},
"app-id": {
"norms": false,
"type": "keyword"
},
"offset": {
"type": "long"
},
"event-type": {
"norms": false,
"type": "keyword"
},
"user-id": {
"norms": false,
"type": "text"
},
"client-ip": {
"type": "ip"
},
"prospector": {
"type": "object",
"properties": {
"type": {
"norms": false,
"type": "text"
}
}
},
"api-query-string": {
"norms": false,
"type": "text"
},
"message": {
"norms": false,
"type": "text"
},
"tags": {
"norms": false,
"type": "text"
},
"api-tenant": {
"norms": false,
"type": "keyword"
},
"@timestamp": {
"type": "date"
},
"ms-result": {
"norms": false,
"type": "keyword"
},
"log-level": {
"norms": false,
"type": "keyword"
},
"execution-time": {
"type": "integer"
},
"fields": {
"type": "object",
"properties": {
"environment": {
"norms": false,
"type": "keyword"
}
}
},
"http-verb": {
"norms": false,
"type": "keyword"
},
"session-id": {
"norms": false,
"type": "text"
},
"username": {
"norms": false,
"type": "text"
}
}
}
},
"index_patterns": [
"agl-api-ds"
],
"data_stream": {},
"composed_of": []
}
```
And there is this 1 lifecycle policy nicely connected to the hidden index.
So somehow everything looks to me as ok and yet I cannot get datainto it.
What am I doing wrong?
For the record, when I index to a different indexname (non-existing) that index is created and indeed filling with data, so the connection is OK. It's something with that datastream.
Any help is appreciated
---
### Author:  [@Tuckson](https://discuss.elastic.co/u/Tuckson)
#### Post date: [September 30, 2020, 2:05pm UTC](https://discuss.elastic.co/t/datastream-gets-no-data/250516/2 "2020-09-30T14:05:58Z")
> [@Tuckson](#):
>
> `only write ops with an op_type of create are allowed in data streams`
I learned that I actually should use a different output plugin in Logstash, so I changed the output conf to:
```auto
output {
if "api-log" in [tags] or "access-log" in [tags] or "tls-proxy" in [tags] {
elasticsearch_data_stream{
hosts => ["esserver1.servers.local:9200","esserver2.servers.local:9200"]
document_id => "%{[@metadata][fingerprint]}"
index => "%{[@metadata][indexbasename]}-ds"
sniffing => false
}
}
}
```
However this gave:
```auto
[2020-09-30T16:02:52,175][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'elasticsearch_data_stream'. Are you sure this is correct? Trying to load the elasticsearch_data_stream output plugin resulted in this error: no such file to load -- logstash/outputs/elasticsearch_data_stream", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:82)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$ __VARARGS__ (/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
[2020-09-30T16:02:52,183][ERROR][logstash.agent] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create`"}
[2020-09-30T16:02:52,223][FATAL][logstash.runner] An unexpected error occurred! {:error=>#`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:370:in `block in converge_state'"]}
[2020-09-30T16:02:52,241][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
```
Got the idea from here:
>
>
> \## Overview
> This is an overview of the Logstash integration with Elasticsearch …data streams. The integration will be added as a feature to the existing \[Elasticsearch output plugin\](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html). This will include new data stream options that will be recommended for indexing any time series datasets (logs, metrics, etc.) into Elasticsearch. The existing options will continue to be used for non-time series use cases. This feature will be available on both the default and OSS Logstash distributions.
>
> \## Indexing Strategy
> The data streams integration will adopt the \[new indexing strategy\](https://github.com/elastic/kibana/blob/f91c795e30b7221cdce36e235470a4cda82f70b2/docs/ingest\_manager/index.asciidoc#indexing-strategy-1) under the \`{type}-{dataset}-{namespace}\` format, leveraging the \[composable templates bundled in Elasticsearch\](https://github.com/elastic/elasticsearch/pull/57629) starting in 7.9.
>
> The default data streams name used will be \`logs-generic-default\`. This default enables users to easily correlate data with other different data sources (e.g. with \`logs-\*\` and \`logs-generic-\*\`) in Elasticsearch. Given the new indexing strategy, the \`type\`, \`dataset\`, and \`namespace\` of the data stream name can all be configured separately.
>
> As Logstash will not be fully \[ECS compliant\](https://github.com/elastic/logstash/issues/11635) until 8.0, there are caveats we need to document (or provide bootstrap checks) for users to avoid ECS conflicts.
> \- Update the Beats input, TCP input, UDP input, and grok filter. If they are using these plugins, they should enable ECS compatibility mode to avoid ECS conflicts. This is work in progress for the 7.9 / 7.10 timeframe.
> \- Users should not introduce any ECS conflicting fields in their pipeline when using this plugin. This should be more systematic in the future when we add ECS validation.
>
> \## Example Configuration
> \### Basic default configuration
> \`\`\`
> output {
> elasticsearch {
> hosts =\> "hostname"
> data\_stream =\> "true"
> }
> }
> \`\`\`
>
> Minimal settings to get started in Logstash 7.x. Events with the \`data\_stream.\*\` fields will automatically get routed to the appropriate data streams. Defaults to \`logs-generic-logstash\` if the fields are missing.
>
> \### Customize data stream name
> \`\`\`
> output {
> elasticsearch {
> hosts =\> "hostname"
> data\_stream =\> "true"
> data\_stream\_timestamp =\> "@timestamp"
> data\_stream\_type =\> "metrics"
> data\_stream\_dataset =\> "foo"
> data\_stream\_namespace =\> "bar"
> }
> }
> \`\`\`
>
> \## Configuration Settings
> These are the net new data stream specific settings that will be added to the Elasticsearch output plugin:
> \- \`data\_stream\` (string, optional) - defines whether data will be indexed into an Elasticsearch data stream. The \`data\_stream\_\*\` settings will only be used if this setting is enabled. This setting supports the values \`true\`, \`false\`, and \`auto\`. Defaults to \`false\` in Logstash 7.x and \`auto\` starting in Logstash 8.0. More details on the \`auto\` behavior can be found in this \[issue\](https://github.com/elastic/dev/issues/1571#issuecomment-739029321).
> \- \`data\_stream\_timestamp\` (timestamp, required) - the timestamp used for the data stream. Defaults to \`@timestamp\`.
> \- \`data\_stream\_type\` (string, optional) - the data stream type used to construct the data stream at index time. Only \`logs\` or \`metrics\` is allowed. This field does not support hyphens (-). Defaults to \`logs\`.
> \- \`data\_stream\_dataset\` (string, optional) - the data stream dataset used to construct the data stream at index time. This field does not support hyphens (-). Defaults to \`generic\`.
> \- \`data\_stream\_namespace\` (string, optional) - the data stream namespace used to construct the data stream at index time. This field does not support hyphens (-). Defaults to \`default\`.
> \- \`data\_stream\_auto\_routing\` (boolean, optional) - automatically routes events by deriving the data stream name using specific event fields with the \`%{data\_stream.type}-%{data\_stream.dataset}-%{data\_stream.namespace}\` format. If enabled, the \`data\_stream.\*\` event fields will take precedence over the \`data\_stream\_type\`, \`data\_stream\_dataset\`, and \`data\_stream\_namespace\` settings, but will fall back to them if any of the fields are missing from the event. Defaults to \`true\`.
> \- \`data\_stream\_sync\_fields\` (boolean, optional) - automatically syncs the \`data\_stream.\*\` event fields if they are missing from the event. This ensures the \`data\_stream.\*\` fields match the data stream name that events are indexed to. The field syncing behavior between this setting and the \`data\_stream\_auto\_routing\` setting can be found in this \[issue\](https://github.com/logstash-plugins/logstash-output-elasticsearch\_data\_streams/issues/2#issue-740088457). Defaults to \`true\`.
>
> \## Elastic Agent Compatibility
> Logstash often acts as an intermediary for receiving data from other systems like the Elastic Agent and Kafka. For these use cases, Logstash will by default use the \`data\_stream.type\`, \`data\_stream.dataset\`, and \`data\_stream.namespace\` event fields to derive the data stream name. This allows events from the Elastic Agent to automatically be routed to the appropriate Elasticsearch data stream when using Logstash in between. This feature can be disabled by configuring the \`data\_stream\_auto\_routing\` setting to \`false\`.
>
> Format: \`%{data\_stream.type}-%{data\_stream.dataset}-%{data\_stream.namespace}\`
>
> Events received from the Elastic Agent should generally have all the \`data\_stream.\*\` fields populated. In the case where any of these fields are missing, the \`data\_stream\_sync\_fields\` setting will be used to sync these fields prior to indexing.
>
> \## Limitations
> The primary limitation of data streams is the ability to perform updates to the documents. Logstash users have historically used the existing Elasticsearch output plugin’s capabilities to conduct document updates and achieve exactly once delivery semantics.
>
> \## Future Considerations
> \- The \`logs-generic-default\` is the default data stream for generic data from Logstash and the Elastic Agent. If users express feedback that it’s difficult to identify Logstash sourced data from the shared data stream, we could consider adding a \`from-logstash\` tag to the \`tags\` ECS base field for events coming from Logstash.
> \- We want to guide users towards using the new indexing strategy, but if users express the need for more flexibility, we could introduce a free form option for specifying the data stream name in the future where template/ILM management would be manual.
But appearantly this is not a standard part of Logstash?
---
### Author:  [@Tuckson](https://discuss.elastic.co/u/Tuckson)
#### Post date: [September 30, 2020, 2:22pm UTC](https://discuss.elastic.co/t/datastream-gets-no-data/250516/3 "2020-09-30T14:22:18Z")
OK.
Seems to work partly now.
Reverted to elasticsearch output plugin and set the 'action'to "create" (i.s.o.the default 'index')
I recieve data now.
Hoewever, I see these to lines in logstash log:
```auto
[2020-09-30T16:19:25,834][WARN][logstash.outputs.elasticsearch][main][ef8f8fd38026ee6b4cb399a9d9dcfe8440065e0aca25750d4cb1f8bb8192b208] Failed action. {:status=>409, :action=>["create", {:_id=>"19319400d8fbb7c0069c70df9b178ccc3860dc2d", :_index=>"agl-api-ds", :routing=>nil, :_type=>"_doc"}, #], :response=>{"create"=>{"_index"=>".ds-agl-api-ds-000001", "_type"=>"_doc", "_id"=>"19319400d8fbb7c0069c70df9b178ccc3860dc2d", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[19319400d8fbb7c0069c70df9b178ccc3860dc2d]: version conflict, document already exists (current version [1])", "index_uuid"=>"h9zN5pZvSgm8ltTfdXesRg", "shard"=>"0", "index"=>".ds-agl-api-ds-000001"}}}}
[2020-09-30T16:19:33,327][WARN][logstash.outputs.elasticsearch][main][ef8f8fd38026ee6b4cb399a9d9dcfe8440065e0aca25750d4cb1f8bb8192b208] Failed action. {:status=>409, :action=>["create", {:_id=>"0645733540c1642af821aa78a270114dbe1603e1", :_index=>"agl-api-ds", :routing=>nil, :_type=>"_doc"}, #], :response=>{"create"=>{"_index"=>".ds-agl-api-ds-000001", "_type"=>"_doc", "_id"=>"0645733540c1642af821aa78a270114dbe1603e1", "status"=>409, "error"=>{"type"=>"version_conflict_engine_exception", "reason"=>"[0645733540c1642af821aa78a270114dbe1603e1]: version conflict, document already exists (current version [1])", "index_uuid"=>"h9zN5pZvSgm8ltTfdXesRg", "shard"=>"2", "index"=>".ds-agl-api-ds-000001"}}}}
```
How can I prevent these?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 28, 2020, 2:22pm UTC](https://discuss.elastic.co/t/datastream-gets-no-data/250516/4 "2020-10-28T14:22:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.