Can not get any logs after create a datastream

Elastic Stack Elasticsearch
,
1

Hi Everyone!

I am trying to change my data to the datastream. Normally I can get logs, I can see indice first. After I create a datastream I could not get any logs even though logs keep coming. I followed this steps:

I created a data stream like below:

{
  "data_streams" : [
    {
      "name" : "git-demo-topic",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-git-demo-topic-2021.12.13-000001",
          "index_uuid" : "y9bC8QZoQgijDKfp1zKO6Q"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "git-demo-topic",
      "hidden" : false,
      "system" : false
    }
  ]
}

This is the template a created:

{
  "index_templates" : [
    {
      "name" : "git-demo-topic",
      "index_template" : {
        "index_patterns" : [
          "git-demo-topic*"
        ],
        "template" : {
          "mappings" : {
            "properties" : {
              "city" : {
                "type" : "text"
              },
              "log_level" : {
                "type" : "text"
              },
              "detail" : {
                "type" : "text"
              },
              "ts" : {
                "type" : "text"
              }
            }
          }
        },
        "composed_of" : [ ],
        "data_stream" : {
          "hidden" : false
        }
      }
    }
  ]
}

This is the indice that created automatically, but it does not get any documents:

{
  ".ds-git-demo-topic-2021.12.13-000001" : {
    "aliases" : { },
    "mappings" : {
      "_data_stream_timestamp" : {
        "enabled" : true
      },
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "city" : {
          "type" : "text"
        },
        "detail" : {
          "type" : "text"
        },
        "log_level" : {
          "type" : "text"
        },
        "ts" : {
          "type" : "text"
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_hot"
            }
          }
        },
        "hidden" : "true",
        "number_of_shards" : "1",
        "provided_name" : ".ds-git-demo-topic-2021.12.13-000001",
        "creation_date" : "1639392538811",
        "number_of_replicas" : "1",
        "uuid" : "-LOWaaddSyWXXwiXfVk6GA",
        "version" : {
          "created" : "7150299"
        }
      }
    },
    "data_stream" : "git-demo-topic"
  }
}

Thank you in advance!

2

I found the problem. I had to add "action-> create" line to my logstash out conf:

  stdout {  
        codec => json_lines  
    } 
  elasticsearch {
      hosts => ["elk:9200"]
      index => "git-demo-topic"
	  action => "create"
  }
}`
3

The bulk and index api will fail with a client errors if the operation type is not create.
When using the bulk or index api directly this is immediately visible, but I guess this
is not easily visible when other components are indexing into a data stream.

When indexing data into a data stream only append-only writes are allowed and
the way this is enforced is by checking whether the operation type is create.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.