# Why default index created logstash-2022.01.01-000001

**URL:** <https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632>\
**Category:** Logstash\
**Created:** [October 14, 2022, 11:50am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632 "2022-10-14T11:50:25Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [October 14, 2022, 11:50am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/1 "2022-10-14T11:50:25Z")

</div>

I'm using filebeat for forwarding logs to logstash and creating multiple datastreams in elasticsearch. I noticed one index created with name "logstash-2022.01.01-000001" and storing some of data from different datasteams. Why is it so ? Can anyone help in uderstanding the behaviour ?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 17, 2022, 3:33am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/2 "2022-10-17T03:33:33Z")

</div>

Hi @connectgeeks

I suggest post your configurations (Filebeat + Logstash) , and some samples and the results then perhaps someone can help answer the question.

---

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [October 17, 2022, 6:29am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/3 "2022-10-17T06:29:17Z")

</div>

Filebeat configuration

```auto
filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /dir1/file1
  tags: ["file1"]

- type: log
  enabled: true
  paths:
    - /dir1/file2
  tags: ["file2"]

- type: log
  enabled: true
  paths:
    - /dir1/file3
  tags: ["file3"]

  
- type: filestream
  enabled: false
  paths:
  fields:
    level: debug
    review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  reload.period: 5s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 12

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

# ================================== Outputs ===================================
output.logstash:
  # The Logstash hosts
  hosts: ["host1:5044","host2:5044"]
  loadbalance: true
  worker: 1

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

logging.level: info
logging.to_files: true
logging.files:
  path: "/var/log/filebeat"
  name: "filebeat"
  keepfiles: 7
  permissions: 0644

```

Logstash Config:

```auto
input {
    beats {
            type => "logs"
            port => "5044"
    }
}
filter {
  if "file1" in [tags]
  {
      grok {
        match => { "message" => ["%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:access_time}\] \""] }
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => ["timestamp", "dd/MMM/YYYY:H:m:s Z"]
        remove_field => "timestamp"
      }
  }
  else if "file2" in [tags]
  {
  json {
    source => "message"
    }
  mutate {
    remove_field => ["ephemeral_id", "cloud", "agent"]
    add_field => { "read_timestamp" => "%{@timestamp}" }
    }
  }
  else if "file3" in [tags]
  {
  json {
    source => "message"
    }
  mutate {
    remove_field => ["ephemeral_id", "cloud", "agent"]
    add_field => { "read_timestamp" => "%{@timestamp}" }
    }
  }
}
output {

  stdout {
    codec => rubydebug
  }
  if "file1" in [tags] {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    user => "xxxxxx"
    password => "xxxxxx"
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "slm"
    data_stream_namespace => "file1"
    data_stream_auto_routing => "false"
    }
  }
  else if "file2" in [tags] {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    user => "xxxxxx"
    password => "xxxxxx"
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "prod"
    data_stream_namespace => "file2"
    data_stream_auto_routing => "false"
    }
  }
  else if "file3" in [tags] {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    user => "xxxxxx"
    password => "xxxxxx"
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "prod"
    data_stream_namespace => "file3"
    data_stream_auto_routing => "false"
    }
  }

```

In elasticsearch:

Datastream created : logs-prod-file1, logs-prod-file2, logs-prod-file3  
Additional index - logstash-{date}-0001 - this index contains some of the records from file1/2/3. Why is this happening ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 17, 2022, 6:39am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/4 "2022-10-17T06:39:21Z")

</div>

My guess would be that you have more than one configuration file in the directory. By default Logstash will concatenate all files found into a single pipeline and all data will be processed by all filters and go to all outputs, so if any other file had a simple Elasticsearch output it would potentially index all events in addition to the outputs shown above.

---

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [October 18, 2022, 10:50am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/5 "2022-10-18T10:50:33Z")

</div>

There is no configuration other than that mentioned above. Its not every records or replication,it's random. Like 5 out 10000 found in `logstash-{date}-0001` for each file1/file2/file3.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 19, 2022, 5:46am UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/6 "2022-10-19T05:46:21Z")

</div>

Is it continously being indexed into?

It looks like you have two Logstash instances. Have you verified none of them have any additional configuration files?

Do you see any unexpected connections to Elasticsearch that could indicate some other process is indexing into Elasticsearch?

---

<div class="post-metadata">

**Author:** ![connectgeeks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/connectgeeks/32/90595_2.png) [@connectgeeks](https://discuss.elastic.co/u/connectgeeks)\
**Post date:** [October 19, 2022, 5:34pm UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/7 "2022-10-19T17:34:16Z")

</div>

Yes, both logstash have exact same configs and pipeline. There is no output other than specific to file1/2/3. No other connections, everything is verified on that part.

@stephenb Can you please take a look at the configs as requested?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [October 19, 2022, 6:35pm UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/8 "2022-10-19T18:35:27Z")

</div>

Looks good to me.

I would isolate to a single logstash while debugging.

What does your pipelines.yml look like.

Also I notice you are missing the last `}` in your logstash pipeline not sure if that is a cut-n-paste or if there is something below that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 16, 2022, 6:36pm UTC](https://discuss.elastic.co/t/why-default-index-created-logstash-2022-01-01-000001/316632/9 "2022-11-16T18:36:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.