# Unexpected indices being created by Logstash

**URL:** <https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230>\
**Category:** Logstash\
**Created:** [November 4, 2020, 8:49am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230 "2020-11-04T08:49:30Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\
**Post date:** [November 4, 2020, 8:49am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/1 "2020-11-04T08:49:30Z")

</div>

I'm using Logstash to ingest logs stored on AWS S3. My configuration file looks like this:

```auto
input {
        s3 {
                "access_key_id" => "REDACTED"
                "secret_access_key" => "REDACTED"
                "bucket" => "REDACTED"
                "exclude_pattern" => "^(?!lin-gitlab-gl-stage-\d+-\d+-\d+-\d+\/gitlab-workhorse\/(current|\S+.s))"
                "gzip_pattern" => "\.s?$"
                "region" => "ap-southeast-1"
                "id" => "gitlab-workhorse"
                "sincedb_path" => "/var/lib/logstash/plugins/inputs/s3/sincedb_gitlab_workhorse"
                "type" => "workhorse"
        }
        s3 {
                "access_key_id" => "REDACTED"
                "secret_access_key" => "REDACTED"
                "bucket" => "REDACTED"
                "exclude_pattern" => "^(?!lin-gitlab-gl-stage-\d+-\d+-\d+-\d+\/gitlab-rails\/api_json.log(.\d+.gz)?)"
                "gzip_pattern" => "\.gz?$"
                "region" => "ap-southeast-1"
                "id" => "gitlab-rails"
                "sincedb_path" => "/var/lib/logstash/plugins/inputs/s3/sincedb_gitlab_rails"
                "type" => "api"
        }
}

filter {
        json {
                source => "message"
        }
        if [remote_ip] == "127.0.0.1" {
                drop {}
        }
}

output {
        amazon_es {
                hosts => ["REDACTED"]
                region => "us-east-1"
                index => "stage-gitlab-%{type}-%{+YYYY.MM.dd}"
        }
}

```

So, given that I'm explicitly using `type` as part of the index name, I would expect this configuration to only create indices with names that start `stage-gitlab-api` and `stage-gitlab-workhorse`.

However, I'm seeing names where the "type" portion of the index name is "w", "ssa", "ss", "slo", "seccft" and so on, e.g. `stage-gitlab-seccft-2020.11.04`.

Can anyone explain why this is happening, please?

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 4, 2020, 8:58am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/2 "2020-11-04T08:58:07Z")

</div>

Hello Philip,

Is it possible that the type already exists in the logs? According to the docs [here](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-type):

> If you try to set a type on an event that already has one (for example when you send an event from a shipper to an indexer) then a new input will not override the existing type. A type set at the shipper stays with that event for its life even when sent to another Logstash server.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\
**Post date:** [November 4, 2020, 9:10am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/3 "2020-11-04T09:10:18Z")

</div>

Oh ... do you mean that "type" can be inferred by Logstash from the logs itself?

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 4, 2020, 9:13am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/4 "2020-11-04T09:13:05Z")

</div>

Yes. From what I understand your pipeline reads logfiles from S3 and parses them as JSON. If your JSON contains a type field in the root level your type in the input will be overridden.

---

<div class="post-metadata">

**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\
**Post date:** [November 4, 2020, 9:26am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/5 "2020-11-04T09:26:38Z")

</div>

What is really weird is that if I try to create an index pattern so that I can dig into this deeper, those "strange" indices don't show up:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/8/78f55127db7c4213cc0dcc768de888520caff986.png)

I'll have a look at the underlying log files to see if there is a "type" attribute in there.

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 4, 2020, 9:33am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/6 "2020-11-04T09:33:43Z")

</div>

> [@Philip\_Colmer](#):
>
> What is really weird is that if I try to create an index pattern so that I can dig into this deeper, those "strange" indices don't show up

Normally, you can switch through the pages on the lower left of the index list:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/c/bc78768a27fb1ee252f2636d334302eb91be7efb.png)

Update:  
In your screenshot there is one of the "strange" indizes:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d029cac2c35cacb1c23870014e8b4d33d202dc3.png)

---

<div class="post-metadata">

**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\
**Post date:** [November 4, 2020, 9:42am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/7 "2020-11-04T09:42:08Z")

</div>

> [@Wolfram\_Haussig](#):
>
> In your screenshot there is one of the "strange" indizes

That is the only one, though. The indices are listed in alphabetical order and I picked that screenshot to show that it jumps from api to auth0\_managed\_certs to workhorse, bypassing all of the unexpected indices.

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [November 4, 2020, 9:45am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/8 "2020-11-04T09:45:59Z")

</div>

> [@Philip\_Colmer](#):
>
> However, I'm seeing names where the "type" portion of the index name is "w", "ssa", "ss", "slo", "seccft" and so on, e.g. `stage-gitlab-seccft-2020.11.04` .

So you see the indizes in the index list but not when creating an index pattern?

Do you have hidden indices?

---

<div class="post-metadata">

**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\
**Post date:** [November 4, 2020, 9:53am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/9 "2020-11-04T09:53:17Z")

</div>

> [@Wolfram\_Haussig](#):
>
> So you see the indizes in the index list but not when creating an index pattern?

Yes

> [@Wolfram\_Haussig](#):
>
> Do you have hidden indices?

Not that I'm aware of. These indices are being created by Logstash so I don't know whether they are hidden or not. Sorry - I'm very much an Elastic newbie.

---

<div class="post-metadata">

**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\
**Post date:** [November 4, 2020, 10:36am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/10 "2020-11-04T10:36:13Z")

</div>

> [@Wolfram\_Haussig](#):
>
> From what I understand your pipeline reads logfiles from S3 and parses them as JSON. If your JSON contains a type field in the root level your type in the input will be overridden.

I've looked at some of the log files now and none of them have "type" in them. However, I will change the configuration to use a phrase that is not under any circumstances going to appear in the files.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 2, 2020, 10:36am UTC](https://discuss.elastic.co/t/unexpected-indices-being-created-by-logstash/254230/11 "2020-12-02T10:36:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.