# Cannot get the name of indices using logstash

**URL:** <https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [September 20, 2023, 10:04am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440 "2023-09-20T10:04:59Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cruz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cruz/32/116126_2.png) [@Cruz](https://discuss.elastic.co/u/Cruz)\
**Post date:** [September 20, 2023, 10:04am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/1 "2023-09-20T10:04:59Z")

</div>

Hello everyone,

How can I get the name of indices using logstash?  
I have this indices which is from data stream called (backing indices if I correct)

```auto
.ds-my-neoada-stream-2023.09.14-000001
.ds-my-neoada-stream-2023.09.14-000002
.ds-my-neoada-stream-2023.09.14-000004
.ds-my-neoada-stream-2023.09.14-000005

```

I tried using this syntax

```auto
index => .ds.my-neoada-stream-%{+yyyy.MM.dd}

```

but what happened was that it created new indices

```auto
.ds-my-neoada-stream-2023.09.20

```

but this is not what I want to happen. What I want to happen is to update my lastest indices.  
using fingerprint

```auto

  fingerprint {
    key => "1234ABCD"
    method => "SHA256"
    source => ["ticket_id", "tick_account" , "tick_teampips" , "tick_current", "tick_stages", "ticket_xagent"]
    target => "[@metadata][_id]"
    concatenate_sources => true
  }

```

I also add the `action => create` , `document_id => "[@metadata][generated_id]"` and `doc_as_upsert => true`

```auto
elasticsearch {
      hosts => ["myIpAdddress"]
      user => "myuser"
      password => "mypassword!"
      cacert => "/etc/elasticsearch/certs/http_ca.crt"
      ssl => true
      ssl_certificate_verification => false
      document_id => "%{[@metadata][_id]}"
      doc_as_upsert => true
      index => ".ds-my-neoada-stream-%{+yyyy.MM.dd}"
      

# template_overwrite => true
      action => "update"

```

my purpose for this is to prevent duplication of my logs.

Any solution for this?

Regards

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 20, 2023, 12:17pm UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/2 "2023-09-20T12:17:48Z")

</div>

I don't think you can do what you want with Logstash.

You seem to be using data streams and data streams are [**append only**](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html#data-streams-append-only), meaning that you cannot update it with Logstash.

You will need to use normal indices without rollover to achieve this.

---

<div class="post-metadata">

**Author:** ![Cruz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cruz/32/116126_2.png) [@Cruz](https://discuss.elastic.co/u/Cruz)\
**Post date:** [September 21, 2023, 3:50am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/3 "2023-09-21T03:50:56Z")

</div>

Is there any other solution for this other than changing my index setup?  
I have this approach, if there is an existing document or ID then don't create for new document and updates the existing fields only.  
is this approach possible? If yes, how?

This is sample logs, and if this document exist just update the fields only like `"ticket_xstat": "Open"`  
what happens when user update to close "ticket\_xstat": "Close"  
it creates new documents again

```auto
{
  "_index": ".ds-my-neoada-stream-2023.09.14-000005",
  "_id": "0txktYoB0eBbzdbC3gSJ",
  "_version": 1,
  "_score": 0,
  "_source": {
    "ticket_zdesc": "I cannot access the shared drives",
    "agent": {
      "name": "Neoada-Server",
      "id": "37d5bd3a-55c8-4b17-8934-6544ecee3353",
      "type": "filebeat",
      "ephemeral_id": "a83bd7a2-69b7-48a5-9674-fa38d43da5af",
      "version": "8.9.0"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "ticket_xagent": "IT Helpdesk",
    "ticket_xdate": "2023-09-21 09:35:31",
    "ticket_xstat": "Open",

}

```

Best regards,  
CRUZ

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 21, 2023, 5:13am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/4 "2023-09-21T05:13:58Z")

</div>

> [@Cruz](#):
>
> Is there any other solution for this other than changing my index setup?

No, not really. Have a look at [this thread](https://discuss.elastic.co/t/rollover-index-duplication-data-data-coming-from-logstash/332726/3), which discussesa quite similar problem.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 21, 2023, 11:31am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/5 "2023-09-21T11:31:56Z")

</div>

> [@Cruz](#):
>
> Is there any other solution for this other than changing my index setup?

There is not, if you want to keep updating documents you cannot use Data Streams nor Rollovers, you need to use normal indices.

You can still use ILM to move your data between data tiers, if you have different data tiers, and to delete your indices after some time, but your use case won't work using data streams or rollovers because you need logstash to write directly on the back indices.

---

<div class="post-metadata">

**Author:** ![Cruz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cruz/32/116126_2.png) [@Cruz](https://discuss.elastic.co/u/Cruz)\
**Post date:** [September 22, 2023, 8:13am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/6 "2023-09-22T08:13:23Z")

</div>

Thank you so much @leandrojmp and @Christian_Dahlqvist for taking the time to answer my concern and for sharing the information.

@Christian_Dahlqvist your blog about duplicate prevention was a great help to my question.

To solve my problem I did what you said guys,

LOGSTASH CONFIG

```auto
input {
    pipeline {
        address => neoadaLog
    }
    elasticsearch {
      hosts => ["myIpAddress"]
      user => "myuser"
      password => "mypassword!"
      ssl => true
      ssl_certificate_verification => false

      index => "neoada-alias"
      query => '{ "sort": ["_doc"] }'
    }
    
} # <--- END OF INPUT --->

filter {

  if [log][file][path] =~ /\/opt\/lampp\/htdocs\/ada_v2\/storage\/app\/.+\.log/ {

    grok {
			match => { "message" => ["%{TIMESTAMP_ISO8601:[@metadata][timestamp]}%{SPACE}\(%{IP:ip}\)-%{DATA:log}: %{GREEDYDATA:[@metadata][logmsg]}"] }
      overwrite => ["[@metadata][timestamp]" , "[@metadata][logmsg]" ]
	  }

    json { source => "[@metadata][logmsg]" }

    date {
      match => ["[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss"]
		  timezone => "Asia/Manila"
		  target => "@timestamp"
    }
    mutate {
      remove_field => ["created_at" , "is_viber" , "assignment_id" , "budget_category", "location_id"]
    }
  
  fingerprint {
      key => "1234ABCD"
      method => "SHA256"
      source => ["ticket_id"]
      target => "[generated_id]"
      concatenate_sources => true
    }
  
    
  }
 
  
} # <--- END OF FILTER --->

output { 
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["myIpAddress"]
      user => "admin"
      password => "Lastikman2023!"
      cacert => "/etc/elasticsearch/certs/http_ca.crt"
      ssl => true
      ssl_certificate_verification => false
    
      #INDEX TEMPLATE
      document_id => "%{[generated_id]}"
      pipeline => "%{[@metadata][pipeline]}"
      template => "/etc/logstash/template/neoada-prod.template.json"
      template_name => "neoada-test"
      index => "neoada-alias"
      data_stream => false

      doc_as_upsert => true
      action => "update"
    }
  }
  else {
    elasticsearch {
      hosts => ["myIpAddress"]
      user => "myuser"
      password => "mypassword!"
      cacert => "/etc/elasticsearch/certs/http_ca.crt"
      ssl => true
      ssl_certificate_verification => false
      

      #INDEX TEMPLATE
      document_id => "%{[generated_id]}"
      template => "/etc/logstash/template/neoada-prod.template.json"
      template_name => "neoada-test"
      index => "neoada-alias"
      data_stream => false

      doc_as_upsert => true
      action => "update"
    }
  }

  stdout {
    codec => rubydebug
  }
  file {
    codec => json
    path => "/var/log/logstash/neoada.log-%{+YYYY-MM-dd}.txt"
  }
} # <--- END OF OUTPUT --->

```

I created a new index template and put my rollover alias in the settings.

```auto
PUT _index_template/index_template
{
  "index_patterns": ["my-index-stream-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.lifecycle.name": "new-lifecycle-policy",      
      "index.lifecycle.rollover_alias": "indexseries"    
    }
  }
}

```

I also created a new index and put the name of my rollover alias there

```auto
PUT %3Cneoada-stream-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "neoada-alias": {
      "is_write_index": true
    }
  }
}

```

That's it, I did it with a rollover and the duplicates disappeared.

Thank you guys.

Best regards!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 22, 2023, 11:34am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/7 "2023-09-22T11:34:15Z")

</div>

> [@Cruz](#):
>
> That's it, I did it with a rollover and the duplicates disappeared.

Keep in mind that this will not always work.

Rollover is similar to data streams in the fact that it also uses a backing indice, so if you receive a document with an id that was already indexed, but the backing indice is not the write indice anymore, it will be duplicated.

For example, assuming that today you receive a document with the id `1234`, and tomorrow your index has a rollover, if after that you receive a document with the same id, the backing indice will be different and the document will be duplicated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2023, 11:35am UTC](https://discuss.elastic.co/t/cannot-get-the-name-of-indices-using-logstash/343440/8 "2023-10-20T11:35:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.