Index names including yyyy-mm-dd

Elastic Stack Elasticsearch
1

I am new to elasticsearch and am trying to figure out index management. I index smallish documents continuously for a week. I would like the index names to include the date in their name along with a project specifier i.e.

foobar-yyyy-mm-dd

It doesn't appear to be a simple thing to do. Is this accomplished by using a template for foobar indexes and then alias all foobar indices to some umbrella term for searching? If this is correct than I assume I use curator to prune?

I just want to make sure I am not over complicating an otherwise simple thing.

Thank you for reading this and your thoughts.

sg

2

Logstash and Filebeat will let you template the Elasticsearch index names. E.g. in my Logstash config

    index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"

How do you ship your data to Elasticsearch?

3

If I understand "ship" correctly, I ingest data using java?

4

Directly to Elasticsearch? You could put everything through Logstash and get index creation and naming "for free" :slight_smile:

5

Hrm. That seems strange but remember I am new to elasticsearch. Can you give me a couple bread crumbs to follow? I use the java RestHighLevelClient to index. How do you "put" everything through logstash?

6

Curator would indeed be the tool of choice to prune.

7

Adding Logstash to the mix is of course one more service to worry about. What you get is a service that can talk fluently with Elasticsearch, can load balance over several ES nodes and it will create and name the ES indices for you. You can also do more advanced stuff like enrich your logs as well.

As you are shipping directly to Elasticsearch your data has to be in JSON.

To run Logstash, configure a TCP or UDP input that accepts JSON. From the TCP input plugin docs

input {
  tcp {
    port => 12345
    codec => json
  }
}

The filter section would be just

filter {}

The output is Elasticsearch

output {

  elasticsearch {
    hosts => ["127.0.0.1:9200","127.0.0.2:9200"]
    index => "foo-%{+YYYY.MM.dd}"
  }

}

So, instead of RestHighLevelClient shipping to Elasticsearch, something like hostname:9200, you ship to logstashhost:12345

8

You can read more here Logging directly to elasticsearch

9

thank you

10

I'd not use logstash when the source of data is available from a Java application.

The index name is something you can easily compute in Java. There's no need IMO to add logstash.
But may be I'm missing something. What kind of data are you indexing @gormanst?

11

I am indexing small blocks of json data received from an outside application via REST. I make some minor transformations then index.

12

So just compute the index name in your REST application.

13

That's kind of where my original question was headed. I just wanted to make sure I was not overly complicating things. So I should create a template with an alias (for searching) and programatically figure out that I want to index to foobar-yyyy-mm-dd. Finally, use Curator to prune off old data?

Again thanks for your thoughts on this.

14

Exactly what I'd do.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.