Indexes with FileBeat

Elastic Stack Beats
1

Hi,
I am using file beat to transfer logs to elastic. Everything is working fine, the only issue is that filebeat adds date to the ES index, so if my index name is "Test", how filebeat creates it on ES is "Test-YYYY.MM.dd" i.e. appending the timestamp. Is there anyway to override this default behavior?

2

Hi,

According to the documentation (see https://www.elastic.co/guide/en/beats/filebeat/1.3/elasticsearch-output.html#_index), you must configure the index option of the Elasticsearch Output:

output:
  elasticsearch:
    index: "foo"
3

Hey,
Thats exactly what I've done.

elasticsearch:
# Array of hosts to connect to.
# Scheme and port can be left out and will be set to the default (http and 9200)
# In case you specify and additional path, the scheme is required: http://localhost:9200/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
hosts: ["localhost:9200"]

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "admin"
#password: "s3cr3t"

# Number of workers per Elasticsearch host.
#worker: 1

# Optional index name. The default is "filebeat" and generates
# [filebeat-]YYYY.MM.DD keys.
index: "test2"

But the index created inside elasticsearch is like test2-2016.08.02. I don't want this timestamp following the index name.

Thanks

4

In the 1.x version it is not possible to disable the daily indices. If you prefer a different pattern, I recommend to use Logstash where you can adjust the index to your needs.

In the upcoming 5.0 release it will be possible to define your own pattern (or leave out the pattern): See https://github.com/elastic/beats/blob/master/filebeat/filebeat.full.yml#L328

This is currently only available in our snapshot builds: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

5

Hi Ruflin,

I am using filebeat 5.0.0, How to disable daily indices in in this?

6

Daily indices are generated for users to be able to drop old indices e.g. using curator (unless you've got infinite disk space). Otherwise dropping old data can become a very expensive operation. Querying all data via REST api is still possible using filebeat-* as index.

In beta1 index will become a format-string with default being filebeat-%{+yyyy.MM.dd}.

7

Thanks Steffens that helped.

Can I except parsing of Logs in JSON in the stable version of filebeat out any soon?

8

JSON support is part of 5.0 release. A 5.0 beta1 release should be available soon.

9

This topic was automatically closed after 21 days. New replies are no longer allowed.