How to use Filebeat with a custom index and pattern

[ajhstn]

I am using Filebeat for supported log formats and using the default index settings and mappings etc.. This is great..

Now i also want to send a custom JSON log file, also using Filebeat, but want to send it into it's own new index, i cannot work out how to do this.

I am using Elastic Cloud, so i cannot use the indices property in Elasticsearch output.

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "path/to/customlog.json"

cloud.id: "${CLOUD_ID}"
cloud.auth: "${ES_PWD}"

[steffens (Steffen Siering)]

You can still populate the output.elasticsearch.X settings. When using cloud, only the hostname setting will not be useable.

By default ILM will be enabled. Which Beats + ES versions have you running?
The settings output.elasticsearch.index and output.elasticsearch.indices get disabled if ILM is enabled, because ILM requires you to have a write alias.

I think it is a good opinion to separate 'setup' and actually running beats. This also allows you to create different users for indexing and setup, and be more strict on permissions for users wantng to index (do not allow them to create/change any resource in the cluster, besides indexing). For some related discussion have a look at this github issue: https://github.com/elastic/beats/issues/10241

The input you have configured does not enable json parsing. Is this on purpose? If you want to enable json parsing (or in general when using another index), you should prepare and install an index template for your new index.

Having some details on what your end result should be, I can maybe give you some more instructions/hints:

• Exactly which Elasticsearch/Beats are you using?
• What would be the index names?
• Does your json log follow ECS, or is there a chance of mapping conflicts?
• Do you want to use ILM for all indices, or not at all?
  • different policies?
• Do you make use of security (users, roles) ?

[ajhstn]

Hi thanks for your reply. There's a lot there for me to go and learn/read.

You can still populate the output.elasticsearch.X settings. When using cloud, only the hostname setting will not be useable.

Amazing!

The settings output.elasticsearch.index and output.elasticsearch.indices get disabled if ILM is enabled, because ILM requires you to have a write alias.

Oh right, so i can remove output.elasticsearch.index then.

Do you want to use ILM for all indices, or not at all?

I think so, I want to keep the index for about 90 days then i can allow old data to drop off.

My use case is to "make AD searchable". I am indexing AD Users and Computers on a 4 hourly schedule. The data is generated with a Powershell script writing out to a JSON file, picked up by Filebeat for shipping. I just now need to convert my field names to be ECS compatible. I want to get it ready so others in the community can use it also, as i feel there is a good use case for it. AD data can then be enriched and correlated with Windows Security log data and other ECS compatible data. I'd be happy to work on this with others if you're interested.

This provides some great search and visuals for AD like

• How many user, or computers
• List the users with passwords about to expire
• Show me inactive users etc.
• Show me new users created or deleted today

[steffens (Steffen Siering)]

The use-case makes me curious. How do you get the AD data (I'm no windows user for a many years...), but if there is a low level API, I think it would be cool to have a filebeat or auditbeat module reading and processing those.

I think so, I want to keep the index for about 90 days then i can allow old data to drop off.

This can be achieved in two ways. 1. turn off ILM (setup.ilm.enabled: false) and use daily indices. Then use curator (or a script) to remove old indices. 2. configure and ILM + write alias to write to.

The later is possible with setup.ilm.enabled: false. One only need to configure ILM + template + write alias manually (via Elasticsearch API, or prepare a special filebeat config to be used with filebeat setup). Once you have the write alias (say active-directory), you can just configure the the write alias as index name via output.elasticsearch.index.