ILM problem

hello all .
I'm using ELK 8.17.1 and have ILM configured with a Hot phase.
my problem is :

my policy :
name = policy_ipas
age = 2min
size = 5mb

and my indexes :


also index2 : No logs are thrown inside it.

=================================
winlogbeat configurations : 
  ssl.certificate_authorities: ["C:\\Program Files\\winlogbeat\\http_ca.cer"]
  ssl.verification_mode: full
  indices:
    - index: "index_ipas"




template configurations : 
name = template_ipas
pattern = index_ipas-*

{
  "index": {
    "lifecycle": {
      "name": "policy_ipas",
    },
    "number_of_shards": "1",
    "number_of_replicas": "0"
  }
}




initial index configuration : 
PUT /index_ipas-000001
{
  "aliases": {
    "index_ipas": {},  
    "rollover_ipas": {  
      "is_write_index": true  
    }
  },
  "settings": {
    "index.lifecycle.name": "policy_ipas",  
    "index.lifecycle.rollover_alias": "rollover_ipas",  
    "number_of_shards": 1,
    "number_of_replicas": 0
  }
}

Which configuration is wrong ?

Hi @Mehran_Noorani Welcome to the community!

ILM is not really design to operate on "Minute" Scheduled or Small KB/MB indices is designed to manage large scale indices GB, Hours, Days etc. and for that it works very well.

I posted a bit about it here and another elastician here

ILM by default only checks every 10 minutes. I believe so. We often see people trying to test like this. It's not designed to work at these micro scales.. It works at the scale that it's supposed to very well

1 Like

Thank you @stephenb for your response to my question.
Do you think my configurations are correct to achieve my goal?
Have I made any configuration mistakes?
I would like to know your professional opinion Because despite my configurations, no logs are being sent to the second index.

Thank you

Hi @Mehran_Noorani

So quick first ... no your configuration is not correct, I really at first did not take a close look due to the 2m etc...

Second you are somewhat close .... but there are some key concepts / configurations you are missing.

Before I try to answer your question / provide direction

This is a great question ... Exactly what are your goals

You appear to want a specially named index... why? What does this accomplish / requirement Does it satisfy more than just the defaults? Do you just want to see the index named what you want ... are you going to create many more?

In Short what are you trying accomplish and why?

In general when people first get started I recommend to use the defaults ... see how everything works and then start to modify... as opposed to modify many setting right from the start.

Let me know and perhaps over the weekend I can give you some advice...

Also when you provide only "snippets" of configurations instead of complete details... something is often missed so when we do next rounds let be complete...

Example You did not show the whole winlogbeat configuration... what are you collecting etc... I ask because if you just rename the index... there is all sorts of OOTB functionality that you might / will not get... so back to the conversation above...

  • What are you trying to accomplish? and Why
  • What data are you collecting and what do you want to do with it?
  • Do you want to take advantage of all the OOTB functionality etc..etc..

All that said .. sure we can rename the index... but it may take a little work to get everything wired together..

1 Like

Hello @stephenb ,
Thank you for your attention to my question.

I am using a specific index because I need to store logs from 50 different software systems on this Elasticsearch server. Therefore, in order to distinguish the indices of each system from one another, I need to assign a unique name to each index .

My requirement is that, for example, once an index reaches a size of 100GB (index_size) or after one month of its lifespan (index_age), it should be closed, and the logs of that system should be written to a new index with the same naming pattern as the previous one.

** despite my configurations, no logs are being sent to the second index.

for example : 
app1_name-000001  ->  100GB
app1_name-000002  ->  100GB
app1_name-000003  ->  34GB (currently in use)

app2_name-000001  ->  100GB
app2_name-000002  ->  100GB
app2_name-000003  ->  12GB (currently in use)

And I can search for some messages in all indexes of appX_name.

Exactly what configurations should I send you to complete the information?

Can I contact you via email to provide you with more detailed information?

Thank you.

A common reason but generally does not actually solve any "real" issue and leads to issues more complexity ... it is basically an anti-pattern especially for windows even logs / winlogbeat

Before we proceed as you ask for my professional opinion ...

I strongly recommend you do not do this...especially if you are new to Elastic This is a tried and true. Anti-pattern I strongly recommend you just start with the defaults and run for a month you two... and see how it works.. simply use the defaults and then edit the Default ILM policy and set to your desired ILM configuration.

Winlogbeat ships Windows event logs....

These will already have metadata of which host/system it comes from. You can even add additional tags to the config that further define the systems etc. they event are coming from. Winlogbeat is set up to collect events from 10s, 100s, or event 1000s of hosts / systems.

BTW I had a user that was shipping 80K windows host into a single index (which is actually a data stream are you aware of that?) .. it was big and rolled over frequently ... but it worked just fine.

You can still configure Winlogbeat to ship different events per host / system

This is how it is built ... then you handle the separation on the Search / Query / Alert etc...

So if you did not name each index by host... and just used all the default it would be VERY easy to see logs per host AND reduce Management AND get the best OOTB experience...

This is how most users do it in the long term... index per host / per system does not scale well.

Still happy to help you how to name your own indices when I have the time ... but a couple of months from now you will most likely come to the conclusion it was not a great decision

No We / I / On this forum we do not do this ... we are not private support...
If you want dedicated support or consulting I suggest reaching out to elastic sales.

I will try to provide some direction in the next couple of days. I am a volunteer, and we all are on this site. Please keep in mind there are many question and few volunteers...

And to start please share you entire winlogbeat.yml (sans credentials and endpoints) and any modules you are using. Perhaps I can get back in a couple days.

If you still want to proceed, perhaps you should look at the docs here... But you'll notice it's not even really meant per host

This is not to say that segmentation does not have a place, but winlogbesat is typically not one of them. If you had entire different application log systems or something like that That require distinct different processing paths, parsing paths, etc. Then perhaps that would make sense. But winlogbeat really doesn't makes sense

1 Like

Yes, that's correct, and thank you @stephenb . First, I should mention that I have various servers, including Windows VMs, Linux VMs, and containers, and I plan to send their logs to the ELK server via API. Currently, I am testing the ELK stack in the lab, and I have many concerns regarding the separation of logs from different software systems and separating them monthly for backup. For example, I need to extract and archive the logs of a specific program from within a single index after one month, as you mentioned. However, I am unsure of what strategy I should use for these 50 different systems to avoid problems when archiving older logs to a separate storage (SAN) in the long term.

On the other hand, I need to maintain the ELK server (or server cluster) in the most stable way possible, so that the least amount of management and manual actions are required for the server and its indices, and it’s important for me that everything is as automated as possible.

Anyway, the Winlogbeat configuration for my test Windows server is as follows:

###################### Winlogbeat Configuration Example ########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains
# all the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

# ======================== Winlogbeat specific options =========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name, id, xml_query, tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml.
# The xml_query key requires an id and must not be used with the name,
# ignore_older, level, event_id, or provider keys. Please visit the
# documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h

  - name: System

  - name: Security

  - name: Microsoft-Windows-Sysmon/Operational

  - name: Windows PowerShell
    event_id: 400, 403, 600, 800

  - name: Microsoft-Windows-PowerShell/Operational
    event_id: 4103, 4104, 4105, 4106

  - name: ForwardedEvents
    tags: [forwarded]

# ====================== Elasticsearch template settings =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboard archive. By default, this URL
# has a value that is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "172.16.10.15:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://172.16.10.15:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "P@ssw0rd"
  ssl.certificate_authorities: ["C:\\Program Files\\winlogbeat\\http_ca.cer"]
  
  ssl.verification_mode: full
  indices:
    - index: "index_ipas"


  # Pipeline to route events to security, sysmon, or powershell pipelines.
  pipeline: "winlogbeat-%{[agent.version]}-routing"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors, use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Winlogbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Winlogbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch outputs are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the winlogbeat.
#instrumentation:
    # Set to true to enable instrumentation of winlogbeat.
    #enabled: false

    # Environment in which winlogbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

#setup.ilm.enabled: true

#setup.ilm.rollover_alias: "policy_ipas"

so do you not recommend this approach either?

[windows server + app1] --- API ----> [ elk : Index-app1]
[container_app2] --- API ----> [ elk : Index-app2]
[ubuntu server + app3] --- API ----> [ elk : Index-app3]

I also performed a new test where I set the index size to 1GB. The new index was created, but the logs are not being written into it.

@Mehran_Noorani

Ok now you are providing some context... You are trying to design a whole system.

After reading that, I would recommend using Fleet and Elastic Agent, which is for the now and the future, not individual beats.

They support OOTB data stream, datasets and name spaces, ingestion pipelines ILM etc...... which allow a better approach to segmentation, control, parsing etc. (I would still be carefull as over segmentation can cause issues)

If I was building a system that would be my approach.

For each "System" you need to identify what you will collect ...
Are they "share / common" are the Specific Uncommon etc..
Are there common processing paths
What has common and different Life Cycle Policies
I would first do a spread sheet exercise of your inventory.

I would read up on Elastic Agent, Integrations, Data stream etc

I don't know what that means... extract but ok... and I do not understand the focus on a single index... but OK as well... I think you are making assumptions about elastic data streams and indices which may or may not be correct....

If you are saying that there are different Lifecycles per System of something like that you need to think in terms of that... Top Down (requirement, retention etc) not Bottom up (per index etc)

You can keep trying with your config... it is not going to work..

DO you have a requirement to be on Winlogbeat? or Would Elastic Agent Work...

If you are stuck on winlogbeat
Note you have to run setup -e once BEFORE you start writing data
Take a look at this post it is for metricbeat but winlog beat works the same
What this does is setup all the scafolding...

Its a more complete version of this...

There is another approach as well when you use data streams..

Look at this...

Ok ... out ... think about what you want to do... I will check check back in at some point But we can't design your whole system. We can only provide you direction...

Do I need to disable ILM on the Elastic server? Because I have set a different rollover_alias in my template. On the other hand, I also create an initial index to start with. Is that necessary?

setup -e --> on elastic server STDIN or in Kibana console (DEV tools) ??

Thank you for your guidance. I need to carefully test the mentioned items and report the results again.

It seems like you're just kind of jumping in the middle of the pool...

If you read the thread I provided, it provides the steps.

I also suggest that you actually read the winlogbeat documentation and the quick start guide..

You are actually making all the points of why I would just start with out of the box and get some things running and then start adjusting... You're trying to change all these configurations on day one. It's going to be a struggle because you don't understand the concepts and the processes The data schema data management... Which is perfectly fine... It's a learning process

There's a lot to elasticsearch and shipping data and ingest pipelines and data management... But out of the box we set up very very good defaults...

The setup is part of the winlogbeat process run on that The host where you're running winlogbeat.

No, you never stop the ilm process, which I don't think that's what you mean. It runs in the background on elasticsearch across thousands of indices at some point.

I've given you a bunch of direction... And it seems like you're pretty much stuck on the original question /direction which is fine.

I've actually given you a thread that shows in detail how to set up what you want to do. You need to test.....

I highly suggest you start reading the documentation in detail... Doing some testing... Search on this forum....

1 Like

Yes, that's correct. I need to understand the processes better. I sincerely appreciate your guidance and the time you have spent.

If you are still insisting to make separate indices for winlogbeat (which of all the data sources that is the last one I would do that with)
At least follow a pattern like below for many reasons including (and more)

  • Default Dataviews will still
  • Dashboards and Visualizations should still work
  • You will be able to aggregate / filter across systems

Assuming the system name is ipas
then...

setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.rollover_alias: winlogbeat-ipas
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false

setup.template.enabled: true  
setup.template.name: "winlogbeat-ipas-%{[agent.version]}" 
setup.template.pattern: "winlogbeat-ipas-%{[agent.version]}" 
setup.template.overwrite: false 
setup.ilm.policy_name: winlogbeat-ipas