Indices based on field value

Hi There,

I am doing a POC on elasticsearch, currently all the cisco asa logs are captured using filebeat cisco module. We would like to create a separate indice when a specific condition is met. Basically we need create two indices one for flow logs and other for config changes so that we can clear only the flowlogs after a week. i am trying to set like this , but this is not creating new Indice.

output.elasticsearch:

Array of hosts to connect to.

hosts: ["10.181.xx.xx:9200"]
compression_level: 9
index: "cisco-filebeat-%{+yyyy.MM.dd}"

indices:
- index: "flowlog-%{+yyyy.MM.dd}"
when.equals:
log.level: "informational"

setup.ilm.enabled: false
setup.template.name: "cisco"
setup.template.pattern: "cisco-*"

Please help.

Could you post your Filebeat configuration escaped within ``` delimiters please?

Hi Shaunak,

Please see the filebeat config below.

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after


#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
  #compression_level: 9
#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  compression_level: 9

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

  index: "cisco-filebeat-%{+yyyy.MM.dd}"

  indices:
   # - index: "palo-%{+yyyy.MM.dd}"
   #   when.equals:
   #     event.module: "panw"

    - index: "flow-%{+yyyy.MM.dd}"
      when.equals:
        log.level: "informational"

setup.ilm.enabled: false
setup.template.name: "cisco"
setup.template.pattern: "cisco-*"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

  - drop_fields:
      fields: ["agent.id", "agent.version", "cloud.availability_zone", "cloud.image.id", "cloud.instance.id", "cloud.machine.type", "cloud.provider", "cloud.region", "ecs.version", "agent.ephemeral_id", "agent.id", "host.architecture", "host.os.platform", "host.os.codename", "host.os.family", "host.os.kernel"]

      ignore_missing: true



#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
logging.metrics.enabled: false
#============================== X-Pack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

Thanks. What indices are you seeing created in Elasticsearch with this configuration? The following Elasticsearch API call should tell you:

GET _cat/indices
Hi Shaunak,

Please see the details below , have trimmed few indices due to the limit in positing 

green open winlogbeat-7.6.2-2020.08.03-000036 OvBqsmdRQyW19ehJ-jMp3w 1 0    78787      0  25.8mb  25.8mb
green open .reporting-2020.07.12              9uzbU06WSxCfRPXexLDgHA 1 1        2      1 308.3kb 154.1kb
green open winlogbeat-7.6.2-2020.08.04-000037 boWA9iyXSiW2c4D6YwxieA 1 0    33850      0  11.7mb  11.7mb
green open winlogbeat-7.6.2-2020.08.05-000038 kNC92DsaTxKGZyU2mjn4Iw 1 0    34589      0    12mb    12mb
green open winlogbeat-7.6.2-2020.08.06-000039 KcvwEGmZTxiPwViDAmIfvA 1 0    27625      0   9.8mb   9.8mb
green open winlogbeat-7.6.2-2020.08.07-000040 jIHBuTyXSLG1ADmqG1un0A 1 0    45580      0  15.8mb  15.8mb
green open .kibana_task_manager_1             sHK0DlQfQvCfmIsQdVYsXQ 1 0        1      0  43.9kb  43.9kb
green open filebeat-7.6.2-2020.08.04-000043   -KgIBLzYQeiVLp5MNS44iA 1 0   420523      0  97.7mb  97.7mb
green open filebeat-7.6.2-2020.07.31-000041   09vPII5VR4m4UOJ3Zo9IkA 1 0   583230      0 122.6mb 122.6mb
green open auditbeat-7.6.2-2020.08.14-000053  2_uwNze6QzaKrcpoFix4Dg 1 1   290460      0 358.8mb 178.8mb
green open auditbeat-7.6.2-2020.08.12-000051  qplfi2gaTS-f7wX_CbpFWw 1 0  7260205      0   4.2gb   4.2gb
green open filebeat-7.6.2-2020.08.07-000046   HHjvq1M0SE2SsgsVpzIW1w 1 0   577994      0 131.7mb 131.7mb
green open palo-2020.07.28                    JCwPNTmmQcudu2pSdWSl8g 1 1       30      0 370.1kb   185kb
green open metricbeat-7.6.2-2020.06.12-000001 u1vyok0BQM63OVKhnVh_CA 1 0     1170      0 259.1kb 259.1kb
green open .monitoring-es-7-2020.08.10        3PrqHcveRsW2fr_rsbVAdg 1 1   509998 361620 452.9mb   235mb
green open .monitoring-es-7-2020.08.11        E5rVyjaKTS2KRRhZmA07eA 1 1   584075 169360 492.8mb 245.4mb
green open .monitoring-es-7-2020.08.12        j1wIM_peSMuVsbRojIZITA 1 1   619951 203954 531.9mb 258.5mb
green open .monitoring-es-7-2020.08.13        YBWcbufcTp-AKKeeqfSoKw 1 1   633380 274899 550.9mb   276mb
green open .monitoring-es-7-2020.08.14        fulztQb9QYef3yG232yJ_w 1 1     1310      0  65.2mb  23.9mb
green open winlogbeat-7.6.2-2020.08.12-000043 AKNQ1pTxSgy_moSZ84QIVw 1 0    26840      0   9.6mb   9.6mb
green open winlogbeat-7.6.2-2020.08.13-000044 Ehc0bFhnTk-NPkd9uRXTyg 1 0     6883      0   3.1mb   3.1mb
green open filebeat-7.6.2-2020.08.10-000047   cPD5ZFDmQVyDl6MOkS7z4A 1 0   964325      0 205.4mb 205.4mb
green open filebeat-7.6.2-2020.08.03-000042   SQCoUpouQOqJS1_68aHVPg 1 0   673879      0 141.4mb 141.4mb
green open filebeat-7.6.2-2020.07.30-000040   GYkpFvnRR2qCB9OHNDRumA 1 0   305308      0  66.9mb  66.9mb
green open auditbeat-7.6.2-2020.08.13-000052  ubKld94JQrietxmzxw_KTQ 1 0  6946352      0     4gb     4gb
green open winlogbeat-7.6.2-2020.07.23-000029 yimO4B9kQnCmwsMoSF5_Dg 1 0    34643      0  11.9mb  11.9mb
green open winlogbeat-7.6.2-2020.07.22-000028 y9Kg-eEMQjavmz4MGxHFrA 1 0    58168      0  19.9mb  19.9mb
green open cisco-filebeat-2020.08.07          fC91ccoyTLW0ltDTVl1q4A 1 0 23157759      0   6.6gb   6.6gb
green open cisco-filebeat-2020.08.06          IVyWt36IR7qv5GfxmDDRrQ 1 0 28224112      0   8.1gb   8.1gb
green open cisco-filebeat-2020.08.05          pIWHdbBrQ0-cN21zm8mCiw 1 0 25965263      0   7.4gb   7.4gb
green open cisco-filebeat-2020.08.11          x56WCfMeSSm2KizeQGMbFQ 1 0 25527590      0   7.2gb   7.2gb
green open winlogbeat-7.6.2-2020.07.27-000031 juglqouDSiqKofPK4oL6Fg 1 0    90374      0  29.3mb  29.3mb
green open .monitoring-es-7-2020.07.20        B15o0OXaTCmZyW3aNE2pjw 1 1   432396 178468 417.8mb   208mb
green open cisco-filebeat-2020.08.10          5ifT9dQ0QruMpTZxPPBRYQ 1 0 22476566      0   6.4gb   6.4gb
green open .monitoring-es-7-2020.07.21        j6qTG7gxQ-yes8J6fUgKxA 1 1   531048 282480 497.7mb 248.2mb
green open winlogbeat-7.6.2-2020.07.28-000032 Vs1teRhvThuosS4fwEJm5g 1 0    42042      0  14.8mb  14.8mb
green open .monitoring-es-7-2020.07.22        cklhcU3qTP2Go8UIYxL3VQ 1 1   441463 185709 403.7mb 201.3mb
green open winlogbeat-7.6.2-2020.07.29-000033 ssqBXV4pTim4Pafej3sYRQ 1 0    35149      0  12.2mb  12.2mb
green open .monitoring-es-7-2020.07.23        7narsD_bRR6fhmM1OLI-Cg 1 1   251567  13468 246.1mb 120.1mb
green open cisco-filebeat-2020.08.14          kP_vBll1RbWG-fSoiRh5eQ 1 1  3715473      0   2.2gb   1.1gb
green open .monitoring-es-7-2020.07.24        MumSUGAHSdWC-g4xITJmuw 1 1   254565  45770 247.6mb 123.5mb
green open cisco-filebeat-2020.08.13          Rpw4QeO_QIGgRvhT4oMQOQ 1 0 25863609      0   7.4gb   7.4gb
green open cisco-filebeat-2020.08.12          9ewRxhlCRV-Wflscgu9CwQ 1 0 28605429      0   8.2gb   8.2gb
green open filebeat-7.6.2-2020.08.11-000048   IWlyMcP6S8-0FP1izV-jXA 1 0   308587      0  67.2mb  67.2mb
green open .monitoring-es-7-2020.07.27        umW5AbLASbmHym_dZPkaYw 1 1   255611  26970 245.5mb 122.1mb
green open metricbeat-7.6.2-2020.08.12-000003 6oIlz0FgRX6x54UuMwEsnw 1 0        0      0    283b    283b
green open .monitoring-es-7-2020.07.28        e4lrBXO2Q9K_Hgh0uva9oQ 1 1   315246  87819 292.7mb 145.7mb
green open filebeat-7.6.2-2020.08.12-000049   XBJsiLQAQey-IcYjyaEk-Q 1 0   315065      0  70.6mb  70.6mb
green open .monitoring-es-7-2020.07.29        oYHgC5b3RmWtohITTTPUyg 1 1   340146 123552 336.5mb 169.2mb
green open elastalert_status_error            wEDL-7gtSICu89FP2_1R6g 1 0       17      0  19.3kb  19.3kb
green open winlogbeat-7.6.2-2020.07.24-000030 cBjBAaNeTZOqEfNfJzvvLQ 1 0   172738      0    62mb    62mb
green open .monitoring-kibana-7-2020.07.28    3h_TWlFwSQyBM7yrVugxAQ 1 1    10057      0     5mb   2.5mb
green open .monitoring-kibana-7-2020.07.27    Iq79FbPMRKiO6H3qiOz3IA 1 1     9345      0   4.8mb   2.2mb
green open winlogbeat-7.6.2-2020.07.31-000035 rF-qytLcQ_aScTEb8wJPug 1 0    59245      0  20.2mb  20.2mb
green open .monitoring-kibana-7-2020.07.22    dKgyQ0Y_TQW_awAe6ES3dw 1 1    10017      0   5.4mb   2.6mb
green open .monitoring-kibana-7-2020.07.21    mgRBJx2ISn6kGQq4-3mGZA 1 1    10048      0   4.9mb   2.4mb
green open .monitoring-kibana-7-2020.07.24    0fsSCLIzTrSJRYUbE-QjRg 1 1     8959      0   5.2mb   2.6mb
green open .apm-agent-configuration           HZyRgvFPQlq5Q8cbqPPBLA 1 0        0      0    283b    283b
green open .monitoring-kibana-7-2020.07.23    F1nVx2ONQQKU4bHDZKlN-A 1 1    10049      0   5.5mb   2.7mb
green open cisco-filebeat-2020.07.29          kgGtZekrRbawNJbNYCse_w 1 0 26206102      0   7.5gb   7.5gb
green open filebeat-7.6.2-2020.08.14-000051   U727_AyZRl2EYQT1dn0FuA 1 0     7290      0   1.9mb   1.9mb
green open metricbeat-7.6.2-2020.07.13-000002 6fner3r4QyCpkMKIt7xGJA 1 0        0      0    283b    283b
green open .monitoring-kibana-7-2020.07.29    20BruRWQTESCQnAuhWvVgg 1 1    10057      0   5.5mb   2.7mb
green open .monitoring-es-7-2020.07.30        VsLT6onBQXaO6J7rRi5pww 1 1   331552 128025 322.2mb 160.2mb
green open oracle_log                         tQSV3wcoQ8uSxxSmM6X4NA 1 0 14748275      0   9.1gb   9.1gb
green open cisco-filebeat-2020.07.31          9uKhR_F1T4eGyfcju5L19g 1 0 22918033      0   6.5gb   6.5gb
green open .monitoring-es-7-2020.07.31        8MrXkOlnQF-rggITj3EBPA 1 1   324899 127980 303.6mb 151.8mb
green open cisco-filebeat-2020.07.30          ss-4WQ6LTQuGt7jDksxDRw 1 0 21752947      0   6.2gb   6.2gb
green open cisco-filebeat-2020.08.04          dqumwhu3TVqXkYnio28CYQ 1 0 24926569      0   7.1gb   7.1gb
green open ilm-history-1-000002               YCBhIQL0STi6-caKboTG2A 1 1      965      0 837.3kb 418.6kb
green open .monitoring-es-7-2020.08.03        Uam4z4waQwKwwHUtaqP5-Q 1 1   361395 172557 336.1mb 168.9mb
green open cisco-filebeat-2020.08.03          CqyZSgjIT7eqG26YFnSP3g 1 0 24528554      0   6.9gb   6.9gb
green open .monitoring-es-7-2020.08.04        qNjvqKfuRy2JRKJP6U-cKA 1 1   417252 234294 391.8mb 196.6mb
green open ilm-history-1-000001               HAwEquA4RI2BhsIAVh4-aA 1 0     8308      0   4.3mb   4.3mb
green open .monitoring-es-7-2020.08.05        2YPHPBvGS4qL2nQu263-Lw 1 1   434977 270644 398.8mb 197.5mb
green open .monitoring-es-7-2020.08.06        l9FZTIUSSb2THD5PUpZMng 1 1   474903  38918 395.6mb 197.9mb
green open .monitoring-es-7-2020.08.07        wSedTxtzSoWO-z4Zz0cTiw 1 1   473228  36287 390.2mb 195.1mb
green open .monitoring-kibana-7-2020.07.20    nWgNHPHNQ_O6CEFx18iAHQ 1 1     8678      0   4.5mb   2.2mb
green open .slm-history-1-000001              PFL1QLAiTHKVBHhLrf2VCw 1 0       66      0 104.3kb 104.3kb
green open .monitoring-kibana-7-2020.08.06    famgmCOpRN2BzoUrztTRyA 1 1    10052      0   4.9mb   2.4mb
green open .monitoring-kibana-7-2020.08.07    B54L0uwOTqG8o0i-hgZDJg 1 1     9324      0   4.5mb   2.2mb
green open .monitoring-kibana-7-2020.08.04    YtEE7bYHTti3LriBwEPmpw 1 1    10053      0   4.9mb   2.4mb
green open .monitoring-kibana-7-2020.08.05    j2SY55QPRvSzBM4rc3DzxQ 1 1    10052      0   4.8mb   2.4mb
green open .monitoring-kibana-7-2020.08.03    5bWmRCfqRIiPiXttORZD8A 1 1     9344      0   4.6mb   2.3mb
green open filebeat-7.6.2-2020.08.06-000045   qJHVNmc2RR-FIdZz0ckydQ 1 0   407339      0    93mb    93mb
green open filebeat-7.6.2-2020.08.05-000044   L0roVXrjSRqrXg2ysRGerg 1 0   407678      0  93.5mb  93.5mb
green open winlogbeat-7.6.2-2020.08.14-000045 DXtDjO36SCuTfJ4ew_kPDA 1 0      131      0 178.5kb 178.5kb
green open filebeat-7.6.2-2020.07.28-000038   Mht7kFm4R-qkmudSfQ0k-A 1 0   314248      0  67.8mb  67.8mb
green open .monitoring-kibana-7-2020.07.31    WEOxf_RyQ2C6DD6xo_9WXw 1 1     9330      0   4.4mb   2.2mb
green open filebeat-7.6.2-2020.07.29-000039   h3tByCkxSPOu_kfueOi8KQ 1 0   310202      0  68.6mb  68.6mb
green open .monitoring-kibana-7-2020.07.30    9SFSFeswRF-oVrzORC9jQA 1 1    10055      0   5.3mb   2.6mb
green open winlogbeat-7.6.2-2020.08.11-000042 AwuJH_wTRgSixEPY7fJqaA 1 0    34424      0  11.9mb  11.9mb
green open winlogbeat-7.6.2-2020.08.10-000041 gMpiIjnFRymlXcBKJ9gGhQ 1 0    81319      0  26.8mb  26.8mb
green open .monitoring-kibana-7-2020.08.13    ESuEqwyWS020w-VaMzBB7w 1 1    10019      0     5mb   2.5mb
green open winlogbeat-7.6.2-2020.07.30-000034 Fe_gID3nR_WTdHSoXv6XjA 1 0    34789      0    12mb    12mb
green open .monitoring-kibana-7-2020.08.14    gU5vYmHYRx2t7xhx_25HGg 1 1        9      0 525.9kb 271.2kb
green open .monitoring-kibana-7-2020.08.11    5Z4VYte0RKaUFdZAc6A8FQ 1 1    10044      0   4.9mb   2.5mb
green open .monitoring-kibana-7-2020.08.12    oxHfSaPUR-KsuV_3HqYFXA 1 1    10050      0     5mb   2.4mb
green open .kibana_1                          DNAifhlMSE68w5I6NG-toQ 1 0     4010     12   2.1mb   2.1mb
green open filebeat-7.6.2-2020.08.13-000050   JkO8vCQ5SRiibJUAVZJfCw 1 0   309341      0  67.8mb  67.8mb
green open .monitoring-kibana-7-2020.08.10    QWVnm8h2RX-zikPgOzovOA 1 1     9337      0   4.7mb   2.3mb

Hi Shaunak,
Cisco indexes are getting created successfully for the same config

Thanks, I see that the cisco-filebeat-* indices are being created but there are no flowlog-* indices at all.

Are you finding events/documents intended to be indexed in the flowlog-* indices in the cisco-filebeat-* indices instead? Or those events/documents not being indexed at all? Basically, I'm trying to figure out if there is a field called log.level in the events being produced by Filebeat and whether it ever has the value of informational.

Hi Shaunak,

Yes , the events are getting generated. Please see the screenshot attached

log_level

We need to retain informational logs only 2 weeks , Other logs like Notification , Alarm , error etc has be to retained for 1 year . Hence we are trying to split this.

Thanks,
Jijo

Hi Shaunak,

Did you got any chance to look at the event screenshot.

Thanks,
Jijo John

I see. Hmm, I don't see anything wrong with the configuration.

Could you temporarily comment out the index: "cisco-filebeat-%{+yyyy.MM.dd}" setting in your filebeat.yml? Let's see if that results in the flowlog-* indices being created and events with log.level: informational being indexed into them.

Thanks Shaunak, Let me try that option also. we are also planning to have below option if this is not working.

Setup another syslog server and point Cisco (Cisco ASA can be pointed to multiple syslog server ) to this new syslog server and drop all the events except informational so that we can get a separate index with only informational logs. Do you this is a feasible solution.

Below is the Option in detail

  1. First Syslog server running Cisco asa module - we will drop only informational Logs (how we can do this )

  2. Second Syslog server running Cisco asa module - drop all the events except informational (how we can do this )

  3. Also the rollover to warm node is not happening since the custom indices dont have the rollover number assigned when created

Default Index :
filebeat-7.6.2-2020.08.18-000054

Custom Index : cisco-filebeat-2020.08.18 (no rollover number )

Please help us with the above so that we close off the POC and proceed with procurement and production setup

Thanks,
Jijo John

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.