Creating indices of older days too

Elastic Stack Logstash
1

Hi

I have deployed filebeat and logstash helm chart in eks.

I am sending logs to amazon es.

I am facing a similar issue  https://github.com/uken/fluent-plugin-elasticsearch/issues/482, but in filebeat do we have option like fulentd

Here is the filebeat and logstash configs.

filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:   
      - type: container
        paths:
          - /var/log/containers/*.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_event.when:
            or:
            - equals:
                kubernetes.container.name: "filebeat"
            - equals:  
                kubernetes.container.name: "logstash"        
      output.logstash:
        host: '${NODE_NAME}'
        hosts: ["logstash.logstash.svc.cluster.local:5044"]
  

Logstash configuration
`
image: "xxxx/devops/logstash-kubernetes/logstash"
imageTag: "7.11.1" 
fullnameOverride: "logstash"
podLabels:
  service: "logstash"
rbac:
  create: true  
  serviceAccountName: logstash 
service:
  type: ClusterIP
  ports:
    - name: beat-me
      port: 5044
      targetPort: 5044
      protocol: TCP
    - name: http
      port: 8080
      targetPort: 8080
      protocol: TCP
persistence:
  enabled: true

logstashPipeline:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }
    filter {
      dissect {
        mapping => {
            "message" => "%{timestamp} | %{level} | %{thread} | %{category} | %{message}"
        }
      }
      mutate {
        remove_field => ["[kubernetes][annotations]", "headers", "ecs", "[agent][ephemeral_id]", "[agent][id]", "[agent][version]", "[container][id]","[container][runtime]","[input][type]", "[kubernetes][pod][uid]", "[log][file][path]", "[log][offset]","tags","[agent][hostname]","[agent][name]","[agent][type]","[host][name]","[kubernetes][labels][pod-template-hash]","[kubernetes][labels][service_istio_io/canonical-revision]","[kubernetes][labels][pod-template-hash]","[kubernetes][labels][istio_io/rev]","[kubernetes][container][image]","[kubernetes][labels][security_istio_io/tlsMode]","[kubernetes][labels][service_istio_io/canonical-name]","[kubernetes][labels][heritage]" ]
      }
      mutate {
        add_field => {
          "cluster_name" => "xxx"
          "environment" => "xxx"
          "region" => "us-west-2"
        }
      }
    }
    output {
      amazon_es {
        hosts => ["vpc-xxxxx"]
        region => "us-east-1"
        port => 443
        protocol => "https"
        index => "%{cluster_name}-%{region}-%{environment}-%{[kubernetes][labels][app]}-%{+YYYY.MM.dd}"
      }
      stdout { codec => rubydebug }
    }


I am creating daily indices, If I install the filebeat and logstash  on mar 07 2021, it's still creating the indices of older dates up to week back dates. 
green open xxxx_ddog-agent_2021.03.05 CHCWjCmKRYq1uoKcpmdEuQ 1 2  61550 0  70.1mb 23.4mb
green open xxx_ddog-agent_2021.03.04 vF7J2MeWS8m-wFrZc4iydg 1 2 121822 0 136.8mb 45.7mb
green open rokumesh-merge-request-test_merge-request-test_ddog-agent_2021.03.07 YeHpPqubTii_OS_vlwOqvg 1 2  11247 0  14.8mb  5.1mb
green open xxxx_ddog-agent_2021.03.06 gYxCtPiMSfi7CDxqsxcLGw 1 2  11963 0  14.9mb    5mb
green open xxxx_ddog-agent_2021.03.03 FnscWP4GTjq7y4_p8CgdNg 1 2 122359 0 134.8mb   45mb
green open xxx_ddog-agent_2021.03.02 wtpJeht_RnOlgSKP5LdOKQ 1 2  32883 0  33.4mb 11.2mb

Thanks in adavce.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.