# Ingest log in specific index **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 21, 2023, 3:08am UTC](https://discuss.elastic.co/t/ingest-log-in-specific-index/347571 "2023-11-21T03:08:29Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Nitin08bisht](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@Nitin08bisht](https://discuss.elastic.co/u/Nitin08bisht)\ **Post date:** [November 21, 2023, 3:08am UTC](https://discuss.elastic.co/t/ingest-log-in-specific-index/347571/1 "2023-11-21T03:08:29Z")
Hi, I'm using ELK version 7.16.2 and I have configured Filebeat and I want to ingest log on dedicated index rather than on default filebeat index. Please help me on this. Please find the `filebeat.ym`l file configuration below. ```auto ###################### Filebeat Configuration Example ######################### # ============================== Filebeat inputs =============================== filebeat.inputs: - type: filestream enabled: true paths: #- /var/log/*.log - D:\Elastic\kibana-7.16.2-windows-x86_64\Audit\*.log max_bytes: 104857600000 # ============================== Filebeat modules ============================== filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false # ======================= Elasticsearch template setting ======================= setup.template.settings: index.number_of_shards: 1 # ================================= Dashboards ================================= setup.dashboards.enabled: true # ================================== Template ================================== setup.template.name: "test" setup.template.pattern: "test-*" setup.template.settings: # =================================== Kibana =================================== setup.kibana: host: "http:// ******** :5601" protocol: "http" # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: # Array of hosts to connect to. hosts: ["http:// ******* :9200/"] username: " *********" password: " *********" index: "test-%{[agent.version]}-%{+yyyy.MM.dd}" # ================================= Processors ================================= processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ - decode_json_fields: fields: ["message"] process_array: false max_depth: 2 target: "" overwrite_keys: false add_error_key: true ``` Thanks, Nitin ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 21, 2023, 6:18am UTC](https://discuss.elastic.co/t/ingest-log-in-specific-index/347571/2 "2023-11-21T06:18:09Z")
Perhaps take a look at > [@Filebeat writing to its own index](https://discuss.elastic.co/t/filebeat-writing-to-its-own-index/310842/28): > > As explained in the post I linked, You have a moduled enabled in your configuration whether it is important now or not... It overrides the output settings in some cases but let's put that aside there are other issues... Also from the docs [here](https://www.elastic.co/guide/en/beats/filebeat/7.12/elasticsearch-output.html#index-option-es)... which is your key issue... When [index lifecycle management (ILM)](https://www.elastic.co/guide/en/beats/filebeat/7.12/ilm.html) is enabled, the default index is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index\_num}" , for example, "filebeat-7.12.1-2022-07-28-000001" . Custom index settings are ignored when… ---
**Author:** ![Nitin08bisht](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@Nitin08bisht](https://discuss.elastic.co/u/Nitin08bisht)\ **Post date:** [November 22, 2023, 6:16pm UTC](https://discuss.elastic.co/t/ingest-log-in-specific-index/347571/3 "2023-11-22T18:16:58Z")
Hi @stephenb Thanks for your support. I have followed 1st approach from below two approaches... 1. Set it all up in filebeat (when you do this create a default ILM policy for you, which you can later edit) 2. setup your template, policy and rollover alias etc... etc. in elasticsearch then use minimal filebeat config I have implemented the **approach 1** and configure `filebeat.yml` file (See below configuration). Its working and now, logs being ingested into dedicated index. ```auto # ============================== Filebeat inputs =============================== filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. # filestream is an input for collecting log messages from files. - type: filestream # Change to true to enable this input configuration. enabled: true paths: #- /var/log/*.log - D:\Users\ABC\Downloads\audit-logs*.csv #fields: # category: audit_log #json.keys_under_root: true #json.overwrite_keys: true #json.add_error_key: true #json.expand_keys: true max_bytes: 104857600000 # ============================== Filebeat modules ============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s # ======================= Elasticsearch template setting ======================= setup.template.settings: index.number_of_shards: 1 #index.codec: best_compression #_source.enabled: false # ================================= Dashboards ================================= # These settings control loading the sample dashboards to the Kibana index. Loading # the dashboards is disabled by default and can be enabled either by setting the # options here or by using the `setup` command. setup.dashboards.enabled: true # The URL from where to download the dashboards archive. By default this URL # has a value which is computed based on the Beat name and version. For released # versions, this URL points to the dashboard archive on the artifacts.elastic.co # website. #setup.dashboards.url: # =================================== Kibana =================================== # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. # This requires a Kibana endpoint configuration. setup.kibana: #setup.kibana.host: "http://localhost:5601/" #setup.kibana.protocol: "http" host: "http://localhost:5601/" #space.id: log protocol: "http" # ================================== Outputs =================================== # Configure what output to use when sending the data collected by the beat. # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: # Array of hosts to connect to. hosts: ["http://localhost:9200/"] # Protocol - either `http` (default) or `https`. #protocol: "http" # Authentication credentials - either API key or username/password. #api_key: "id:api_key" username: " ********" password: " ********" setup.ilm: enabled: true policy_name: "myindex" overwrite: true rollover_alias: "myindex-alias-%{[agent.version]}" pattern: "{now/d}-0000001" # ================================= Processors ================================= processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ - decode_json_fields: fields: ["message"] process_array: false max_depth: 2 target: "" overwrite_keys: false add_error_key: true ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 20, 2023, 8:17pm UTC](https://discuss.elastic.co/t/ingest-log-in-specific-index/347571/4 "2023-12-20T20:17:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.