(?<=StatusCode=\")\d+)"
]
}
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
#ilm_enabled => false
index => "DataEdgeAppServer"
#index => "%{[fields][log_type]}"
}
stdout {
codec => rubydebug
}
}
```
filebeat.yml-
```auto
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
input_type: log
fields:
tags: ["DataEdgeApp"]
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'C:\Program Files (x86)\ESQ SST\DataEdgev1.2\ngta-distribution-app-3.2.0.0-bin\logs\*'
#- C:\ngta.log
#- 'C:\Program Files (x86)\ESQ SST\DataEdgev1.2\ngta-distribution-web-3.2.0.0-bin\logs\ngta.log'
#- c:\programdata\elasticsearch\logs\*
- type: log
enabled: true
input_type: log
fields:
tags: ["DataEdgeWeb"]
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'C:\Program Files (x86)\ESQ SST\DataEdgev1.2\ngta-distribution-web-3.2.0.0-bin\logs\*'
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["192.168.x.x:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 24, 2020, 3:51am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/69 "2020-01-24T03:51:26Z")
Is this the same logstash server?
if yes, you can include the filters in your existing pipeline.conf file and see whether it is taking effect.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 24, 2020, 4:14am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/70 "2020-01-24T04:14:58Z")
@mancharagopan No, logstash is on the ubuntu on my local machine which we worked on. FIlebeat is on a remote and different server. And sometimes the index is just logstash and other times it is logstash-date-00001.
Previously the index issue resolved after I refreshed my log files passed in filebeat. But right now, the files being passed are real time logs.
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 24, 2020, 4:23am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/71 "2020-01-24T04:23:56Z")
did you try uncommenting
> [@Mehak\_Bhargava](#):
>
> #ilm\_enabled =\> false
create a separate pipeline for the new config file in your pipeline.yml file.
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 24, 2020, 4:29am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/72 "2020-01-24T04:29:11Z")
> [@mancharagopan](#):
>
> did you try uncommenting
I did uncomment and then the index went from logstahs to logstash-date-0001.
> [@mancharagopan](#):
>
> new config file
which new config file? I just updated my old pipeline.conf which had errors, access, dispatcher with the new file I posted above. So there is only one pipeline.conf used. And the pipeline.yml has one pipeline.id defined.
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 24, 2020, 4:44am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/73 "2020-01-24T04:44:07Z")
Can you check the logstash logs whether the configuration file is loaded correctly and if any errors occurred while parsing?
What are the event stored in logstash index or logstash-date-0001 index?
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 24, 2020, 4:54am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/74 "2020-01-24T04:54:20Z")
The data in the logstash and logstash-date-0001 was what the fikebeat was passing from the server. So the content under index is exactly what we wanted. And even the log\_type i have added under field in logstash as "DataEdgeApp" was shown in kibana.
Although this all worked when I deleted the else if part and only had if.
No error is seen in logstash conf. Anything specific I should see to see file loading?
---
### Author:  [@mancharagopan](https://discuss.elastic.co/u/mancharagopan)
#### Post date: [January 24, 2020, 6:44am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/75 "2020-01-24T06:44:03Z")
[See This](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/49)
---
### Author:  [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)
#### Post date: [January 25, 2020, 1:34am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/76 "2020-01-25T01:34:10Z")
Worked again as main was taken in pipeline.id. SO, when I run ./logstash -f pipeline.conf, it ignored the pipeline.yml file which defines the pipeline.conf location.
Best way is to just run ./logstash and then pipelien.yml will be read with correct pipeline.id:test.
Thanks again!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 22, 2020, 1:34am UTC](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806/77 "2020-02-22T01:34:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
[Previous page](https://discuss.elastic.co/t/all-files-collected-under-on-index-only-is-it-possible-to-have-multiple-indexes-for-multiple-files/212806.md?page=3)