My system as below
Filebeat --> Logstash --> Elasticsearch (3 server in cluster) --> KIbana
I have few servers where many web vhosts are configured. From that server, I have configured different doc_type (abc-access , efg-access & xyz-access ) defined into filebeat.yml and defined index abc into file-beat configuration. So, All logs coming through that server goes into same index file i.e.
abc
On the same server, I have defined new doc-type i.e mno-access & I want to send all which matching to same doc-type goes to new index.
i.e. mno
I searched a lots & I didn't find anything where, we can defined new index name under filebeat.yml configuration.
So, I have decided to handle this through logstash. For that, I have setup filter into logstash & using if condition try to define new index name with matching to mno-access doc-type. But that _index meta-field doesn't changed. I tried to overwrite, remore_field options but that didn't worked. Kindly help, How to defined different index name here.
PS : We can't remove defined index name from filebeat configuration as this was configured 1 year back & many scripts are already defined with this name. That's why, we are looking for solution into logstash.
Jun 20 14:21:49 12-108-IDC filebeat: filebeat2019/06/20 08:51:49.985148 beat.go:346: CRIT Exiting: error loading config file: yaml: line 28: did not find expected '-' indicator
Jun 20 14:21:49 12-108-IDC filebeat: Exiting: error loading config file: yaml: line 28: did not find expected '-' indicator
Jun 20 14:21:49 12-108-IDC systemd: filebeat.service: main process exited, code=exited, status=1/FAILURE
input_type: log
paths:
/var/log/nginx/akbingbot_website.access.log ### - Line no 28.
fields:
document_type: mno-access
fields_under_root: true
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.