# Need to define different index name

**URL:** <https://discuss.elastic.co/t/need-to-define-different-index-name/186451>\
**Category:** Logstash\
**Created:** [June 19, 2019, 11:29am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451 "2019-06-19T11:29:48Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 19, 2019, 11:29am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/1 "2019-06-19T11:29:48Z")

</div>

I am running with ELK cluster 5.6.16.

My system as below  
Filebeat --\> Logstash --\> Elasticsearch (3 server in cluster) --\> KIbana

I have few servers where many web vhosts are configured. From that server, I have configured different doc\_type (abc-access , efg-access & xyz-access ) defined into filebeat.yml and defined index abc into file-beat configuration. So, All logs coming through that server goes into same index file i.e.

> abc

On the same server, I have defined new doc-type i.e mno-access & I want to send all which matching to same doc-type goes to new index.

> i.e. mno

I searched a lots & I didn't find anything where, we can defined new index name under filebeat.yml configuration.

So, I have decided to handle this through logstash. For that, I have setup filter into logstash & using if condition try to define new index name with matching to mno-access doc-type. But that \_index meta-field doesn't changed. I tried to overwrite, remore\_field options but that didn't worked. Kindly help, How to defined different index name here.

- Logstash-output section

> output {  
> if [type] =~ "mno-access" {
> 
> elasticsearch {  
> hosts =\> ["192.168.2.121:9200", "192.168.2.122:9200", "192.168.2.123:9200"]  
> sniffing =\> true  
> manage\_template =\> true  
> index =\> "mno-%{+YYYY.MM.dd}"  
> document\_type =\> "%{[@metadata][type]}"  
> }  
> }  
> else {  
> elasticsearch {  
> hosts =\> ["192.168.2.121:9200", "192.168.2.122:9200", "192.168.2.123:9200"]  
> sniffing =\> true  
> manage\_template =\> true  
> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
> document\_type =\> "%{[@metadata][type]}"  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![robymemo](https://avatars.discourse-cdn.com/v4/letter/r/82dd89/32.png) [@robymemo](https://discuss.elastic.co/u/robymemo)\
**Post date:** [June 19, 2019, 3:57pm UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/2 "2019-06-19T15:57:27Z")

</div>

have you solved this ?

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 20, 2019, 4:02am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/3 "2019-06-20T04:02:23Z")

</div>

Not yet.. I am still getting abc index name against doc\_type mno-access.

---

<div class="post-metadata">

**Author:** ![robymemo](https://avatars.discourse-cdn.com/v4/letter/r/82dd89/32.png) [@robymemo](https://discuss.elastic.co/u/robymemo)\
**Post date:** [June 20, 2019, 5:17am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/4 "2019-06-20T05:17:27Z")

</div>

Can you post your filebeat conf?

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 20, 2019, 5:25am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/5 "2019-06-20T05:25:28Z")

</div>

PS : We can't remove defined index name from filebeat configuration as this was configured 1 year back & many scripts are already defined with this name. That's why, we are looking for solution into logstash.

Filebeat configuration :

> filebeat.prospectors:
> 
> - input\_type: log  
> paths:
> - /var/log/nginx/www.access.log  
> document\_type: abc-access
> 
> - input\_type: log  
> paths:
> - /var/log/nginx/akbingbot\_website.access.log  
> document\_type: mno-access
> 
> - input\_type: log  
> paths:
> - /var/log/nginx/akamai-wap.access.log  
> document\_type: xyz-access
> 
> - input\_type: log  
> paths:
> - /var/log/php-fpm/wap-error.log  
> include\_lines: ['Fatal error']  
> document\_type: wap-fpm-error
> 
> - input\_type: log  
> paths:
> - /var/log/php-fpm/www-error.log  
> include\_lines: ['Fatal error']  
> document\_type: php-fpm-error  
> output.logstash:  
> hosts: ["192.168.2.122:5044", "192.168.2.121:5044"]  
> loadbalance: true  
> index: abc  
> ssl.certificate\_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt", "/etc/pki/tls/certs/logstash-forwarder\_122.crt"]

---

<div class="post-metadata">

**Author:** ![robymemo](https://avatars.discourse-cdn.com/v4/letter/r/82dd89/32.png) [@robymemo](https://discuss.elastic.co/u/robymemo)\
**Post date:** [June 20, 2019, 6:31am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/6 "2019-06-20T06:31:04Z")

</div>

Try tu use the fields instead of document\_type  
fields:  
document\_type: mno-access  
fields\_under\_root: true

In logstash conf this in the if :  
If "mno-access" in [field][document\_type]...

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 20, 2019, 7:25am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/7 "2019-06-20T07:25:17Z")

</div>

Is my understanding correct ?

> - input\_type: log  
> paths:
> - /var/log/nginx/www.access.log  
> document\_type: abc-access
> 
> - input\_type: log  
> paths:
> - /var/log/nginx/akbingbot\_website.access.log  
> fields:  
> document\_type: mno-access  
> fields\_under\_root: true
> 
> - input\_type: log  
> paths:
> - /var/log/nginx/akamai-wap.access.log  
> document\_type: xyz-access

Logstash config :

> output {
> 
> ### # if [type] =~ "mno-access" {
> 
> ```
> if "mno-access" in [field][document_type] {
> 
> elasticsearch {
> hosts => ["192.168.2.121:9200", "192.168.2.122:9200", "192.168.2.123:9200"]
> sniffing => true
> manage_template => true
> index => "mno-%{+YYYY.MM.dd}"
> document_type => "mno-access"
> }
> }
> else {
> elasticsearch {
> hosts => ["192.168.2.121:9200", "192.168.2.122:9200", "192.168.2.123:9200"]
> sniffing => true
> manage_template => true
> index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
> document_type => "%{[@metadata][type]}"
> }
> }
> 
> ```
> 
> }

---

<div class="post-metadata">

**Author:** ![robymemo](https://avatars.discourse-cdn.com/v4/letter/r/82dd89/32.png) [@robymemo](https://discuss.elastic.co/u/robymemo)\
**Post date:** [June 20, 2019, 8:34am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/8 "2019-06-20T08:34:10Z")

</div>

yes ... working ?

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 20, 2019, 8:55am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/9 "2019-06-20T08:55:50Z")

</div>

Nope. I got error in filebeat.yml.

> Jun 20 14:21:49 12-108-IDC filebeat: filebeat2019/06/20 08:51:49.985148 beat.go:346: CRIT Exiting: error loading config file: yaml: line 28: did not find expected '-' indicator  
> Jun 20 14:21:49 12-108-IDC filebeat: Exiting: error loading config file: yaml: line 28: did not find expected '-' indicator  
> Jun 20 14:21:49 12-108-IDC systemd: filebeat.service: main process exited, code=exited, status=1/FAILURE

> - input\_type: log  
> paths:
> - /var/log/nginx/akbingbot\_website.access.log ### - Line no 28.  
> fields:  
> document\_type: mno-access  
> fields\_under\_root: true

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 20, 2019, 9:07am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/10 "2019-06-20T09:07:16Z")

</div>

Filebeat service started working after making changes as below. But logs are not getting into kibana.

```
- input_type: log
  paths:
    - /var/log/nginx/akbingbot_website.access.log
  fields:
    document_type: mno-access
  fields_under_root: true
```

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 20, 2019, 1:45pm UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/11 "2019-06-20T13:45:20Z")

</div>

I have enabled debug on filebeat & found that, logs are getting pushed to logstash server. but didn't get inserted into elasticsearch.

```
2019/06/20 13:01:18.117779 spooler.go:89: DBG Flushing spooler because of timeout. Events flushed: 4
2019/06/20 13:01:18.118206 client.go:214: DBG Publish: {
  "@timestamp": "2019-06-20T13:01:13.118Z",
  "beat": {
    "hostname": "12-108-IDC.justdial.com",
    "name": "12-108-IDC.justdial.com",
    "version": "5.6.16"
  },
  "document_type": "mno-access",
  "input_type": "log",
  "message": "40.77.25.145, 67.220.142.5 - - [04/Jun/2019:23:02:05 +0530] \"GET /Chandigarh/Hotels HTTP/1.1\" 200 314832 \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 (BingLocalSearch)\" \"REMOTE_ADDR : 67.220.142.5\" \"TRUE_CLIENT : 40.77.25.145\" \"AKAXFF : -\" www.justdial.com 0.491 0.393 US \"AKBINGBT : yes\" .",
  "offset": 47470,
  "source": "/var/log/nginx/akbingbot_website.access.log",
  "type": "log"
}
2019/06/20 13:01:18.118349 client.go:214: DBG Publish: {
  "@timestamp": "2019-06-20T13:01:13.118Z",
  "beat": {
    "hostname": "12-108-IDC.justdial.com",
    "name": "12-108-IDC.justdial.com",
    "version": "5.6.16"
  },
  "document_type": "mno-access",
  "input_type": "log",
  "message": "40.77.25.145, 67.220.142.5 - - [04/Jun/2019:23:02:08 +0530] \"GET /Delhi/mtnl-in-jamia-nagar HTTP/1.1\" 302 5 \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 (BingLocalSearch)\" \"REMOTE_ADDR : 67.220.142.5\" \"TRUE_CLIENT : 40.77.25.145\" \"AKAXFF : -\" www.justdial.com 0.542 0.542 US \"AKBINGBT : yes\" .",
  "offset": 47833,
  "source": "/var/log/nginx/akbingbot_website.access.log",
  "type": "log"
}
2019/06/20 13:01:18.118550 client.go:214: DBG Publish: {
  "@timestamp": "2019-06-20T13:01:13.119Z",
  "beat": {
    "hostname": "12-108-IDC.justdial.com",
    "name": "12-108-IDC.justdial.com",
    "version": "5.6.16"
  },
  "document_type": "mno-access",
  "input_type": "log",
  "message": "40.77.25.145, 69.174.30.168 - - [04/Jun/2019:23:02:09 +0530] \"GET /Delhi/search?q=mtnl HTTP/1.1\" 200 285678 \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 (BingLocalSearch)\" \"REMOTE_ADDR : 69.174.30.168\" \"TRUE_CLIENT : 40.77.25.145\" \"AKAXFF : -\" www.justdial.com 0.612 0.612 US \"AKBINGBT : yes\" .",
  "offset": 48197,
  "source": "/var/log/nginx/akbingbot_website.access.log",
  "type": "log"
}
2019/06/20 13:01:18.118634 output.go:109: DBG output worker: publish 3 events
2019/06/20 13:01:18.118661 context.go:93: DBG forwards msg with attempts=-1
2019/06/20 13:01:18.118733 context.go:98: DBG message forwarded
2019/06/20 13:01:18.118754 context.go:138: DBG events from worker worker queue
2019/06/20 13:01:18.120342 sync.go:96: DBG 3 events out of 3 events sent to logstash host 192.168.2.122:5044:10200. Continue sending
```

---

<div class="post-metadata">

**Author:** ![Nishant\_Hurnekar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant_hurnekar/32/47955_2.png) [@Nishant\_Hurnekar](https://discuss.elastic.co/u/Nishant_Hurnekar)\
**Post date:** [June 27, 2019, 10:58am UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/12 "2019-06-27T10:58:35Z")

</div>

Can anyone help out here ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 27, 2019, 1:59pm UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/13 "2019-06-27T13:59:40Z")

</div>

> [@Nishant\_Hurnekar](#):
>
> if "mno-access" in [field][document\_type] {

If you have set fields\_under\_root in filebeat.yml then you should be testing [document\_type]

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 25, 2019, 1:59pm UTC](https://discuss.elastic.co/t/need-to-define-different-index-name/186451/14 "2019-07-25T13:59:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.