# Unable to change index off default logstash-\*

**URL:** <https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989>\
**Category:** Logstash\
**Created:** [April 21, 2016, 3:42am UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989 "2016-04-21T03:42:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![kprime](https://avatars.discourse-cdn.com/v4/letter/k/76d3ee/32.png) [@kprime](https://discuss.elastic.co/u/kprime)\
**Post date:** [April 21, 2016, 3:42am UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/1 "2016-04-21T03:42:07Z")

</div>

Hello,

I am trying to get logstash to not use the default index but it isn't working for me. I have a JSON template that is successful updating in ES but logstash keeps trying to use the default index and parsing errors occur. I have set the index on the ES output but I am not sure why it is skipping it.

Error:

> {:timestamp=\>"2016-04-20T22:46:04.554000-0400", :message=\>"Failed action. ", :status=\>400, :action=\>["index", {:\_id=\>nil, :\_index=\>" **logstash-2016.04.21**", :\_type=\>"logs", :\_routing=\>nil}, #\<LogStash::Event:0x180d5390 @metadata\_accessors=#\<LogStash::Util::Accessors:0x406ba261 @store={"path"=\>"/events-test.csv"}, @lut={"[path]"=\>[{"path"=\>"/events-test.csv"}, "path"]}\>, @cancelled=false, @data={"message"=\>"

Input and output:

> input {  
> file {  
> path =\> "/events-test.csv"  
> start\_position =\> "beginning"  
> type =\> "firepower"  
> }  
> }

> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> template =\> "/firepower.json"  
> index =\> "firepower"  
> template\_name =\> "firepower"  
> manage\_template =\> true  
> template\_overwrite =\> true  
> workers =\> 1  
> }

I deleted all my templates to double check that the template was being created. It is indeed being done.

curl -XGET '[http://localhost:9200/\_template/\*?pretty](http://localhost:9200/_template/*?pretty)'

> {  
> "firepower" : {  
> "order" : 0,  
> "template" : "firepower\*",  
> "settings" : { },  
> "mappings" : {  
> "firepower" : {  
> "properties" : {  
> "security\_zone\_egress\_uuid" : {  
> "index" : "not\_analyzed",  
> "type" : "string"  
> },  
> \<! more in here !\>

I also created the index:

curl 'localhost:9200/\_cat/indices?v'  
health status index pri rep docs.count docs.deleted store.size pri.store.size  
yellow open .kibana 1 1 1 0 3.1kb 3.1kb  
yellow open firepower 5 1 0 0 795b 795b

Thoughts??

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 21, 2016, 6:08pm UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/2 "2016-04-21T18:08:42Z")

</div>

Wild guess: You have another outdated file in /etc/logstash/conf.d that lacks the `index => "firepower"` option for its output. If that's not it, start Logstash with `--debug` to see exactly what configuration it loads.

---

<div class="post-metadata">

**Author:** ![kprime](https://avatars.discourse-cdn.com/v4/letter/k/76d3ee/32.png) [@kprime](https://discuss.elastic.co/u/kprime)\
**Post date:** [April 21, 2016, 7:59pm UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/3 "2016-04-21T19:59:25Z")

</div>

There are no other configuration files and looking at the logs it is very quiet.

So I booted it up in debug and went through the mountain of text and found this...

> :response=\>{"create"=\>{"\_index"=\>"firepower", "\_type"=\>"firepower", "\_id"=\>"AVQ6WKjSHNuH89\_3Iwfv", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse [@timestamp]", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Invalid format: "2016-04-21T19:42:48.652Z" is malformed at "-04-21T19:42:48.652Z""}}}}, :level=\>:warn, :file=\>"logstash/outputs/elasticsearch/common.rb", :line=\>"119", :method=\>"submit"}

I do have a date filter setup, it must be wrong.

> ```
> date {
> match => ["date", "ISO8601"]
> target => "date"
> }
> 
> ```
> 
> }

from the debug:

> Date/@match = ["date", "ISO8601"]", :level=\>:debug,

I just read a post you had two months back and I am going to be deleting the timestamp:

> date {  
> locale =\> "en"  
> match =\> ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]  
> remove\_field =\> ["timestamp"]  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 21, 2016, 8:43pm UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/4 "2016-04-21T20:43:04Z")

</div>

No, I don't think that's it. It's the `@timestamp` field it's complaining about so the presence of `timestamp` shouldn't matter (although it's still a good idea too remove it—why keep multiple timestamp fields?). I think the problem is related to the fact that since the index is named firepower Logstash's index template that maps `@timestamp` as a ISO8601 timestamp doesn't apply.

---

<div class="post-metadata">

**Author:** ![kprime](https://avatars.discourse-cdn.com/v4/letter/k/76d3ee/32.png) [@kprime](https://discuss.elastic.co/u/kprime)\
**Post date:** [April 22, 2016, 2:12am UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/5 "2016-04-22T02:12:41Z")

</div>

I removed the date timestamp (as above). I then checked the debug to make sure that there where no errors this time. No errors. In fact, everything looked promising. I then checked the indexes in ES and I got a full throated chuckle (or a crazy persons laugh - haven't decided yet -).

kyle@ELK:~$ curl 'localhost:9200/\_cat/indices?v'  
health status index pri rep docs.count docs.deleted store.size pri.store.size  
yellow open .kibana 1 1 1 0 3.1kb 3.1kb  
yellow open firepower 5 1 0 0 795b 795b  
yellow open **logstash-2016.04.21** 5 1 1 0 14.3kb 14.3kb

Nowhere do I have logstash-%{+YYYY.MM.dd}.... It almost seems that by removing the timestamp it no longer selects that index.

UPDATE: restarted the service and tailed the logs instead of 'sudo /opt/logstash/bin/logstash agent --debug --log test.log --config /etc/logstash/conf.d/10-estreamer.conf'

and I get this:

> {"create"=\>{"\_index"=\>"firepower", "\_type"=\>"firepower", "\_id"=\>"AVQ7x\_NbHNuH89\_3Iwf1", "status"=\>400, "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"failed to parse [@timestamp]", "caused\_by"=\>{"type"=\>"illegal\_argument\_exception", "reason"=\>"Invalid format: "2016-04-22T02:23:59.664Z" is malformed at "-04-22T02:23:59.664Z""}}}}, :level=\>:warn}

Now I am even more confused.

---

<div class="post-metadata">

**Author:** ![kprime](https://avatars.discourse-cdn.com/v4/letter/k/76d3ee/32.png) [@kprime](https://discuss.elastic.co/u/kprime)\
**Post date:** [April 25, 2016, 1:31pm UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/6 "2016-04-25T13:31:51Z")

</div>

Fixed. Changed the format in the JSON template.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:00am UTC](https://discuss.elastic.co/t/unable-to-change-index-off-default-logstash/47989/7 "2017-07-06T05:00:43Z")

</div>