# Logstash does not create index and doesn't give out any errors. Help

**URL:** <https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630>\
**Category:** Logstash\
**Created:** [March 7, 2016, 9:21am UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630 "2016-03-07T09:21:49Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![MRamakri](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@MRamakri](https://discuss.elastic.co/u/MRamakri)\
**Post date:** [March 7, 2016, 9:21am UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/1 "2016-03-07T09:21:49Z")

</div>

Hi everyone,

I'm new to Elasticsearch. I've been playing with Logstash, and I was able to create a couple of indexes using logstash. Now suddenly, it's not creating any indices. Whenever I run a conf file, it says startup completed, but I don't see the index created on elastichq.

Here is my conf file:  
input {  
file {  
path =\> "/Users/xxxxxx/Downloads/testfile/\*.csv"  
start\_position =\> "beginning"  
type =\> "TSVImport"  
}  
}

filter {  
csv {  
separator =\> " "  
}  
mutate {  
convert =\> { "column1" =\> "float" }  
convert =\> { "column10" =\> "integer"}  
convert =\> { "column11" =\> "float"}  
convert =\> { "column12" =\> "integer"}  
convert =\> { "column2" =\> "integer"}  
convert =\> { "column9" =\> "integer"}  
}  
}

output {  
elasticsearch {  
action =\> "index"  
index =\> "shopper\_dat"  
}  
stdout { codec =\> rubydebug }  
}

Here is my logstash -v output.

[http://pastebin.com/7unWDJ9x](http://pastebin.com/7unWDJ9x)

---

<div class="post-metadata">

**Author:** ![mick66](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@mick66](https://discuss.elastic.co/u/mick66)\
**Post date:** [March 7, 2016, 2:14pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/2 "2016-03-07T14:14:06Z")

</div>

Logstash uses sincedb to track the the current position of a monitored log file, so this could be the cause of your issue.

Check the $HOME directory of the user that is running logstash for files named .sincedb\* and if you find them delete them.

Then change your input filter to include the sincedb\_path option:

```
input {
  file {
  path => "/Users/xxxxxx/Downloads/testfile/*.csv"
  start_position => "beginning"
  type => "TSVImport"
  sincedb_path => "/dev/null"
 }
}

```

Once you have finished testing remove this value logstash will never import the same log entry twice from the same file.

---

<div class="post-metadata">

**Author:** ![MRamakri](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@MRamakri](https://discuss.elastic.co/u/MRamakri)\
**Post date:** [March 7, 2016, 3:13pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/3 "2016-03-07T15:13:05Z")

</div>

Thanks for the response!

I tried that, but it still does not work. Is there some other log file I should post here?

Here's the output  
[http://pastebin.com/tVirPuqv](http://pastebin.com/tVirPuqv)

Here's the updated conf file:  
input {  
file {  
path =\> "/Users/manikandan/Downloads/ShopperData/\*.csv"  
start\_position =\> "beginning"  
type =\> "TSVImport"  
sincedb\_path =\> "/dev/null"  
}  
}

filter {  
csv {  
separator =\> " "  
}  
mutate {  
convert =\> { "column1" =\> "float" }  
convert =\> { "column10" =\> "integer"}  
convert =\> { "column11" =\> "float"}  
convert =\> { "column12" =\> "integer"}  
convert =\> { "column2" =\> "integer"}  
convert =\> { "column9" =\> "integer"}  
}  
}

output {  
elasticsearch {  
action =\> "index"  
index =\> "shopper\_data"  
}  
stdout { codec =\> rubydebug }  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 7, 2016, 6:26pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/4 "2016-03-07T18:26:52Z")

</div>

Is this the only output you get to stdout when starting Logstash like that? Because there should be log messages related to the file input. You could try starting with `--debug` instead. Then Logstash will e.g. dump the actual configuration that it's using.

---

<div class="post-metadata">

**Author:** ![MRamakri](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@MRamakri](https://discuss.elastic.co/u/MRamakri)\
**Post date:** [March 7, 2016, 7:49pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/5 "2016-03-07T19:49:07Z")

</div>

That was the output of -v flag. However, I just resolved this issue. During one of the runs, I noted that there was an info message saying that the input file was last modified over 86400 seconds ago, so it was being skipped. I touched that file and ran again, and it works now.

Now I have a different question regarding Analyzed strings. One of the fields is store name, with data along the lines of "Store #1", "Store #2", etc. When I try Terms panel on this column, it is taking all "Store" as one value, and "#1", "#2", etc as other values. This skews the terms panel. How do I use the raw data (or the whole string) for the terms panel without having to reindex?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 8, 2016, 3:47am UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/6 "2016-03-08T03:47:17Z")

</div>

Also check your ES logs/

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2016, 6:39am UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/7 "2016-03-08T06:39:44Z")

</div>

> Now I have a different question regarding Analyzed strings. One of the fields is store name, with data along the lines of "Store #1", "Store #2", etc. When I try Terms panel on this column, it is taking all "Store" as one value, and "#1", "#2", etc as other values. This skews the terms panel. How do I use the raw data (or the whole string) for the terms panel without having to reindex?

With the default Logstash template you get a .raw subfield for each string field and you can use that field for aggregations. If your mapping doesn't such an unanalyzed field you will have to reindex your data.

---

<div class="post-metadata">

**Author:** ![MRamakri](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@MRamakri](https://discuss.elastic.co/u/MRamakri)\
**Post date:** [March 8, 2016, 3:02pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/8 "2016-03-08T15:02:15Z")

</div>

Thanks for the suggestion. I deleted the and reindexed, but I still don't see the raw fields. Here's the mapping for the index.

{"shopper\_data\_full":{"mappings":{"TSVImport":{"properties":{"@timestamp":{"type":"date","format":"strict\_date\_optional\_time||epoch\_millis"},"@version":{"type":"string"},"column1":{"type":"double"},"column10":{"type":"long"},"column11":{"type":"double"},"column12":{"type":"long"},"column13":{"type":"string"},"column14":{"type":"string"},"column15":{"type":"string"},"column16":{"type":"string"},"column17":{"type":"string"},"column18":{"type":"string"},"column19":{"type":"string"},"column2":{"type":"long"},"column20":{"type":"string"},"column23":{"type":"string"},"column24":{"type":"string"},"column25":{"type":"string"},"column26":{"type":"string"},"column27":{"type":"string"},"column28":{"type":"string"},"column29":{"type":"string"},"column3":{"type":"string"},"column30":{"type":"string"},"column31":{"type":"string"},"column32":{"type":"string"},"column33":{"type":"string"},"column34":{"type":"string"},"column35":{"type":"string"},"column36":{"type":"string"},"column37":{"type":"string"},"column38":{"type":"string"},"column39":{"type":"string"},"column4":{"type":"string"},"column40":{"type":"string"},"column41":{"type":"string"},"column5":{"type":"string"},"column6":{"type":"string"},"column7":{"type":"date","format":"strict\_date\_optional\_time||epoch\_millis"},"column8":{"type":"string"},"column9":{"type":"long"},"host":{"type":"string"},"message":{"type":"string"},"path":{"type":"string"},"tags":{"type":"string"},"type":{"type":"string"}}}}}}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2016, 3:37pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/9 "2016-03-08T15:37:43Z")

</div>

Did you update the index template or similar so that unanalyzed fields would be created for your string fields? The reindexing doesn't accomplish anything itself.

---

<div class="post-metadata">

**Author:** ![MRamakri](https://avatars.discourse-cdn.com/v4/letter/m/bcef8e/32.png) [@MRamakri](https://discuss.elastic.co/u/MRamakri)\
**Post date:** [March 8, 2016, 8:09pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/10 "2016-03-08T20:09:05Z")

</div>

I can see that the default template has provisions to create raw fields:

{"logstash":{"order":0,"template":"logstash-_","settings":{"index":{"refresh\_interval":"5s"}},"mappings":{"default":{"dynamic\_templates":[{"message\_field":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit\_norms":true,"type":"string"},"match\_mapping\_type":"string","match":"message"}},{"string\_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit\_norms":true,"type":"string","fields":{"raw":{"ignore\_above":256,"index":"not\_analyzed","type":"string","doc\_values":true}}},"match\_mapping\_type":"string","match":"_"}},{"float\_fields":{"mapping":{"type":"float","doc\_values":true},"match\_mapping\_type":"float","match":"_"}},{"double\_fields":{"mapping":{"type":"double","doc\_values":true},"match\_mapping\_type":"double","match":"_"}},{"byte\_fields":{"mapping":{"type":"byte","doc\_values":true},"match\_mapping\_type":"byte","match":"_"}},{"short\_fields":{"mapping":{"type":"short","doc\_values":true},"match\_mapping\_type":"short","match":"_"}},{"integer\_fields":{"mapping":{"type":"integer","doc\_values":true},"match\_mapping\_type":"integer","match":"_"}},{"long\_fields":{"mapping":{"type":"long","doc\_values":true},"match\_mapping\_type":"long","match":"_"}},{"date\_fields":{"mapping":{"type":"date","doc\_values":true},"match\_mapping\_type":"date","match":"_"}},{"geo\_point\_fields":{"mapping":{"type":"geo\_point","doc\_values":true},"match\_mapping\_type":"geo\_point","match":"_"}}],"\_all":{"omit\_norms":true,"enabled":true},"properties":{"@timestamp":{"type":"date","doc\_values":true},"geoip":{"dynamic":true,"type":"object","properties":{"ip":{"type":"ip","doc\_values":true},"latitude":{"type":"float","doc\_values":true},"location":{"type":"geo\_point","doc\_values":true},"longitude":{"type":"float","doc\_values":true}}},"@version":{"index":"not\_analyzed","type":"string","doc\_values":true}}}},"aliases":{}}}

Here is the job picking up the default template:  
Using mapping template from {:path=\>nil, :level=\>:info}  
Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"\_all"=\>{"enabled"=\>true, "omit\_norms"=\>true}, "dynamic\_templates"=\>[{"message\_field"=\>{"match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"string", "index"=\>"analyzed", "omit\_norms"=\>true, "fielddata"=\>{"format"=\>"disabled"}, "fields"=\>{"raw"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true, "ignore\_above"=\>256}}}}}, {"float\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"float", "mapping"=\>{"type"=\>"float", "doc\_values"=\>true}}}, {"double\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"double", "mapping"=\>{"type"=\>"double", "doc\_values"=\>true}}}, {"byte\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"byte", "mapping"=\>{"type"=\>"byte", "doc\_values"=\>true}}}, {"short\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"short", "mapping"=\>{"type"=\>"short", "doc\_values"=\>true}}}, {"integer\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"integer", "mapping"=\>{"type"=\>"integer", "doc\_values"=\>true}}}, {"long\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"long", "mapping"=\>{"type"=\>"long", "doc\_values"=\>true}}}, {"date\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"date", "mapping"=\>{"type"=\>"date", "doc\_values"=\>true}}}, {"geo\_point\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"geo\_point", "mapping"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date", "doc\_values"=\>true}, "@version"=\>{"type"=\>"string", "index"=\>"not\_analyzed", "doc\_values"=\>true}, "geoip"=\>{"type"=\>"object", "dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip", "doc\_values"=\>true}, "location"=\>{"type"=\>"geo\_point", "doc\_values"=\>true}, "latitude"=\>{"type"=\>"float", "doc\_values"=\>true}, "longitude"=\>{"type"=\>"float", "doc\_values"=\>true}}}}}}}, :level=\>:info}  
New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["127.0.0.1"], :level=\>:info}

Why does it not create raw fields? Is there some setting I'm missing?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 8, 2016, 8:41pm UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/11 "2016-03-08T20:41:17Z")

</div>

The Logstash default index template covers indexes whose names match the pattern logstash-\*. Your index doesn't.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:07am UTC](https://discuss.elastic.co/t/logstash-does-not-create-index-and-doesnt-give-out-any-errors-help/43630/12 "2017-07-06T05:07:57Z")

</div>