Logstash does not create index and doesn't give out any errors. Help

Elastic Stack Logstash
1

Hi everyone,

I'm new to Elasticsearch. I've been playing with Logstash, and I was able to create a couple of indexes using logstash. Now suddenly, it's not creating any indices. Whenever I run a conf file, it says startup completed, but I don't see the index created on elastichq.

Here is my conf file:
input {
file {
path => "/Users/xxxxxx/Downloads/testfile/*.csv"
start_position => "beginning"
type => "TSVImport"
}
}

filter {
csv {
separator => " "
}
mutate {
convert => { "column1" => "float" }
convert => { "column10" => "integer"}
convert => { "column11" => "float"}
convert => { "column12" => "integer"}
convert => { "column2" => "integer"}
convert => { "column9" => "integer"}
}
}

output {
elasticsearch {
action => "index"
index => "shopper_dat"
}
stdout { codec => rubydebug }
}

Here is my logstash -v output.

http://pastebin.com/7unWDJ9x

2

Logstash uses sincedb to track the the current position of a monitored log file, so this could be the cause of your issue.

Check the $HOME directory of the user that is running logstash for files named .sincedb* and if you find them delete them.

Then change your input filter to include the sincedb_path option:

input {
  file {
  path => "/Users/xxxxxx/Downloads/testfile/*.csv"
  start_position => "beginning"
  type => "TSVImport"
  sincedb_path => "/dev/null"
 }
}

Once you have finished testing remove this value logstash will never import the same log entry twice from the same file.

3

Thanks for the response!

I tried that, but it still does not work. Is there some other log file I should post here?

Here's the output
http://pastebin.com/tVirPuqv

Here's the updated conf file:
input {
file {
path => "/Users/manikandan/Downloads/ShopperData/*.csv"
start_position => "beginning"
type => "TSVImport"
sincedb_path => "/dev/null"
}
}

filter {
csv {
separator => " "
}
mutate {
convert => { "column1" => "float" }
convert => { "column10" => "integer"}
convert => { "column11" => "float"}
convert => { "column12" => "integer"}
convert => { "column2" => "integer"}
convert => { "column9" => "integer"}
}
}

output {
elasticsearch {
action => "index"
index => "shopper_data"
}
stdout { codec => rubydebug }
}

4

Is this the only output you get to stdout when starting Logstash like that? Because there should be log messages related to the file input. You could try starting with --debug instead. Then Logstash will e.g. dump the actual configuration that it's using.

5

That was the output of -v flag. However, I just resolved this issue. During one of the runs, I noted that there was an info message saying that the input file was last modified over 86400 seconds ago, so it was being skipped. I touched that file and ran again, and it works now.

Now I have a different question regarding Analyzed strings. One of the fields is store name, with data along the lines of "Store #1", "Store #2", etc. When I try Terms panel on this column, it is taking all "Store" as one value, and "#1", "#2", etc as other values. This skews the terms panel. How do I use the raw data (or the whole string) for the terms panel without having to reindex?

6

Also check your ES logs/

7

Now I have a different question regarding Analyzed strings. One of the fields is store name, with data along the lines of "Store #1", "Store #2", etc. When I try Terms panel on this column, it is taking all "Store" as one value, and "#1", "#2", etc as other values. This skews the terms panel. How do I use the raw data (or the whole string) for the terms panel without having to reindex?

With the default Logstash template you get a .raw subfield for each string field and you can use that field for aggregations. If your mapping doesn't such an unanalyzed field you will have to reindex your data.

8

Thanks for the suggestion. I deleted the and reindexed, but I still don't see the raw fields. Here's the mapping for the index.

{"shopper_data_full":{"mappings":{"TSVImport":{"properties":{"@timestamp":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"@version":{"type":"string"},"column1":{"type":"double"},"column10":{"type":"long"},"column11":{"type":"double"},"column12":{"type":"long"},"column13":{"type":"string"},"column14":{"type":"string"},"column15":{"type":"string"},"column16":{"type":"string"},"column17":{"type":"string"},"column18":{"type":"string"},"column19":{"type":"string"},"column2":{"type":"long"},"column20":{"type":"string"},"column23":{"type":"string"},"column24":{"type":"string"},"column25":{"type":"string"},"column26":{"type":"string"},"column27":{"type":"string"},"column28":{"type":"string"},"column29":{"type":"string"},"column3":{"type":"string"},"column30":{"type":"string"},"column31":{"type":"string"},"column32":{"type":"string"},"column33":{"type":"string"},"column34":{"type":"string"},"column35":{"type":"string"},"column36":{"type":"string"},"column37":{"type":"string"},"column38":{"type":"string"},"column39":{"type":"string"},"column4":{"type":"string"},"column40":{"type":"string"},"column41":{"type":"string"},"column5":{"type":"string"},"column6":{"type":"string"},"column7":{"type":"date","format":"strict_date_optional_time||epoch_millis"},"column8":{"type":"string"},"column9":{"type":"long"},"host":{"type":"string"},"message":{"type":"string"},"path":{"type":"string"},"tags":{"type":"string"},"type":{"type":"string"}}}}}}

9

Did you update the index template or similar so that unanalyzed fields would be created for your string fields? The reindexing doesn't accomplish anything itself.

10

I can see that the default template has provisions to create raw fields:

{"logstash":{"order":0,"template":"logstash-","settings":{"index":{"refresh_interval":"5s"}},"mappings":{"default":{"dynamic_templates":[{"message_field":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string"},"match_mapping_type":"string","match":"message"}},{"string_fields":{"mapping":{"fielddata":{"format":"disabled"},"index":"analyzed","omit_norms":true,"type":"string","fields":{"raw":{"ignore_above":256,"index":"not_analyzed","type":"string","doc_values":true}}},"match_mapping_type":"string","match":""}},{"float_fields":{"mapping":{"type":"float","doc_values":true},"match_mapping_type":"float","match":""}},{"double_fields":{"mapping":{"type":"double","doc_values":true},"match_mapping_type":"double","match":""}},{"byte_fields":{"mapping":{"type":"byte","doc_values":true},"match_mapping_type":"byte","match":""}},{"short_fields":{"mapping":{"type":"short","doc_values":true},"match_mapping_type":"short","match":""}},{"integer_fields":{"mapping":{"type":"integer","doc_values":true},"match_mapping_type":"integer","match":""}},{"long_fields":{"mapping":{"type":"long","doc_values":true},"match_mapping_type":"long","match":""}},{"date_fields":{"mapping":{"type":"date","doc_values":true},"match_mapping_type":"date","match":""}},{"geo_point_fields":{"mapping":{"type":"geo_point","doc_values":true},"match_mapping_type":"geo_point","match":""}}],"_all":{"omit_norms":true,"enabled":true},"properties":{"@timestamp":{"type":"date","doc_values":true},"geoip":{"dynamic":true,"type":"object","properties":{"ip":{"type":"ip","doc_values":true},"latitude":{"type":"float","doc_values":true},"location":{"type":"geo_point","doc_values":true},"longitude":{"type":"float","doc_values":true}}},"@version":{"index":"not_analyzed","type":"string","doc_values":true}}}},"aliases":{}}}

Here is the job picking up the default template:
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["127.0.0.1"], :level=>:info}

Why does it not create raw fields? Is there some setting I'm missing?

11

The Logstash default index template covers indexes whose names match the pattern logstash-*. Your index doesn't.