Not able to create index with the same name

vikas_gopal · June 22, 2015, 9:00am

Hi Experts,

I am using ES 1.6.0 and LS 1.5.0 . I am parsing CSV file and store it to ES. Since I am in Dev environment so I am changing my LS conf file regularly, e.g applying different filters , mutate etc. Now every time I make some changes in the conf file I have to recreate the index with the different name , else LS is not able to create index in ES.It is not just with the index name I have to change the name of the CSV file as well. Got lots of suggestions like (Delete index from API , Delete Documents rather index, Delete index from Directory , Clear index Cache etc ) but still no solution to this problem . Now because of this problem I have multiple CSV files (same data but with different name) and multiple indexes (same no of documents but with different index name). Please suggest How i can rid of with this situation ?

My observation
I was checking ES logs. I observed if I use same CSV file name or Index name , it adds a client node into the cluster which has blank documents.I am using all the default settings for ES and LS i.e no change in the yml files .

ES log

[2015-06-22 14:04:53,943][INFO ][node ] [Cloak] started
[2015-06-22 14:04:54,952][INFO ][cluster.service ] [Cloak] added {[logstash-LP-54EE752450D4-9468-9796][3jjhY28IQUGEvoc-qUjhrQ][LP-54EE752450D4][inet[/10.92.16.118:9302]]{data=false, client=true},}, reason: zen-disco-receive(join from node[[logstash-LP-54EE752450D4-9468-9796][3jjhY28IQUGEvoc-qUjhrQ][LP-54EE752450D4][inet[/10.92.16.118:9302]]{data=false, client=true}])

warkolm · June 23, 2015, 8:49am

The problem with the file is likely to be a sincedb one.

However LS should just keep indexing to the existing index in LS, that shouldn't be a problem. Why do you think you have to delete the index, are you getting errors?

vikas_gopal · June 23, 2015, 9:03am

No ! I am not getting any error . Let me explain my Senior--lets say I want to change the data type for one filed e.g Employee_salary from string to integer , for this I need to use Mutate filter . Now to apply this I have to stop ES , LS and edit conf file . Here is the problem ,If I am using same csv file and same index name LS is not able to create/store index in ES so to get it done I have to change the index name along with csv file name . My LS conf file is like

input {
file {
path => "E:/ELK_Traning/logstash-1.5.0/bin/class1.csv"
start_position => "beginning"
}
}
filter {
csv {
columns => ["Name","Roll No","Class","Marks"]
separator => ","

}
mutate {
convert => [ "Roll No", "integer" ]
convert => [ "Class", "integer" ]
convert => [ "Marks", "integer" ]
}
}
output {
elasticsearch {
action => "index"
host => "localhost"
index => "test1"
}
}

So every time I make some changes in the above conf file I have to create new Index (test1,test2, and so on ) along with csv file name (class1, class2 .....and so on ). My question why i cannot perform or update changes on the same index by using same CSV file . Earlier I was thinking because Index has already been created , so I cannot update it using LS . That is why I tried to delete it .

warkolm · June 23, 2015, 9:09am

You don't need to change the filename, that's a sincedb problem as I mentioned.

As for the ES "problem", then yes, if you change the mapping like that then deleting the index and then reindexing is the best option as mixing different field mappings is a bad thing.

vikas_gopal · June 23, 2015, 12:47pm

Since it is a sincedb issue , so can we resolve this ? I changed "start_position =>"end" and keep the same file name , still no luck . I changed the index name but used same file name as mentioned "class1". LS is not bale to create index in ES. It seems LS reads the file , parse it but not able to send it ES. In KOPF plugin I can see 2nd node has been added but new index has not been created .

ES log

[2015-06-23 18:11:35,349][INFO ][cluster.service ] [Vikas] added {[logstash-LP-54EE752450D4-9084-9788][6zqfa_wXTaSMdB6bcx-ZaQ][LP-54EE752450D4][inet[/192.168.1.2:9301]]{data=false, client=true},}, reason: zen-disco-receive(join from node[[logstash-LP-54EE752450D4-9084-9788][6zqfa_wXTaSMdB6bcx-ZaQ][LP-54EE752450D4][inet[/192.168.1.2:9301]]{data=false, client=true}])

vikas_gopal · June 23, 2015, 1:00pm

OK , I get it , Since I was mentioning "start_position =>"end" so every time it reads from the beginning so after inserting some new rows to the csv file LS immediately creates an index .

WOW !! thanks Mark you are awesome.

Cheers

lkekeng · September 19, 2015, 9:46am

iHi,
Has been struggling for many days and seem couldn't figure out what happen.
I am running ElasticSearch 1.7.1 with Logstash 1.5.4.

my logstash conf file named "logstash-apache" is stored in the path c:\users\student\logstash-1.5.4\logstash-1.5.4\conf\logstash-apache

my log file is stored in the path c:\users\student\logstash-1.5.4\logstash-1.5.4\log\access_log

I followed the instruction from this link" http://logstash.net/docs/1.4.1/tutorials/getting-started-with-logstash" and was able to index to ES with the test files. I was able to see the output in Logstash window.

However, after I deleted the index using curl -XDELETE and try to run the same command, the index was not created in ES. Below are the last few lines of logstash output:

INFO: [logstash-myPC] detected_master [myNode1][dIlKd954RWypKyKh-
20H5A][KL-PC][inet[/192.168.1.4:9300]], added {[myNode1][dIlKd954RWypKyKh-20H
5A][myPC][inet[/192.168.1.4:9300]],}, reason: zen-disco-receive(from master [[myNode1][dIlKd954RWypKyKh-20H5A][myPC][inet[/192.168.1.4:9300]]])
Sep 19, 2015 5:19:42 AM org.elasticsearch.node.internal.InternalNode start
INFO: [logstash-myPC] started
Logstash startup completed

This is my conf file:
input {
file {
path => ["C:\Users\student\logstash-1.5.4\logstash-1.5.4\log\access_log1*"]
start_position => "beginning"
}
}

filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}

}

output {
elasticsearch { host => "localhost"}
stdout { codec =,> rubydebug }
}

I have tried the following tricks but still can't seem to get it work again:

Changed the path for the input file to use "/" or "" slash or enclosed it with square bracket.
Change the start_position to "beginning" and "end"
Add a empty line in the log file
Tried to change the index name using action => "index" and index => "test-apache".

Hope some one is able to help. Thank you in advance.

magnusbaeck · September 19, 2015, 4:59pm

@lkekeng – please start a new topic for your question.

vikas_gopal · September 24, 2015, 3:27pm

I would suggest 2 solutions...

Solution 1

Rename index name and data file name which you are parsing . I know this is not a good solution but I guess you will understand it batter if you know more about sincedb and LS file filter.

First let me make you understand how file plugin works . When you mention start position to beginning LS will start parsing data file from 1st row till end , so at this point LS set a pointer(sincedb) at the last row of the file . Now the next time you parse the same file without any new data to the log file LS will not create index because file already been read by LS and pointer(sincedb) is set to last row.If you add new row at the end of the log file you will see that will be parsed and so on .So simply deleting index won't work here . You need to either set the pointer(sincedb) to the beginning or use new index name along with data file

Solution 2:- Use sincedb path in file filter something like

input {
file {
type => "cust1"
path => "E:/CEF/overall.log"
start_position => "beginning"
sincedb_path => "E:/db/cust1_comma"

}

Now you just need to delete the cust1_comma from sincedb_path which will reset pointer for you .Now do whatever you want , delete index , recreate it blah blah ....

let me know if you need more clarity on this .

PKA · April 20, 2016, 10:18am

I did both the options you suggested, but none worked
Herez the conf file:

input {
  file {
    path => "/etc/logstash/mycsvdata.txt"
    type => "core2"
    start_position => "beginning"
    sincedb_path => "/home/ec2-user/cust1_comma"
  }
}
filter {
  csv {
      separator => ","
      columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
  }
  mutate {convert => ["High", "float"]}
  mutate {convert => ["Open", "float"]}
  mutate {convert => ["Low", "float"]}
  mutate {convert => ["Close", "float"]}
  mutate {convert => ["Volume", "float"]}
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost:9200"
        index => "logstash-%{+YYYY.MM.dd}"
	workers => 1
    }
	stdout {}
}

The sincedb file, cust1_comma, was not even created this time

PKA · April 20, 2016, 11:53am

I added a few line breaks at the end of my csv file and things got started. The index is getting created now