Getting error when TTL is executed

Arun_Prasath · March 27, 2014, 4:19pm

Getting error when i applied TTL while creating Index. Able to search data.
I set TTL for 4m , after 4m still i see data available. How to make TTL
work and delete records after specified time.

I'm using elasticsearch 1.09 (latest).

Any help would be much appreciated.

Index Creation

"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url" : "jdbc:sqlserver://localhost:1433;databaseName=cdt",
"user" : "CDT_Admin",
"password" : "cdtpassword",
"index": "local-test",
"type": "logs-test",
"_ttl": "4m",
"sql": "select [AccountContextId] AS
_id,[NumberOfPhoneLines],[AccountNumber],TransactionTime from
AccountInformation.CustomerInformation"

Error -

failed to execute ttl purge
java.lang.ClassCastException:
org.elasticsearch.index.mapper.core.StringFieldMap
per cannot be cast to org.elasticsearch.index.mapper.internal.TTLFieldMapper
at
org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.getShard
sToPurge(IndicesTTLService.java:162)
at
org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run(Indi
cesTTLService.java:121)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6f9e9a4d-9f8f-4afc-ae63-0890b84d938d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 27, 2014, 5:43pm

Can you please show the mapping of your local-test index?

curl localhost:9200/local-test/_mapping?pretty

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f556e404-ac04-45b9-8f63-6862c525ad6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arun_Prasath · March 27, 2014, 6:23pm

{
"local-test" : {
"mappings" : {
"logs-test" : {
"properties" : {
"AccountNumber" : {
"type" : "string"
},
"NumberOfPhoneLines" : {
"type" : "long"
},
"TransactionTime" : {
"type" : "date",
"format" : "dateOptionalTime"
}
}
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09116865-acae-4cb4-96eb-b1e73f6547d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 27, 2014, 6:43pm

Hmmm not sure, do you have any other indexes, for example, can you show the
result of this?

curl localhost:9200/_mapping?pretty

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/af44b3be-47f9-465e-9470-83fb4a6a0027%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arun_Prasath · March 27, 2014, 6:47pm

Binh,

this is what i get... Is this a bug need to be fixed or am i doing
something wrong. As i posted initially is that right way to set ttl.
Please confirm.

{
"local-test" : {
"mappings" : {
"logs-test" : {
"properties" : {
"AccountNumber" : {
"type" : "string"
},
"NumberOfPhoneLines" : {
"type" : "long"
},
"TransactionTime" : {
"type" : "date",
"format" : "dateOptionalTime"
}
}
}
}
},
"_river" : {
"mappings" : {
"my_jdbc_river" : {
"properties" : {
"active" : {
"type" : "boolean"
},
"counter" : {
"type" : "long"
},
"custom" : {
"properties" : {
"autocommit" : {
"type" : "boolean"
},
"columnCreatedAt" : {
"type" : "string"
},
"columnEscape" : {
"type" : "boolean"
},
"columnUpdatedAt" : {
"type" : "string"
},
"fetchsize" : {
"type" : "long"
},
"interval" : {
"type" : "string"
},
"locale" : {
"type" : "string"
},
"maxretrywait" : {
"type" : "string"
},
"maxrows" : {
"type" : "long"
},
"retries" : {
"type" : "long"
},
"riverName" : {
"type" : "string"
},
"settings" : {
"properties" : {
"jdbc" : {
"properties" : {
"_ttl" : {
"type" : "string"
},
"driver" : {
"type" : "string"
},
"index" : {
"type" : "string"
},
"password" : {
"type" : "string"
},
"sql" : {
"type" : "string"
},
"strategy" : {
"type" : "string"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
},
"user" : {
"type" : "string"
}
}
},
"type" : {
"type" : "string"
}
}
},
"sql" : {
"type" : "string"
}
}
},
"jdbc" : {
"properties" : {
"_ttl" : {
"type" : "string"
},
"driver" : {
"type" : "string"
},
"index" : {
"type" : "string"
},
"password" : {
"type" : "string"
},
"sql" : {
"type" : "string"
},
"strategy" : {
"type" : "string"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
},
"user" : {
"type" : "string"
}
}
},
"name" : {
"type" : "string"
},
"node" : {
"properties" : {
"id" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"transport_address" : {
"type" : "string"
}
}
},
"started" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"type" : {
"type" : "string"
}
}
}
}
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aadcf888-7e27-4c26-a326-4b5fcef8ae30%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 27, 2014, 7:00pm

Arun, I think I was just confused. ES has a TTL capability using the _ttl
field (which is why I was asking for your mapping). It "seems" like you are
using a river that has may have its own "TTL" capabilities, outside of ES.
The only thing I can tell you that is causing this error is that your river
has field named _ttl of type string which conflicts with the built-in ES
_ttl field. If you delete your index and rename that _ttl field in your
river to something else, the error will likely go away. But I still don't
know the answer to your original Q which is how do you enable TTL using
this river. Unfortunately I'm not very familiar with this river to know the
answer.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/747ebe72-453b-4254-8f76-269d4b8fcd89%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 27, 2014, 7:03pm

Sorry, I meant to say delete the _river index and rename the _ttl field
when recreating it again.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ebccff7a-fc50-43cd-a9f3-5e0e0fbf5b52%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arun_Prasath · March 27, 2014, 7:14pm

Binh,

Thanks for the update.

I will delete and recreate the index. Still I want to know how to make TTL
working in ES. I need to delete records after certain period.

I guess I'm using river to pull data from SQL Server though JDBC driver and
push it to ES.

I'm unclear how to enable and make TTL working in ES.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/87528295-98fb-4dd6-8585-b29d0ae0a7af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 27, 2014, 7:23pm

Enabling TTL in ES is actually very easy. Here's a mapping to enable _ttl:

curl -XPUT 'localhost:9200/ttltest'
{
"mappings": {
"doc": {
"_ttl" : {"enabled": true},
"properties": {
"foo": {
"type": "string"
}
}
}
}
}

And then when you index a document, just specify a ttl value:

curl -XPUT 'http://localhost:9200/ttltest/doc/1?ttl=1d' -d '{ }'

And the above do will automatically get deleted after 1 day.

The part I do not know is if that river you are using has ttl capabilities,
and if yes, how do you configure it from the river?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/877f7915-31a8-4e31-84e6-06d8b2e0d383%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arun_Prasath · March 27, 2014, 7:52pm

Binh,

Where to execute the command you have given me. I don't know how to
execute. Is there any tool to run curl ? I'm using sense plug in.

I'm trying to do like below in sense - How to make sure its enabled after
running this command.

local-test - Index.
*logs-test - Type. *

PUT /local-test/logs-test/_mapping
{
"mappings": {
"logs-test": {
"_ttl" : {"enabled": true}
}
}
}

On Thursday, March 27, 2014 3:23:15 PM UTC-4, Binh Ly wrote:

Enabling TTL in ES is actually very easy. Here's a mapping to enable _ttl:

curl -XPUT 'localhost:9200/ttltest'
{
"mappings": {
"doc": {
"_ttl" : {"enabled": true},
"properties": {
"foo": {
"type": "string"
}
}
}
}
}

And then when you index a document, just specify a ttl value:

curl -XPUT 'http://localhost:9200/ttltest/doc/1?ttl=1d' -d '{ }'

And the above do will automatically get deleted after 1 day.

The part I do not know is if that river you are using has ttl
capabilities, and if yes, how do you configure it from the river?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/78b4b2a5-ea9b-4c55-9798-d2c8655dd0cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

brian_yoder · March 27, 2014, 8:28pm

Arun,

*Getting error when i applied TTL while creating Index. Able to search

data. I set TTL for 4m , after 4m still i see data available. How to make
TTL work and delete records after specified time. *

TTL processing is not something that can be depended upon to delete a
document exactly after a specified amount of time; instead it is kind of
lazy to keep performance as high as possible. By default, it handles TTL
cleanup once per minute (if I remember correctly).

What I've done when using TTL is to get the TTL value from each document
that is returned in a response. If the TTL value is negative I throw away
the document as if it never came back. A negative value only means that the
document has expired but ES hasn't yet run its cleanup to actually delete
it.

Many people use the HTTP REST interface directly; I prefer writing Java.
This makes a lot of the work with ES much easier, as I can more tightly
integrate my follow-on wrapping with ES and make ES sparkle and shine even
more. I also can incorporate my business logic in my Java layer and present
a more directed interface to clients. This, by the way, is common even in
the commercial relational database (RDB) world; our RDB work wraps custom
DAOs around the RDB to handle our business logic; we never permit direct
SQL to the database.

I hope this helps.

I'm using elasticsearch 1.09 (latest).

Actually, Elasticsearch 1.1.0 is now the latest.

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6270ca69-8f4f-4e6e-8f6e-15492c6255dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Arun_Prasath · March 27, 2014, 9:00pm

Brian,

Thanks for clarification.

By default, it handles TTL cleanup once per minute (if I remember
correctly).

If i'm not mistaken, Do you mean the record will be deleted & I should not
see the record when i search again after one minute of TTL.

If the TTL value is negative I throw away the document as if it never came
back. A negative value only means that the document has expired but ES
hasn't yet run its cleanup to actually delete it.

I don't see a TTL value returned in the response. so the TTL is not set ?

I set TTL as below - Not sure its correct. I'm using river JDBC.

PUT /_river/my_jdbc_river/_meta
{
"type" : "jdbc",
"jdbc" : {
"strategy" : "oneshot",
"driver" : "com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url" : "jdbc:sqlserver://localhost:1433;databaseName=cdt",
"user" : "CDT_Admin",
"password" : "cdtpassword",
"index": "local-test",
* "ttl" : { "enabled" : true, "default" : "3m" },*
"type": "logs-test",
"sql": "select [AccountContextId] AS
_id,[NumberOfPhoneLines],[AccountNumber],TransactionTime from
AccountInformation.CustomerInformation"
}
}

Arun

On Thursday, March 27, 2014 4:28:37 PM UTC-4, InquiringMind wrote:

Arun,

*Getting error when i applied TTL while creating Index. Able to search

data. I set TTL for 4m , after 4m still i see data available. How to make
TTL work and delete records after specified time. *

TTL processing is not something that can be depended upon to delete a
document exactly after a specified amount of time; instead it is kind
of lazy to keep performance as high as possible. By default, it handles TTL
cleanup once per minute (if I remember correctly).

What I've done when using TTL is to get the TTL value from each document
that is returned in a response. If the TTL value is negative I throw away
the document as if it never came back. A negative value only means that the
document has expired but ES hasn't yet run its cleanup to actually delete
it.

Many people use the HTTP REST interface directly; I prefer writing Java.
This makes a lot of the work with ES much easier, as I can more tightly
integrate my follow-on wrapping with ES and make ES sparkle and shine even
more. I also can incorporate my business logic in my Java layer and present
a more directed interface to clients. This, by the way, is common even in
the commercial relational database (RDB) world; our RDB work wraps custom
DAOs around the RDB to handle our business logic; we never permit direct
SQL to the database.

I hope this helps.

I'm using elasticsearch 1.09 (latest).

Actually, Elasticsearch 1.1.0 is now the latest.

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4a236967-67f9-4a50-bda7-4603123d7317%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

brian_yoder · March 28, 2014, 3:01pm

Thanks for clarification.

By default, it handles TTL cleanup once per minute (if I remember
correctly).

*If i'm not mistaken, Do you mean the record will be deleted & I should
not see the record when i search again after one minute of TTL. *

After about 1 minute after the document has expired is when you should
expect to see it deleted. This would mean (assuming default configurations
and my memory is accurate) that if you configure 3m as the default TTL,
then a document that is added with no TTL value will be expired after 3m
and deleted after 4.

If the TTL value is negative I throw away the document as if it never
came back. A negative value only means that the document has expired but ES
hasn't yet run its cleanup to actually delete it.

I don't see a TTL value returned in the response. so the TTL is not set ?

I believe that you need to ask for the _ttl field to come back in the
response. And in that case, you must also ask for _source since any
explicit fields mean that _source won't be returned by default.

I set TTL as below - Not sure its correct. I'm using river JDBC.

I'm not sure. I push bulk updates though a BulkRequestBuilder. I don't like
rivers they rarely all match up with the same version of ES that I am
using; but my own Java code always matches up with the ES version I am
using!

Regards,
Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f9b15d06-73a2-447b-a8f9-1a2d67610b40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.