I have a system which picks feed , does XYZ data transformations and then
drops to ES.
For each feed there is a feed ID associated. And when the feed reaches ES ,
we will receive a related docID.
The feed ID and the doc ID are mapped somewhere to trace the path feed was
picked , transformed and pushed to storage.
I was wondering if there is some mechanism where a doc ID is generated at a
point long before it reaches ES and i just have to maintain a single ID for
the entire feed.
So basically i should be able to generate a doc ID much before i push the
feed to elastic Search and i should be able to guarantee that when the feed
reaches the ES , that ID is free (as in no document is present with that
ID).
I have a system which picks feed , does XYZ data transformations and then
drops to ES.
For each feed there is a feed ID associated. And when the feed reaches ES
, we will receive a related docID.
The feed ID and the doc ID are mapped somewhere to trace the path feed was
picked , transformed and pushed to storage.
I was wondering if there is some mechanism where a doc ID is generated at
a point long before it reaches ES and i just have to maintain a single ID
for the entire feed.
So basically i should be able to generate a doc ID much before i push
the feed to elastic Search and i should be able to guarantee that when the
feed reaches the ES , that ID is free (as in no document is present with
that ID).
I have a system which picks feed , does XYZ data transformations and
then drops to ES.
For each feed there is a feed ID associated. And when the feed reaches ES
, we will receive a related docID.
The feed ID and the doc ID are mapped somewhere to trace the path feed
was picked , transformed and pushed to storage.
I was wondering if there is some mechanism where a doc ID is generated at
a point long before it reaches ES and i just have to maintain a single ID
for the entire feed.
So basically i should be able to generate a doc ID much before i push
the feed to elastic Search and i should be able to guarantee that when the
feed reaches the ES , that ID is free (as in no document is present with
that ID).
I have a system which picks feed , does XYZ data
transformations and then drops to ES.
For each feed there is a feed ID associated. And when the feed
reaches ES , we will receive a related docID.
The feed ID and the doc ID are mapped somewhere to trace the
path feed was picked , transformed and pushed to storage.
I was wondering if there is some mechanism where a doc ID is
generated at a point long before it reaches ES and i just have
to maintain a single ID for the entire feed.
So basically i should be able to generate a doc ID much
before i push the feed to elastic Search and i should be able
to guarantee that when the feed reaches the ES , that ID is
free (as in no document is present with that ID).
Is this possible with ES ?
Thanks
Vineeth
I have a system which picks feed , does XYZ data
transformations and then drops to ES.
For each feed there is a feed ID associated. And when the feed
reaches ES , we will receive a related docID.
The feed ID and the doc ID are mapped somewhere to trace the
path feed was picked , transformed and pushed to storage.
I was wondering if there is some mechanism where a doc ID is
generated at a point long before it reaches ES and i just have
to maintain a single ID for the entire feed.
So basically i should be able to generate a doc ID much
before i push the feed to elastic Search and i should be able
to guarantee that when the feed reaches the ES , that ID is
free (as in no document is present with that ID).
Is this possible with ES ?
Thanks
Vineeth
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.