I have a rails application and use Elasticsearch as a search engine in my rails app. this app collects data from the mobile application and could collect from any kind of mobile app. mobile app sends two types of data user profile details and user actions details. my app admins could search over this data with multiple conditions and operations and fetch the specific results and which are user profile details. after that my app admins could communicate with this profile, for example, send an email, SMS, or even chat online. In my case I have two options to save user data; first of all, I want to save user profiles details and user action details in a separate document with this structure profile doc:
POST profilee-2022-06-09/_doc
{
"profile": {
"app_id": "abbccddeeff",
"profile_id": "2faae1d6-5875-4b36-b119-74a14589c841",
"whatsapp_number": "whatsapp:+61478421940",
"phone": "+61478421940",
"email": "user@mail.com",
"first_name": "john",
"last_name": "doe"
}
}
user actions details:
POST events_app_id_2022-05-17/_doc
{
"app_id": "9vlgwrr6rg",
"event": "Email_Sign_Up",
"profile_id": "2faae1d6-5875-4b36-b119-74a14589c840",
"media": "x1z1",
"date_time": "2022-05-17T11:48:02.511Z",
"device_id": "2faae1d6-5875-4b36-b119-74a14589c840",
"lib": "android",
"lib_version": "1.0.0",
"os": "Android",
"os_version": "12",
"manufacturer": "Google",
"brand": "google",
"model": "sdk_gphone64_arm64",
"google_play_services": "available",
"screen_dpi": 440,
"screen_height": 2296,
"screen_width": 1080,
"app_version_string": "1.0",
"app_build_number": 1,
"has_nfc": false,
"has_telephone": true,
"carrier": "T-Mobile",
"wifi": true,
"bluetooth_version": "ble",
"session_id": "b1ad31ab-d440-435f-ac12-3d03c30ac44f",
"insert_id": "1e285b51-abcf-46ae-8359-9a9d58970cdf"
}
As I said before app admins search over this document to fetch specific profiles and use that result to communicate with them, in this case, the problem is the mobile user could create a profile and a few days or a few months later create some actions so user profile details and user action details are generated in different days so if app admins want to fetch specific result from this data and wrote some complex query I have at least two queries by application on my Elasticsearch in my app it's impossible because each query must save for later use by admin, so As a result of business logic it's impossible to me, and I have to add in some case I need to implement join query that based on Elasticsearch documentation It has cost so it's impossible In the second scenario I decided to save both user profile and action in one docs somethings like this:
POST profilee-2022-06-09/_doc
{
"profile": {
"app_id": "abbccddeeff",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"whatsapp_number": "whatsapp:+61478421940",
"phone": "+61478421940",
"email": "user@mail.com",
"first_name": "john",
"last_name": "doe",
"events": [
{
"app_id": "abbccddeeff",
"event": "sign_in",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"media": "x1z1",
"date_time": "2022-06-06T11:52:02.511Z"
},
{
"app_id": "abbccddeeff",
"event": "course_begin",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"media": "x1z1",
"date_time": "2022-06-06T11:56:02.511Z"
},
{
"app_id": "abbccddeeff",
"event": "payment",
"profile_id": "urm-2faae1d6-5875-4b36-b119-74a14589c841",
"media": "x1z1",
"date_time": "2022-06-06T11:58:02.511Z"
}
]
}
}
In this case, In the same state, I have to do as same as I do in before and I have to generate a profile index per day and append user action to it, so It means I have to update continuously each day, assume I have 100,000 profile and each one have 50 actions it means 100,000 * 50 per day update that have severity on my server so still it's impossible. So Could you please help me what is the best model to save my data in Elasticsearch based on my descriptions?
Update: Does Elasticsearch useful for my requirements? If I switch to other databases like MongoDB or add Hadoop it be more useful in my case?