# Issue With Parsing Unique Problem. Please Please help **URL:** \ **Category:** Logstash\ **Created:** [March 17, 2020, 12:46pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924 "2020-03-17T12:46:46Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 17, 2020, 12:46pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/1 "2020-03-17T12:46:47Z")
Hy Everyone myfile.log ``` 17:54:24,429 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@4ab41e5e: Principal: {"lastName":"","address":null,"relativeIds":"135,117,118,120,124,","gender":"Male","city":433,"userId":4265,"firstName":"Shiv","mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर","imageUrl":"","name":"Shiv","middleName":"","enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare","state":21,"email":"","key":"kuchbhi@1234"}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff4c9c: RemoteIpAddress: 192.168.0.119; SessionId: ta5yfu0ukunudcea7j8t1ekd; Granted Authorities: com.kreativ.hellodox.core.components.security.PatientAccountDetails$1@57f53d7 ``` logstash.conf ``` input { file { path => "/home/elk/Downloads/alllogs/myfile.log" sincedb_path => "/dev/null" start_position => "beginning" } } filter { grok{ match => {"message" => "%{TIME} %{WORD}\s(?[^(]*)%{GREEDYDATA:message}\{%{GREEDYDATA:key_pairs}\}%{GREEDYDATA:more_data}"} } if "ExUsernamePasswordAuthenticationFilter.successfulAuthenticatio" in [java_method] { mutate { gsub => [ "key_pairs", "[\\?\t?#-]", "", "key_pairs", '"', ''] } mutate { split => ["key_pairs", ","] } }else{ drop { } } } output { stdout {} } ``` Issue: The output of this conf file is ``` "key_pairs" => [ [0] "lastName:", [1] "address:null", [2] "relativeIds:135", [3] "117", [4] "118", [5] "120", [6] "124", [7] "", [8] "gender:Male", [9] "city:433", [10] "userId:4265", [11] "firstName:Shiv", [12] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर", [13] "imageUrl:", [14] "name:Shiv", [15] "middleName:", [16] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare", [17] "state:21", [18] "email:", [19] "key:kuchbhi@1234" ] ``` Output Needed ``` "key_pairs" => [ [0] "lastName:", [1] "address:null", [2] "relativeIds:135","117","118","120","124", [3] "gender:Male", [4] "city:433", [5] "userId:4265", [6] "firstName:Shiv", [7] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर", [8] "imageUrl:", [9] "name:Shiv", [10] "middleName:", [11] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare", [12] "state:21", [13] "email:", [14] "key:kreativsarg@1234" ] ``` Here is the issue with **relativeIds** key beacause values inside relativeIds changes it can be null also Please help any suggestion will be helpful ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 17, 2020, 2:52pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/2 "2020-03-17T14:52:28Z")
Do you believe me if I tell you I didn't get anything of what you wrote? We're in the Logstash section of the forum. Logstash is basically a pipeliner. Something goes in, something comes out, often edited somehow. Now, 1 - What is going in ( **input** ) in your case? You can let us know basically posting here the output of the following pipeline: ``` input { whatever_input_you_are_using } filter {} output { stdout{} } ``` 2 - What would you like the pipeline to spit out? Then, if you really want to add something else, add your current pipeline configuration. And please, every part of the code you're going to add, be sure it is **properly spaced, indented and formatted** (let's say something more similar to the pipeline I wrote than to the first message you posted). Thanks ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 17, 2020, 2:57pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/3 "2020-03-17T14:57:47Z")
Thank for Reply @Fabio-sama Sorry for the inconvenience Here is my field ``` filter{ #Something here "key_pairs" => "lastName:,address:null,relativeIds:135,117,118,120,124,,gender:Male,city:433,userId:4265,firstName:Shiv,mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर,imageUrl:,name:Shiv,middleName:,enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare,state:21,email:,key:kuchbhi@1234", } ``` I want this field to be parsed as key value but, When I apply ``` mutate { split => ["key_pairs", ","] } ``` the output will be like ``` "key_pairs" => [ [0] "lastName:", [1] "address:null", [2] "relativeIds:135", [3] "117", [4] "118", [5] "120", [6] "124", [7] "", [8] "gender:Male", [9] "city:433", [10] "userId:4265", [11] "firstName:Shiv", [12] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर", [13] "imageUrl:", [14] "name:Shiv", [15] "middleName:", [16] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare", [17] "state:21", [18] "email:", [19] "key:kreativsarg@1234" ] ``` Here you can see that due to key **relativeIds** I got such type of output, whereas I wanted something like ``` "key_pairs" => [ [0] "lastName:", [1] "address:null", [2] "relativeIds:135","117","118","120","124", [3] "gender:Male", [4] "city:433", [5] "userId:4265", [6] "firstName:Shiv", [7] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर", [8] "imageUrl:", [9] "name:Shiv", [10] "middleName:", [11] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare", [12] "state:21", [13] "email:", [14] "key:kreativsarg@1234" ] ``` How can I get the output like the second one.Please let me know if you want something more Thank You ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 17, 2020, 3:38pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/4 "2020-03-17T15:38:26Z")
Ok so, first of all I think you did something wrong in your post because this has no sense at all: ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1de51ddd6e100b5d6baf9a0944f4ec1fd255b7a9.png) Anyway, are you sure you want your output to be like that and not something like: ``` { "lastName": null, "address": null, "relativeIds": ["135","117","118","120","124"] "gender": "Male", "city": "433", "userId": "4265", "firstName": "Shiv", "mrTrendingTags": "निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर", "imageUrl": null, "name:Shiv": null, "middleName": null, "enTrendingTags": "Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare", "state": "21", "email": null, "key": "kreativsarg@1234" } ``` with each of those values being a separate field? Anyway, if what you wrote is what you really want, does the `gender` field always come immediately after the `relativeIds` field? ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 17, 2020, 4:07pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/5 "2020-03-17T16:07:08Z")
Really Sorry @Fabio-sama Thanks for replying I edited the post , I think i may provide a better understanding of problem ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 17, 2020, 5:46pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/7 "2020-03-17T17:46:14Z")
Ok so, please, try to indent your code or I'll have to do it each time in my editor and it gets kinda annoying. Anyway, are you sure the comment `# replace backslashes, question marks, hashes, and minuses with a dot "."` is correct? Because you're replacing those stuff with nothing. Also, I'm sorry if I insist with my previous question but > are you sure you want your output to be like that and not something like: > > ``` > { > "lastName": null, > "address": null, > "relativeIds": ["135","117","118","120","124"] > "gender": "Male", > "city": "433", > "userId": "4265", > "firstName": "Shiv", > "mrTrendingTags": "निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर", > "imageUrl": null, > "name:Shiv": null, > "middleName": null, > "enTrendingTags": "Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare", > "state": "21", > "email": null, > "key": "kreativsarg@1234" > } > > ``` You sure you want an array with all that items inside, impossible to search through? And finally (again another unanswered previous question of mine) > does the `gender` field always come immediately after the `relativeIds` field? ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 18, 2020, 4:13am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/8 "2020-03-18T04:13:12Z")
Hy @Fabio-sama ``` Anyway, are you sure the comment `# replace backslashes, question marks, hashes, and minuses with a dot "."` is correct ``` The output I got after grok is something like ``` {\"lastName\":\"\",\"address\":null,\"relativeIds\":\"135,117,118,120,124,\",\"gender\":\"Male\",\"city\":433,\"userId\":4265,\"firstName\":\"Shiv\",\"mrTrendingTags\":\"#निरोगी जिवन\\t\\t#आरोग्याचे फायदे\\t\\t#आहार आणि पोषण\\t\\t#वजन कमी होणे\\t\\t#स्किनकेअर\",\"imageUrl\":\"\",\"name\":\"Shiv\",\"middleName\":\"\",\"enTrendingTags\":\"#Healthy Living\\t\\t#Health Benefits\\t\\t#Diet and Nutrition\\t\\t#Weight Loss\\t\\t#SkinCare\",\"state\":21,\"email\":\"\",\"key\":\"kuchbhi@1234\"} ``` so I replaced backslash with nothing ``` 2.are you sure you want your output to be like that and not something like: ``` Output you provided will be fine and never gonna search for relativIds, so it will not be any issue ``` 3. does the gender field always come immediately after the relativeIds field? ``` Yes, As you and see in mylog the gender always come after relativIds field ``` Attention: The actual issue here is that my relativeIds field value varies from null to any number how can I handle them ``` Thank You ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 18, 2020, 7:36am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/9 "2020-03-18T07:36:08Z")
Ok then, > so I replaced backslash with nothing it means (as I suggested) the comment is wrong since you do replace those things with "nothing", not with "dots". Anyway, you won't possibly need that part anymore. > Output you provided will be fine and never gonna search for relativIds, so it will not be any issue Ok so, since your `key_pairs` field looks like a perfect json without the opening and trailing curly brackets (because in the grok you used them to find that field and obviously exclude them from the field itself), what about adding them back and then parse it as a json? Like this: ``` filter { grok{ match => {"message" => "%{TIME} %{WORD}\s(?[^(]*)%{GREEDYDATA:message}\{%{GREEDYDATA:key_pairs}\}%{GREEDYDATA:more_data}"} } if "ExUsernamePasswordAuthenticationFilter.successfulAuthenticatio" in [java_method] { # Add leading and trailing curly brackets to the key_pairs field mutate { replace => { "key_pairs" => "{%{key_pairs}}" } } # Extracting fields from the key_pairs field json { source => "key_pairs" } } else { drop { } } } ``` The whole point is you used the `split` filter in an unusual ways, because you use it on a key-value structure (like a json). In that case a [kv filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) would best suit your case, or if the field is a valid JSON (as in your example) a [json filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) is even better. Let me know if your pipeline works replacing your filter section with the one I wrote for you. ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 18, 2020, 8:33am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/10 "2020-03-18T08:33:46Z")
Thanks @Fabio-sama Its work, now I just have to remove unwanted fields. ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 18, 2020, 8:36am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/11 "2020-03-18T08:36:14Z")
Glad it worked. For the removing part, just add a ``` mutate { remove_field => ["field1", "field2"] } ``` wherever you want in your filter section. ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 18, 2020, 11:12am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/12 "2020-03-18T11:12:16Z")
HY @Fabio-sama Sorry to disturb you again I want a suggestion I have two logs here ``` 18:59:21,523 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ef3c8321: Principal: {"lastName":null,"address":null,"relativeIds":"","gender":"Male","city":433,"userId":4279,"firstName":"Vishnu","mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर","imageUrl":"","name":"Vishnu","middleName":null,"enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare","state":21,"email":null,"key":"kreativsarg@1234"}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ef30: RemoteIpAddress: 103.9.74.17; SessionId: 5pn40bk7kcpngf68c978cqgp; Granted Authorities: com.kreativ.hellodox.core.components.security.PatientAccountDetails$1@3916e904 19:03:32,776 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ec812ac0: Principal: {"firstName":"Sandip ","phone":1989062742,"profileId":627002,"docUpload":1,"pictureUrl":"https:\/\/s3.ap-south-1.amazonaws.com\/fortmirrorprod\/627002\/userPhoto\/small6270021528951513196.jpg","name":"Sandip Jagtap","verified":1,"mobileVerified":0,"userId":627002,"status":1}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@7798: RemoteIpAddress: 103.9.74.17; SessionId: 1lv3ha0pnj0eq1kioppe4pocub; Granted Authorities: com.kreativ.hellodox.core.components.security.AccountUserDetails$1@2f658aa8 ``` you can see they are almost similar and the if condition I created in my config is true for both the only difference is the json Please can you suggest how can I place them in different indicies. Thanks ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 18, 2020, 11:37am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/13 "2020-03-18T11:37:13Z")
No bother. How would you distinguish the two jsons? I mean, is there any particular field you'd use to distinguish which one must go to a specific index? In other words, on which basis would you create separate indices? ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 18, 2020, 11:51am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/14 "2020-03-18T11:51:28Z")
@Fabio-sama These two logs contain different information and the JSONs are somewhat different from each other so can we separate them dependent on there key present in them I have no idea about that . Please whatever you suggest. I will follow. The thing I wanted is to send them at two different indicies because they are different. So may be its better ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 18, 2020, 12:12pm UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/15 "2020-03-18T12:12:46Z")
Well to be honest the two logs look pretty similar apart from the json inside. Whether to create two separate indices or not depends on what you have to do with these docs eventually. At a first glance I'd put them in two different indices with a common root (in case in future I want to make some aggregations only on the `userId` field for example). To do this though, you have to be sure they do not have an identical field but mapped in different way (for example phone number once mapped as a integer and another time as a string). If that's the case, you'll have a conflict on the shared index-pattend. That said, as I previously said, it depends on your final implementation. Let's assume these two are your jsons: ``` { "firstName":"Vishnu", "lastName":null, "address":null, "relativeIds":"", "gender":"Male", "city":433, "userId":4279, "mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर", "imageUrl":"", "name":"Vishnu", "middleName":null, "enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare", "state":21, "email":null, "key":"kreativsarg@1234" } { "firstName":"Sandip ", "phone":1989062742, "profileId":627002, "docUpload":1, "pictureUrl":"https:\/\/s3.ap-south-1.amazonaws.com\/fortmirrorprod\/627002\/userPhoto\/small6270021528951513196.jpg", "name":"Sandip Jagtap", "verified":1, "mobileVerified":0, "userId":627002, "status":1 } ``` I don't know where these two come from and what make them unique (you should) but let's assume you want to make a index full of those events having the `enTrendingTags` field and another one with events having the `pictureUrl` field. If that's the case, after you parse the json in the filter section, in your output you can do something like: ``` output { if [enTrendingTags] { elasticsearch { hosts => ["your_host:your_port"] index => "same_root-enTrendingTags-events" } } if [pictureUrl] { elasticsearch { hosts => ["your_host:your_port"] index => "same_root-pictureUrl-events" } } } ``` Obviously you have to find more appropriate names for the indices but that is the basic concepts. I cannot tell you how to divide your indices since I do not know where those data come from, what you will have to do with that and how your architecture is structured. I can only tell you how to do things, not what you have to do. ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 19, 2020, 5:27am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/16 "2020-03-19T05:27:18Z")
Thank @Fabio-sama Here is the complete scenario. These logs are of UserLogin ```auto 18:59:21,523 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ef3c8321: Principal: {"lastName":null,"address":null,"relativeIds":"","gender":"Male","city":433,"userId":4279,"firstName":"Vishnu","mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर","imageUrl":"","name":"Vishnu","middleName":null,"enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare","state":21,"email":null,"key":"kreativsarg@1234"}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ef30: RemoteIpAddress: 103.9.74.17; SessionId: 5pn40bk7kcpngf68c978cqgp; Granted Authorities: com.kreativ.hellodox.core.components.security.PatientAccountDetails$1@3916e904 ``` Some User Information ```auto 19:03:32,776 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ec812ac0: Principal: {"firstName":"Sandip ","phone":1989062742,"profileId":627002,"docUpload":1,"pictureUrl":"https:\/\/s3.ap-south-1.amazonaws.com\/fortmirrorprod\/627002\/userPhoto\/small6270021528951513196.jpg","name":"Sandip Jagtap","verified":1,"mobileVerified":0,"userId":627002,"status":1}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@7798: RemoteIpAddress: 103.9.74.17; SessionId: 1lv3ha0pnj0eq1kioppe4pocub; Granted Authorities: com.kreativ.hellodox.core.components.security.AccountUserDetails$1@2f658aa8 ``` These logs are saved in same file and the file get updated when new user logged in . Now I wanted to store these data to elasticsearch and using kibana I want to visulaize which user is logged in now that's it. Any plan you suggest I will follow. I think i need to create mapping and edit elasticsearch.yml ---
**Author:** ![Fabio-sama](https://avatars.discourse-cdn.com/v4/letter/f/b9e5f3/32.png) [@Fabio-sama](https://discuss.elastic.co/u/Fabio-sama)\ **Post date:** [March 19, 2020, 9:21am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/17 "2020-03-19T09:21:34Z")
Ok so, not knowing what you might want to do with these data in the future and assuming the first type of info are linked to the single login while the second to static user data, I'd possibly create the following indices: - {projectname}-user-login - {projectname}-user-registry - {projectname}-users And I'd put the first type of info (the login ones) in the first index, the second type of info in the registry one. I'd use the third one to mix the two types of info (admitting they only overlap on identical fields) using the `userId` field as id of the document (both json should have the `userId` field obviously and it should uniquely identify a user). Obviously you'll update docs in this last index only with the latest info of the login type and it'll help you in case you want to show some personal details (from the user information type of log) linked to a user's latest login. It means in your pipeline you should do something like: ``` output { if [key] { elasticsearch { hosts => ["your_host:your_port"] index => "yourproject-user-login" } } if [profileId] { elasticsearch { hosts => ["your_host:your_port"] index => "yourproject-user-registry" } } elasticsearch { hosts => ["your_host:your_port"] index => "yourproject-user-registry" action => "update" document_id => "%{userId}" } } ``` Obviously you have to analyze your logs and find some unique and always present fields that can distinguish between the two types (based on the logs you provided, I used here the fields `key` and `profileId` for the login and information type of logs respectively). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 16, 2020, 9:21am UTC](https://discuss.elastic.co/t/issue-with-parsing-unique-problem-please-please-help/223924/18 "2020-04-16T09:21:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.