Issue With Parsing Unique Problem. Please Please help

pariksh · March 17, 2020, 12:46pm

Hy Everyone

myfile.log

17:54:24,429 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@4ab41e5e: Principal: {"lastName":"","address":null,"relativeIds":"135,117,118,120,124,","gender":"Male","city":433,"userId":4265,"firstName":"Shiv","mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर","imageUrl":"","name":"Shiv","middleName":"","enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare","state":21,"email":"","key":"kuchbhi@1234"}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff4c9c: RemoteIpAddress: 192.168.0.119; SessionId: ta5yfu0ukunudcea7j8t1ekd; Granted Authorities: com.kreativ.hellodox.core.components.security.PatientAccountDetails$1@57f53d7

logstash.conf

input {
file {
path => "/home/elk/Downloads/alllogs/myfile.log"
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
filter {
grok{
match => {"message" => "%{TIME} %{WORD}\s(?<java_method>[^(]*)%{GREEDYDATA:message}\{%{GREEDYDATA:key_pairs}\}%{GREEDYDATA:more_data}"}
}
if "ExUsernamePasswordAuthenticationFilter.successfulAuthenticatio" in [java_method]
{
  mutate {
  gsub => [
  "key_pairs", "[\\?\t?#-]", "",
  "key_pairs", '"', '']
   }
  mutate {
  split => ["key_pairs", ","]
    }
    }else{
    drop { }
    }
    }
output {
stdout {}
    }

Issue:
The output of this conf file is

"key_pairs" => [
[ 0] "lastName:",
[ 1] "address:null",
[ 2] "relativeIds:135",
[ 3] "117",
[ 4] "118",
[ 5] "120",
[ 6] "124",
[ 7] "",
[ 8] "gender:Male",
[ 9] "city:433",
[10] "userId:4265",
[11] "firstName:Shiv",
[12] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर",
[13] "imageUrl:",
[14] "name:Shiv",
[15] "middleName:",
[16] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare",
[17] "state:21",
[18] "email:",
[19] "key:kuchbhi@1234"
]

Output Needed

"key_pairs" => [
[ 0] "lastName:",
[ 1] "address:null",
[ 2] "relativeIds:135","117","118","120","124",
[ 3] "gender:Male",
[ 4] "city:433",
[5] "userId:4265",
[6] "firstName:Shiv",
[7] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर",
[8] "imageUrl:",
[9] "name:Shiv",
[10] "middleName:",
[11] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare",
[12] "state:21",
[13] "email:",
[14] "key:kreativsarg@1234"
]

Here is the issue with relativeIds key beacause values inside relativeIds changes it can be null also

Please help any suggestion will be helpful

Fabio-sama · March 17, 2020, 2:52pm

Do you believe me if I tell you I didn't get anything of what you wrote?

We're in the Logstash section of the forum. Logstash is basically a pipeliner. Something goes in, something comes out, often edited somehow.

Now,

1 - What is going in (input) in your case? You can let us know basically posting here the output of the following pipeline:

input {
  whatever_input_you_are_using
}

filter {}

output {
  stdout{}
}

2 - What would you like the pipeline to spit out?

Then, if you really want to add something else, add your current pipeline configuration.

And please, every part of the code you're going to add, be sure it is properly spaced, indented and formatted (let's say something more similar to the pipeline I wrote than to the first message you posted).

Thanks

pariksh · March 17, 2020, 2:57pm

Thank for Reply @Fabio-sama

Sorry for the inconvenience
Here is my field

filter{
#Something here

"key_pairs" => "lastName:,address:null,relativeIds:135,117,118,120,124,,gender:Male,city:433,userId:4265,firstName:Shiv,mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर,imageUrl:,name:Shiv,middleName:,enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare,state:21,email:,key:kuchbhi@1234",
}

I want this field to be parsed as key value but, When I apply

mutate {
split => ["key_pairs", ","]    
}

the output will be like

"key_pairs" => [
    [ 0] "lastName:",
    [ 1] "address:null",
    [ 2] "relativeIds:135",
    [ 3] "117",
    [ 4] "118",
    [ 5] "120",
    [ 6] "124",
    [ 7] "",
    [ 8] "gender:Male",
    [ 9] "city:433",
    [10] "userId:4265",
    [11] "firstName:Shiv",
    [12] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर",
    [13] "imageUrl:",
    [14] "name:Shiv",
    [15] "middleName:",
    [16] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare",
    [17] "state:21",
    [18] "email:",
    [19] "key:kreativsarg@1234"
]

Here you can see that due to key relativeIds I got such type of output, whereas I wanted something like

"key_pairs" => [
    [ 0] "lastName:",
    [ 1] "address:null",
    [ 2] "relativeIds:135","117","118","120","124",
    [ 3] "gender:Male",
    [ 4] "city:433",
    [5] "userId:4265",
    [6] "firstName:Shiv",
    [7] "mrTrendingTags:निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर",
    [8] "imageUrl:",
    [9] "name:Shiv",
    [10] "middleName:",
    [11] "enTrendingTags:Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare",
    [12] "state:21",
    [13] "email:",
    [14] "key:kreativsarg@1234"
]

How can I get the output like the second one.Please let me know if you want something more

Thank You

Fabio-sama · March 17, 2020, 3:38pm

Ok so, first of all I think you did something wrong in your post because this has no sense at all:

Anyway, are you sure you want your output to be like that and not something like:

{
  "lastName": null,
  "address": null,
  "relativeIds": ["135","117","118","120","124"]
  "gender": "Male",
  "city": "433",
  "userId": "4265",
  "firstName": "Shiv",
  "mrTrendingTags": "निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर",
  "imageUrl": null,
  "name:Shiv": null,
  "middleName": null,
  "enTrendingTags": "Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare",
  "state": "21",
  "email": null,
  "key": "kreativsarg@1234"
}

with each of those values being a separate field?

Anyway, if what you wrote is what you really want, does the gender field always come immediately after the relativeIds field?

pariksh · March 17, 2020, 4:07pm

Really Sorry @Fabio-sama
Thanks for replying

I edited the post , I think i may provide a better understanding of problem

Fabio-sama · March 17, 2020, 5:46pm

Ok so,

please, try to indent your code or I'll have to do it each time in my editor and it gets kinda annoying.

Anyway, are you sure the comment # replace backslashes, question marks, hashes, and minuses with a dot "." is correct? Because you're replacing those stuff with nothing.

Also, I'm sorry if I insist with my previous question but

are you sure you want your output to be like that and not something like:

{
  "lastName": null,
  "address": null,
  "relativeIds": ["135","117","118","120","124"]
  "gender": "Male",
  "city": "433",
  "userId": "4265",
  "firstName": "Shiv",
  "mrTrendingTags": "निरोगी जिवनttआरोग्याचे फायदेttआहार आणि पोषणttवजन कमी होणेttस्किनकेअर",
  "imageUrl": null,
  "name:Shiv": null,
  "middleName": null,
  "enTrendingTags": "Healthy LivingttHealth BenefitsttDiet and NutritionttWeight LossttSkinCare",
  "state": "21",
  "email": null,
  "key": "kreativsarg@1234"
}

You sure you want an array with all that items inside, impossible to search through?

And finally (again another unanswered previous question of mine)

does the gender field always come immediately after the relativeIds field?

pariksh · March 18, 2020, 4:13am

Hy @Fabio-sama

 Anyway, are you sure the comment `# replace backslashes, question marks, hashes, and minuses with a dot "."` is correct

The output I got after grok is something like

{\"lastName\":\"\",\"address\":null,\"relativeIds\":\"135,117,118,120,124,\",\"gender\":\"Male\",\"city\":433,\"userId\":4265,\"firstName\":\"Shiv\",\"mrTrendingTags\":\"#निरोगी जिवन\\t\\t#आरोग्याचे फायदे\\t\\t#आहार आणि पोषण\\t\\t#वजन कमी होणे\\t\\t#स्किनकेअर\",\"imageUrl\":\"\",\"name\":\"Shiv\",\"middleName\":\"\",\"enTrendingTags\":\"#Healthy Living\\t\\t#Health Benefits\\t\\t#Diet and Nutrition\\t\\t#Weight Loss\\t\\t#SkinCare\",\"state\":21,\"email\":\"\",\"key\":\"kuchbhi@1234\"}

so I replaced backslash with nothing

2.are you sure you want your output to be like that and not something like:

Output you provided will be fine and never gonna search for relativIds, so it will not be any issue

3. does the gender field always come immediately after the relativeIds field?

Yes, As you and see in mylog the gender always come after relativIds field

Attention:  The actual issue here is that my relativeIds field value
            varies from null to any number how can I handle them

Thank You

Fabio-sama · March 18, 2020, 7:36am

Ok then,

so I replaced backslash with nothing

it means (as I suggested) the comment is wrong since you do replace those things with "nothing", not with "dots". Anyway, you won't possibly need that part anymore.

Output you provided will be fine and never gonna search for relativIds, so it will not be any issue

Ok so, since your key_pairs field looks like a perfect json without the opening and trailing curly brackets (because in the grok you used them to find that field and obviously exclude them from the field itself), what about adding them back and then parse it as a json? Like this:

filter {
  grok{
    match => {"message" => "%{TIME} %{WORD}\s(?<java_method>[^(]*)%{GREEDYDATA:message}\{%{GREEDYDATA:key_pairs}\}%{GREEDYDATA:more_data}"}
  }

  if "ExUsernamePasswordAuthenticationFilter.successfulAuthenticatio" in [java_method] {
    # Add leading and trailing curly brackets to the key_pairs field
    mutate {
      replace => { "key_pairs" => "{%{key_pairs}}" }
    }
    
    # Extracting fields from the key_pairs field
    json {
      source => "key_pairs"
    }
  } else {
    drop { }
  }
}

The whole point is you used the split filter in an unusual ways, because you use it on a key-value structure (like a json). In that case a kv filter would best suit your case, or if the field is a valid JSON (as in your example) a json filter is even better.

Let me know if your pipeline works replacing your filter section with the one I wrote for you.

pariksh · March 18, 2020, 8:33am

Thanks @Fabio-sama

Its work, now I just have to remove unwanted fields.

Fabio-sama · March 18, 2020, 8:36am

Glad it worked. For the removing part, just add a

mutate {
  remove_field => ["field1", "field2"]
}

wherever you want in your filter section.

pariksh · March 18, 2020, 11:12am

HY @Fabio-sama

Sorry to disturb you again

I want a suggestion

I have two logs here

18:59:21,523 DEBUG       ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ef3c8321: Principal: {"lastName":null,"address":null,"relativeIds":"","gender":"Male","city":433,"userId":4279,"firstName":"Vishnu","mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर","imageUrl":"","name":"Vishnu","middleName":null,"enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare","state":21,"email":null,"key":"kreativsarg@1234"}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ef30: RemoteIpAddress: 103.9.74.17; SessionId: 5pn40bk7kcpngf68c978cqgp; Granted Authorities: com.kreativ.hellodox.core.components.security.PatientAccountDetails$1@3916e904


19:03:32,776 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ec812ac0: Principal: {"firstName":"Sandip ","phone":1989062742,"profileId":627002,"docUpload":1,"pictureUrl":"https:\/\/s3.ap-south-1.amazonaws.com\/fortmirrorprod\/627002\/userPhoto\/small6270021528951513196.jpg","name":"Sandip Jagtap","verified":1,"mobileVerified":0,"userId":627002,"status":1}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@7798: RemoteIpAddress: 103.9.74.17; SessionId: 1lv3ha0pnj0eq1kioppe4pocub; Granted Authorities: com.kreativ.hellodox.core.components.security.AccountUserDetails$1@2f658aa8

you can see they are almost similar and the if condition I created in my config is true for both the only difference is the json

Please can you suggest how can I place them in different indicies.

Thanks

Fabio-sama · March 18, 2020, 11:37am

No bother.

How would you distinguish the two jsons? I mean, is there any particular field you'd use to distinguish which one must go to a specific index?

In other words, on which basis would you create separate indices?

pariksh · March 18, 2020, 11:51am

@Fabio-sama

These two logs contain different information and the JSONs are somewhat different from each other so can we separate them dependent on there key present in them

I have no idea about that . Please whatever you suggest. I will follow. The thing I wanted is to send them at two different indicies because they are different. So may be its better

Fabio-sama · March 18, 2020, 12:12pm

Well to be honest the two logs look pretty similar apart from the json inside. Whether to create two separate indices or not depends on what you have to do with these docs eventually.

At a first glance I'd put them in two different indices with a common root (in case in future I want to make some aggregations only on the userId field for example).
To do this though, you have to be sure they do not have an identical field but mapped in different way (for example phone number once mapped as a integer and another time as a string). If that's the case, you'll have a conflict on the shared index-pattend.

That said, as I previously said, it depends on your final implementation. Let's assume these two are your jsons:

{
   "firstName":"Vishnu",
   "lastName":null,
   "address":null,
   "relativeIds":"",
   "gender":"Male",
   "city":433,
   "userId":4279,
   "mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर",
   "imageUrl":"",
   "name":"Vishnu",
   "middleName":null,
   "enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare",
   "state":21,
   "email":null,
   "key":"kreativsarg@1234"
}

{
   "firstName":"Sandip ",
   "phone":1989062742,
   "profileId":627002,
   "docUpload":1,
   "pictureUrl":"https:\/\/s3.ap-south-1.amazonaws.com\/fortmirrorprod\/627002\/userPhoto\/small6270021528951513196.jpg",
   "name":"Sandip Jagtap",
   "verified":1,
   "mobileVerified":0,
   "userId":627002,
   "status":1
}

I don't know where these two come from and what make them unique (you should) but let's assume you want to make a index full of those events having the enTrendingTags field and another one with events having the pictureUrl field.

If that's the case, after you parse the json in the filter section, in your output you can do something like:

output {
  if [enTrendingTags] {
    elasticsearch {
      hosts => ["your_host:your_port"]
      index => "same_root-enTrendingTags-events"
    }
  }

  if [pictureUrl] {
    elasticsearch {
      hosts => ["your_host:your_port"]
      index => "same_root-pictureUrl-events"
    }
  }
}

Obviously you have to find more appropriate names for the indices but that is the basic concepts.

I cannot tell you how to divide your indices since I do not know where those data come from, what you will have to do with that and how your architecture is structured.
I can only tell you how to do things, not what you have to do.

pariksh · March 19, 2020, 5:27am

Thank @Fabio-sama
Here is the complete scenario. These logs are of
UserLogin

18:59:21,523 DEBUG       ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ef3c8321: Principal: {"lastName":null,"address":null,"relativeIds":"","gender":"Male","city":433,"userId":4279,"firstName":"Vishnu","mrTrendingTags":"#निरोगी जिवन\t\t#आरोग्याचे फायदे\t\t#आहार आणि पोषण\t\t#वजन कमी होणे\t\t#स्किनकेअर","imageUrl":"","name":"Vishnu","middleName":null,"enTrendingTags":"#Healthy Living\t\t#Health Benefits\t\t#Diet and Nutrition\t\t#Weight Loss\t\t#SkinCare","state":21,"email":null,"key":"kreativsarg@1234"}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ef30: RemoteIpAddress: 103.9.74.17; SessionId: 5pn40bk7kcpngf68c978cqgp; Granted Authorities: com.kreativ.hellodox.core.components.security.PatientAccountDetails$1@3916e904

Some User Information

19:03:32,776 DEBUG ExUsernamePasswordAuthenticationFilter.successfulAuthentication():319 - Authentication success. Updating SecurityContextHolder to contain: org.springframework.security.authentication.UsernamePasswordAuthenticationToken@ec812ac0: Principal: {"firstName":"Sandip ","phone":1989062742,"profileId":627002,"docUpload":1,"pictureUrl":"https:\/\/s3.ap-south-1.amazonaws.com\/fortmirrorprod\/627002\/userPhoto\/small6270021528951513196.jpg","name":"Sandip Jagtap","verified":1,"mobileVerified":0,"userId":627002,"status":1}; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@7798: RemoteIpAddress: 103.9.74.17; SessionId: 1lv3ha0pnj0eq1kioppe4pocub; Granted Authorities: com.kreativ.hellodox.core.components.security.AccountUserDetails$1@2f658aa8

These logs are saved in same file and the file get updated when new user logged in . Now I wanted to store these data to elasticsearch and using kibana I want to visulaize which user is logged in now that's it. Any plan you suggest I will follow.
I think i need to create mapping and edit elasticsearch.yml

Fabio-sama · March 19, 2020, 9:21am

Ok so, not knowing what you might want to do with these data in the future and assuming the first type of info are linked to the single login while the second to static user data, I'd possibly create the following indices:

{projectname}-user-login
{projectname}-user-registry
{projectname}-users

And I'd put the first type of info (the login ones) in the first index, the second type of info in the registry one.
I'd use the third one to mix the two types of info (admitting they only overlap on identical fields) using the userId field as id of the document (both json should have the userId field obviously and it should uniquely identify a user).
Obviously you'll update docs in this last index only with the latest info of the login type and it'll help you in case you want to show some personal details (from the user information type of log) linked to a user's latest login.

It means in your pipeline you should do something like:

output {
  if [key] {
    elasticsearch {
      hosts => ["your_host:your_port"]
      index => "yourproject-user-login"
    }
  }

  if [profileId] {
    elasticsearch {
      hosts => ["your_host:your_port"]
      index => "yourproject-user-registry"
    }
  }

  elasticsearch {
    hosts => ["your_host:your_port"]
    index => "yourproject-user-registry"
    action => "update"
    document_id => "%{userId}"
  }
}

Obviously you have to analyze your logs and find some unique and always present fields that can distinguish between the two types (based on the logs you provided, I used here the fields key and profileId for the login and information type of logs respectively).

system · April 16, 2020, 9:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.