Logstash Extract Elasticsearch Nested Fields

Elastic Stack Logstash
21

Could you please help me with what I am missing because of which my fields are not getting captured in csv file..

22

There is one hope now ....

I used mutate fields ... and i got the record successfully.

Only problem is now if event is empty .. then it prints entire mutate field.

Could you please suggest ways to avoid this...

input {
  elasticsearch {
    hosts => "10.10.100.10:9200"
    query => '{"_source" : ["userId", "timeStamp", "backupSettings.backgroundUploading", "backupSettings.connectionType", "backupSettings.contactBackup", "backupSettings.contactPermission", "backupSettings.photoBackup", "backupSettings.photoQuality", "backupSettings.storagePermission", "backupSettings.videoBackup", "appEvents.attribute_num.filesPendingForUpload","appEvents.attribute_num.filesUploadedSinceLastEvent"],"query" : { "match_all": {} }}'
    size => 10000
    scroll => "5m"
    index => "backup"
  }
}



filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
}

output {
  csv {
    fields => ["userId", "timeStamp", "[backupSettings][backgroundUploading]", "[backupSettings][connectionType]", "[backupSettings][contactBackup]", "[backupSettings][contactPermission]", "[backupSettings][photoBackup]", "[backupSettings][photoQuality]", "[backupSettings][storagePermission]", "[backupSettings][videoBackup]", "filesPendingForUpload", "filesUploadedSinceLastEvent"]
    path => "/tmp/exp.csv"
  }
}



6018a66855b6471ebf976b2d551c9731,1623241661046,Y,WLC,Y,Y,CAM,Original,Y,OFF,8,173
6018a66855b6471ebf976b2d551c9731,1623256154896,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}
6018a66855b6471ebf976b2d551c9731,1623254680762,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}
23

Possibly use a prune filter where the default value for blacklist_names will delete fields where a sprintf reference failed. I think the unresolved sprintf reference will be top-level so then you should get an empty field.

24

Thank You for your response.

I will check and reply in here ...

25

Hello Badger

I tried below ways .. but it did not helped me ...
I am sure , my writing of prune filter is not correct .. .hence its not working properly as expected ..

Could you please help me with its syntax to be mentioned ...

filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    prune {
        blacklist_names => [ "filesPendingForUpload", "filesUploadedSinceLastEvent" ]
    }
}


filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    prune {
        blacklist_names => [ "filesPendingForUpload", "filesUploadedSinceLastEvent" ]
        blacklist_values => [ "" , "" ]
    }
}
26

I also tried below :

filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    prune {
        blacklist_values => [ "filesPendingForUpload" ,"" , "filesUploadedSinceLastEvent", "" ]
    }
}
27

I would like to high light in here that for some documents we have this field missing ....

for example below is the set of document does not have
[appEvents][0][attribute_num][filesUploadedSinceLastEvent] and [appEvents][0][attribute_num][filesPendingForUpload]

{
        "userId" => "6018a66855b6471ebf976b2d551c9731",
    "@timestamp" => 2021-06-16T13:40:32.584Z,
      "@version" => "1",
     "timeStamp" => 1623256154896
}

In such cases , output is like below in csv file

6018a66855b6471ebf976b2d551c9731,1623256154896,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}
6018a66855b6471ebf976b2d551c9731,1623254680762,,,,,,,,,%{[appEvents][0][attribute_num][filesPendingForUpload]},%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}
28

Finally I have achieved it ...

filter {
    mutate {
        add_field => { "filesPendingForUpload" => "%{[appEvents][0][attribute_num][filesPendingForUpload]}" }
        add_field => { "filesUploadedSinceLastEvent" => "%{[appEvents][0][attribute_num][filesUploadedSinceLastEvent]}" }
    }
    if ![appEvents][0][attribute_num][filesPendingForUpload] {
        mutate { remove_field => [ "filesPendingForUpload" ]  }
}
    if ![appEvents][0][attribute_num][filesUploadedSinceLastEvent] {
        mutate { remove_field => [ "filesUploadedSinceLastEvent" ] }
}
}
29

Thank You very much Badger for your responses ..

  1. You mentioned that mutate + add_filed should have helped me ...

Then i added my mutate code back to config file and it was giving results.

  1. You mentioned to use prune filter ...

Although it did not helped me .. but I got the direction to think in the manner of fields and I used if conditions and finally i got the expected output ...

30

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.