Filter records that contain a specfic field name

Elastic Stack Logstash
1

I have a json input with different fields name and I would like to filter the data in order to keep at the end of the chain only the records that have a specific field

For example in the data set below I just want to keep records that have field " recordid"

{
"records": [
{
"fields": {
"last_reported": 1519897717,
"lon": 2.275342586850291,
"station_id": 7234,
"xy": [
48.8296843301,
2.27534258685
],
"lat": 48.8296843300915,
"name": "Colonel Pierre Avia",
"numbikesavailable": 4,
"capacity": 31,
"is_installed": 1,
"numdocksavailable": 25,
"is_renting": 1,
"is_returning": 1
},
"geometry": {
"coordinates": [
2.27534258685,
48.8296843301
],
"type": "Point"
},
"recordid": "e9a4f7a8874ba8dd9b3d004640d99759b3683005"
},
{
"fields": {
"last_reported": 1519897717,
"lon": 2.275342586850291,
"station_id": 7234,
"xy": [
48.8296843301,
2.27534258685
],
"lat": 48.8296843300915,
"name": "Colonel Pierre Avia",
"numbikesavailable": 4,
"capacity": 31,
"is_installed": 1,
"numdocksavailable": 25,
"is_renting": 1,
"is_returning": 1
},
"geometry": {
"coordinates": [
2.27534258685,
48.8296843301
],
"type": "Point"
},
"recordid": "u9h4f5a0874ba8dd9b3d004874d99759b3689740"
},
{
"fields": {
"last_reported": 1519897923,
"station_id": 54000632,
"is_installed": 0,
"numbikesavailable": 26,
"numdocksavailable": 10,
"is_renting": 0,
"is_returning": 0
}
},
{
"fields": {
"last_reported": 1519897923,
"station_id": 54000632,
"is_installed": 0,
"numbikesavailable": 26,
"numdocksavailable": 10,
"is_renting": 0,
"is_returning": 0
}
},
{
"fields": {
"last_reported": 1519897923,
"station_id": 54000632,
"is_installed": 0,
"numbikesavailable": 26,
"numdocksavailable": 10,
"is_renting": 0,
"is_returning": 0
}
}
]
}

I did not find how to do this with prune filter plugin,
Do you have idea how to do this and could you help me to solve my problem please?

2

I think you could use IF with the drop filter....kinda hacky but I think it should work. The below basically looks for the existence of the Name field, if it doesn't exist it drops the event.

if [Name] {
  } else { 
  drop { }
}
3

thank you for answere , it should work with simple json data but I have an array with nested fields

the filter bellow let pass all data :

filter {
  if [records] {
 }
else { 
  drop { }
}
}

but code bellow won't let pass any record through

> 
> filter {
> 	  if [fields] in [records] //or if [records.fields]  or if [fields]
>   {
>   }
> else { 
>   drop { }
> }
> }

Could you help me to understand how to do this type of filter

4

Here's the documentation for using IF in Logstash, but going off your example, I don't believe you can specify multiple fields, though I've never tried. Also, if you are looking for the existence of a particular value in a field, I have had more success with the below than the example in the documentation.

filter {
  if "John" in [Name] {
  } else {
    drop { }
  }
}

I've seen other filters and outputs use a pipe to separate multiple conditions, though I don't know if that would work. If you want to try, I think it would be something like:

filter {
  if ([Name] | [DOB]) {
  } else {
    drop { }
  }
}
5

thank you, I will ask a more specific question to know if I can solve my specifc issue

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.