Not able to create the filter with correct pattern

Elastic Stack Logstash
1

input file -
{"apps":{"app":[{"id":"application_1571691267503_939654","user":"Z013J5W","name":"HIVE-d9904734e7fa7249f0","queue":"default","state":"RUNNING","finalStatus":"UNDEFINED","progress":55.259087}],"logAggregationStatus":"NOT_START","unmanagedApplication":false,"amNodeLabelExpression":""}

filter using like below. Something worked when I used 5.2 now I am with 7.2. I am getting warning saying "[WARN ] 2019-10-30 17:19:02.329 [[main]>worker49] split - Only String and Array types are splittable. field:apps is of type = Hash" and nothing is ingested in Elasticsearch.

filter {
json {
source => "message"
}
split { field => "[apps]" }
split { field => "[apps][app]" }
}

2

As it says, [apps] is not an array so you cannot do this. Remove that line.

However, that should not prevent the event being indexed. Are there any errors in the elasticsearch logs?

3

Hi Badger,

Issue is resolved by removing the first [apps] and data is correctly loading with the following changes.

filter {
json {
source => "message"}
split { field => "[apps][app]" }
split { field => "[apps][app][resourceRequests]" }
}

Now I have a problem
I want to load only few values to elasticsearch output.. How to specify that specific values?
Reason : I am getting huge data as part of resource Requests and it is the big hurdle to load the data, so I want to cut it down while loading.

4

filter worked -
split {remove_field => "apps.app.resourceRequests"}

5

I want to remove multiple fields, which is not working..

split {remove_field => ["apps.app.resourceRequests","apps.app.resourceRequests.capability","apps.app.resourceRequests.resourceName","apps.app.resourceRequests.relaxLocality","apps.app.trackingUrl","command","apps.app.amContainerLogs","apps.app.amNodeLabelExpression","tags"]}

Actually, I have a lot of data to parse. This particular logstash script not loading. Not able to find any log...even logstash also not showing anything

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.