How to replace backslash from a string

Elastic Stack Logstash
1

Payload:"Raw Customer Details Stored Successfully."

tried to use below filters

mutate {

gsub => [ "Payload", "[\/]+", "" ]
}

2

The payload you show contains neither slash nor backslash. If you want to remove backslash from a field then use

mutate { gsub => [ "Payload", "[\\]", "" ] }
3 
Hi Badger ,

Thanks for your reply.

Here is my complete message structure

APPLOG->|Datetime:2018-07-04 20:50:45:376+0000|Hostname:667fc49f-3ae3-450b-4e92-bc0e/10.255.223.47|ClassName:class com.pepsico.eip.latam.customer.service.CustomerStatusService|MethodName:receivedStatus|Entity:Customer|Type:CustomerRequests|Identifier:80011|LogMessage:Create Customer Status Record|Event:to DataService|Payload:{\"AddressName\":\"Street name\",\"AddressType\":\"1\",\"AddressUsage\":\"code\",\"BarCode\":\"str1234\",\"CityName\":\"City name\",\"CommunicationEmail\":\"exampleemail@example.com\",\"CommunicationPhone\":\"1234\",\"Complement\":\"Address line 5\",\"ContactDepartment\":\"code\",\"ContactName\":\"John Smith\",\"ContactType\":\"code\",\"CustomerName\":\"John Smith\",\"CustomerType\":\"DE\",\"DistributionCenterID\":\"123\",\"Document1\":\"str1234\",\"LastUpdate\":\"2018-05-02T12:31:56.4375764-03:00\",\"Latitude\":123.45,\"Longitude\":123.45,\"MarketUnitID\":1000,\"Neighborhood\":\"Address line 4\",\"Number\":\"str1234\",\"Number2\":\"AdrLine3\",\"Observation\":\"str1234\",\"ObservationType\":\"code\",\"Reference1\":\"Cross street 1\",\"Reference2\":\"Cross street 2\",\"RelationshipStartDate\":\"1900-01-01T00:00:00\",\"RelationshipType\":\"code\",\"RequestID\":\"80011\",\"RouteID\":\"rte1\",\"SourceSystemID\":2,\"StateName\":\"State\",\"SystemID\":549,\"TaxClassification\":\"code\",\"TaxStartDate\":\"1900-01-01T00:00:00\",\"User\":\"userID\",\"VisitPlanStartDate\":\"1900-01-01T00:00:00\",\"VisitPlanType\":\"code\",\"WeekDay\":\"234561\",\"WeekMonth\":\"0\",\"ZipCode\":\"str1234\"}

Here is my Grok filters:

%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}\|Payload:%{GREEDYDATA:Payload}

All the fields are getting parsed correctly but issue is with Event and Payload fields are not getting filtered into seperate fields

"Hostame": "667fc49f-3ae3-450b-4e92-bc0e/10.255.223.47",
    "Type": "CustomerRequests",
    "LogMessage": "Create Customer Status Record",
    "logType": "APPLOG",
    "@version": "1",
    "@timestamp": "2018-07-04T20:50:46.439Z",
    "MethodName": "receivedStatus",
    "host": "10.0.0.28",
    "type": "syslog",
    "port": 56528,
    "Datetime": "2018-07-04 20:50:45:376+0000",
    "class": "class com.pepsico.eip.latam.customer.service.CustomerStatusService"

**"Event": "to DataService|Payload:{\"AddressName\":\"Street name\",\"AddressType\":\"1\",\"AddressUsage\":\"code\",\"BarCode\":\"str1234\",\"CityName\":\"City name\",\"CommunicationEmail\":\"exampleemail@example.com\",\"CommunicationPhone\":\"1234\",\"Complement\":\"Address line 5\",\"ContactDepartment\":\"code\",\"ContactName\":\"John Smith\",\"ContactType\":\"code\",\"CustomerName\":\"John Smith\",\"CustomerType\":\"DE\",\"DistributionCenterID\":\"123\",\"Document1\":\"str1234\",\"LastUpdate\":\"2018-05-02T12:31:56.4375764-03:00\",\"Latitude\":123.45,\"Longitude\":123.45,\"MarketUnitID\":1000,\"Neighborhood\":\"Address line 4\",\"Number\":\"str1234\",\"Number2\":\"AdrLine3\",\"Observation\":\"str1234\",\"ObservationType\":\"code\",\"Reference1\":\"Cross street 1\",\"Reference2\":\"Cross street 2\",\"RelationshipStartDate\":\"1900-01-01T00:00:00\",\"RelationshipType\":\"code\",\"RequestID\":\"80011\",\"RouteID\":\"rte1\",\"SourceSystemID\":2,\"StateName\":\"State\",\"SystemID\":549,\"TaxClassification\":\"code\",\"TaxStartDate\":\"1900-01-01T00:00:00\",\"User\":\"userID\",\"VisitPlanStartDate\":\"1900-01-01T00:00:00\",\"VisitPlanType\":\"code\",\"WeekDay\":\"234561\",\"WeekMonth\":\"0\",\"ZipCode\":\"str1234\"}"**,
    "Entity": "Customer",
    "Identifier": "80011"

i have tried with below grok pattern as well

filter {

 grok {
  match => [ "message" ,'%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}',
             "message" ,'%{WORD:logType}<-\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}',
            "message" ,'%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}\|ErrorCode:%{GREEDYDATA:ErrorCode}\|Payload:%{GREEDYDATA:Payload}',
            "message" ,'%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}\|Payload:%{GREEDYDATA:Payload}']
 }

mutate {
gsub => ["Payload","[\\]",""]
}
}
4

You need to format your posting to make it readable. Select the text of the event and the configuration and click on </> in the toolbar above the composition window.

5

Hi Badger,

I have formatted the post now, can you please check it once.

Regards,
Praveen

6

With the 4th pattern you will get Event matched "to DataService" and Payload matching the JSON. However, because the first pattern is less specific, it will always match that.

I do not see the difference between the 2nd and 3rd patterns. Am I missing something? I would add a start of line anchor (^) to each pattern and change the order of the patterns to be

grok {
    match => [ 
        "message" ,'^%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}\|ErrorCode:%{GREEDYDATA:ErrorCode}\|Payload:%{GREEDYDATA:Payload}',
        "message" ,'^%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}\|Payload:%{GREEDYDATA:Payload}',
        "message" ,'^%{WORD:logType}->\|Datetime:%{GREEDYDATA:Datetime}\|Hostname:%{GREEDYDATA:Hostame}\|ClassName:%{GREEDYDATA:class}\|MethodName:%{WORD:MethodName}\|Entity:%{WORD:Entity}\|Type:%{WORD:Type}\|Identifier:%{WORD:Identifier}\|LogMessage:%{GREEDYDATA:LogMessage}\|Event:%{GREEDYDATA:Event}'
    ]
}

If you want to improve performance I would replace those GREEDYDATA patterns. For example, change %{GREEDYDATA:Datetime} to (?<Datetime>[^|]+)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.