# Combined grok pattern for customized logs **URL:** \ **Category:** Logstash\ **Created:** [July 17, 2023, 10:30am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535 "2023-07-17T10:30:14Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 17, 2023, 10:30am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/1 "2023-07-17T10:30:14Z")
i am looking some help and guidenace for parsing the customized logs in one file. i have httpd access logs which have two format and i need to prepare the logstash config/filtering the data. so i tried two different patterns to combined in one grok and as well different grok method under the filter but both of them are not getting filters data. is there any way, that i can put two grok patterns for filtering the customized httpd access logs ? please guide . thanks in advance. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [July 17, 2023, 11:55am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/2 "2023-07-17T11:55:57Z")
Hello, You need to share sample of your logs and also the grok configuration you are using. Please use the preformatted text option, the `` button while shargin configurations and sample messages. ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 18, 2023, 6:18am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/3 "2023-07-18T06:18:08Z")
thanks mate for the response, samples below for logs one trainsications ```auto 10.92.10.11 - - [12/Jul/2023:08:00:02 +0800] - "GET /isalive HTTP/1.1" 200 15bytes "-" "ZK3tAiGP-VpKu4Cf5HWLGgxxxxxx" [-] 0ms ``` another logs ```auto 10.10.110.10 - - [12/Jul/2023:08:00:37 +0800] appxxx2-ig "GET /am/oauth2/.well-known/openid-configuration HTTP/1.1" 200 3897bytes "10.10.110.10" "ZK3tJSGP-VpKu4Cf5xxxxxxxxxx" [-] 37ms ``` another logs ```auto 10.52.11.10 - - [12/Jul/2023:08:09:05 +0800] "POST /am/json/sessions/?_action=getSessionInfo HTTP/1.1" 200 6440bytes "10.52.5.16" "ZK3vIR_wxxxx3QRmANzxxxxxxxx" [-] 17ms ``` these are the types of logs avilable in access\_logs which needs to parse and apply the filtering. i tried to follow multiple grok pattern one is below ````auto %{IP:clientIP} \- \- \[%{HTTPDATE:timestamp}\] %{DATA:client_name} "%{WORD:httpMethod} %{URIPATHPARAM:httpPath} HTTP/%{NUMBER:httpVersion}" %{NUMBER:httpStatusCode} %{NUMBER:responseSize}bytes "%{IP:httpReferer}" "%{DATA:httpUserAgent}" \[%{DATA:requestId}\] %{NUMBER:responseTime}ms``` another one is also ```` '%{IP:client\_ip} - - [%{HTTPDATE:timestamp}] - "%{WORD:client\_method} %{URIPATHPARAM:request\_path} %{DATA:http\_version}" %{NUMBER:status\_code} %{NUMBER:response\_size}bytes "%{DATA:extra}" "%{DATA:user\_agent}" [-] %{NUMBER:response\_time.ms}' } ```auto but all of them are failing. ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 18, 2023, 8:19am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/4 "2023-07-18T08:19:27Z")
This works in grokconstructor for all three cases: `%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[access][user][identity]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HTTPDATE:timestamp}\] (%{HTTPDUSER:[user][name]})? \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int}bytes) \"(?:-|%{IPORHOST:[destination][address]})\" \"%{DATA:session}\" \[%{DATA:something}\] (%{INT:time_taken:int})ms` Pls review column names. ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 18, 2023, 8:58am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/5 "2023-07-18T08:58:46Z")
thanks for the response. i tried to follow the grok pattern to match my all 3 types of logs in my pipeline filter but its giving an failure as below ```auto error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'Tis0aIkBqDh-xxxxxxx'. Preview of field's value: '{name=x01xxxxxxxxxx7a.vsi.uat.abc.com}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:327"}}}} ``` also below is my logstash filter in pipeline. ```auto if [application] in ["CXXX-ENT-X-S"] { grok { match => { "messag" => '%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[access][user][identity]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HTTPDATE:timestamp}\] (%{HTTPDUSER:[user][name]})? \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int}bytes) \"(?:-|%{IPORHOST:[destination][address]})\" \"%{DATA:session}\" \[%{DATA:something}\] (%{INT:time_taken:int})ms' } } } ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 18, 2023, 9:17am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/6 "2023-07-18T09:17:57Z")
> [@shailendra1](#): > > `"type"=>"mapper_parsing_exception", ` > "reason"=\>"Can't get text on a START\_OBJECT at 1:327" Kibana data view is not OK. Try with a new index name or delete and create a new data pattern. ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 18, 2023, 10:16am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/7 "2023-07-18T10:16:28Z")
i have changed the data view in the kibana itself but its giving me the same errors as below - ````auto [2023-07-18T18:11:07,350][WARN][logstash.outputs.elasticsearch][csec][failed] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"csec_logstash_fail_events-2023.07", :routing=>nil}, {"tags"=>["_jsonparsefailure"], "input"=>{"type"=>"filestream"}, "message"=>"10.1.1.10 - - [12/Jul/2023:08:03:54 +0800] - \"GET /isalive HTTP/1.1\" 200 15bytes \"-\" \"ZK3t6h_wNPwl3QRmAxxxxxxx\" [-] 0ms", "ecs"=>{"version"=>"8.0.0"}, "host"=>{"name"=>"x01xxxxx7a.vsi.uat.abc.com"}, "metadata"=>{"timezone"=>"Asia/APAC", "application"=>"CSEC-ENT-HD-ACC"}, "@timestamp"=>2023-07-18T10:11:00.143Z, "@version"=>"1", "topic"=>"testtopic", "agent"=>{"ephemeral_id"=>"fad60b7a-286f-4e4c-b2da-d9c75fb5077c", "id"=>"bb93e0a2-e822-4033-aa8a-d905cdxxxxxx", "name"=>"x01xxxxxx7a.vsi.uat.abc.com", "version"=>"8.5.3", "type"=>"filebeat"}, "log"=>{"file"=>{"path"=>"/tmp/csec/am-logs/access_log1.log"}, "offset"=>13416}}], response: {"index"=>{"_index"=>"csec_logstash_fail_events-2023.07", "_id"=>"H5l7aIkBDQg7IgHxxxxx", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'H5l7aIkBDQg7IgHlbzPM'. Preview of field's value: '{name=x01xxxxx7a.vsi.uat.abc.com}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:238"}}}}``` ```` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 18, 2023, 12:12pm UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/8 "2023-07-18T12:12:35Z")
Grok is OK: ```auto input { generator { message => ['10.1.1.10 - - [12/Jul/2023:08:03:54 +0800] - "GET /isalive HTTP/1.1" 200 15bytes "-" "ZK3t6h_wNPwl3QRmAxxxxxxx" [-] 0ms' ] count => 1 } } filter { grok { match => { "message" => '%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[access][user][identity]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HTTPDATE:timestamp}\] (%{HTTPDUSER:[user][name]})? \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int}bytes) \"(?:-|%{IPORHOST:[destination][address]})\" \"%{DATA:session}\" \[%{DATA:something}\] (%{INT:time_taken:int})ms' } id => "grok" } } output { stdout { } } ``` You have error: - "tags"=\>["\_jsonparsefailure"] - Do you have a json conversion? - "error"=\>{"type"=\>"mapper\_parsing\_exception", "reason"=\>"**failed to parse field [host] of type [text] in document with id 'H5l7aIkBDQg7IgHlbzPM**'. Preview of field's value: '{[name=x01xxxxx7a.vsi.uat.abc.com](http://name=x01xxxxx7a.vsi.uat.abc.com)}'", "caused\_by"=\>{"type"=\>"illegal\_state\_exception", "reason"=\>"Can't get text on a START\_OBJECT at 1:238"}}}}``` You have an issue with the host field, either you are inserting host as JSON in text field either you are trying to do something with JSON transformation. Result: ```auto { "something" => "-", "timestamp" => "12/Jul/2023:08:03:54 +0800", "message" => "10.1.1.10 - - [12/Jul/2023:08:03:54 +0800] - \"GET /isalive HTTP/1.1\" 200 15bytes \"-\" \"ZK3t6h_wNPwl3QRmAxxxxxxx\" [-] 0ms", "session" => "ZK3t6h_wNPwl3QRmAxxxxxxx", "user" => { "name" => "-" }, "url" => { "original" => "/isalive" }, "http" => { "request" => { "method" => "GET" }, "response" => { "status_code" => 200, "body" => { "bytes" => 15 } }, "version" => "1.1" }, "time_taken" => 0, "@version" => "1", "@timestamp" => 2023-07-18T12:08:42.630534700Z, "source" => { "address" => "10.1.1.10" } } ``` And change the something field according to column names for access\_log. ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 19, 2023, 2:12am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/9 "2023-07-19T02:12:21Z")
ahh, understood . i found that my index have the host entry field already which is having type text. ```auto "host": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 ``` here the type is different, for particularly these logs samples. is there any other ways we can update the values in the host type in indices or any other better options. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 19, 2023, 2:48am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/10 "2023-07-19T02:48:42Z")
[This thread](https://discuss.elastic.co/t/getting-illegal-state-exception-error-while-pushing-logs-to-elasticsearch/290029/2) expands upon the problem. You cannot change the type of a field in an existing index without re-indexing. And all the events need to be consistent about what type a field is. ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 19, 2023, 8:55am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/11 "2023-07-19T08:55:04Z")
Thank you , Badger and Rios for your response. yeah, that make sense to me now. i have changed indices for another test of my pipelines, during test, its giving me parsing error code 46 for some transactions. below are two samples error in my logstash logs - ```auto [2023-07-19T16:35:15,488][WARN][logstash.filters.json][csec][312699f9f6dc8aebf11f8827008e30d3c3f562469142d2f278d4db7bd72177cb] Error parsing json {:source=>"message", :raw=>"10.10.20.10 - - [12/Jul/2023:08:00:34 +0800] - \"GET /isalive HTTP/1.1\" 200 15bytes \"-\" \"ZK3tIngceATC133RawX14xxxxx\" [-] 0ms", :exception=>#} [2023-07-19T16:35:15,488][WARN][logstash.filters.json][csec][312699f9f6dc8aebf11f8827008e30d3c3f562469142d2f278d4db7bd72177cb] Error parsing json {:source=>"message", :raw=>"10.52.245.67 - - [12/Jul/2023:08:00:37 +0800] appbau2a-ig \"GET /am/oauth2/.well-known/openid-configuration HTTP/1.1\" 200 3897bytes \"10.10.10.10\" \"ZK3tJSGP-VpKu4Cf5HWxxxxxxx\" [-] 37ms", :exception=>#} ``` ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 20, 2023, 8:48am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/12 "2023-07-20T08:48:18Z")
it also seems that i have two type of logs format - .json and .log so wherever the json format is there its getting parsed correctly but if its .log format its not parsing all filed from my 3 type of log samples. is there any other better ways to filter the data into my pipeline. thanks in advance . ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 21, 2023, 10:41am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/13 "2023-07-21T10:41:21Z")
@Rios , may you help me for the grok pattern. it seems all messages are going into one liners only. i am testing my .log pipelines for httpd access logs ```auto "message" => "{\"@timestamp\":\"2023-07-21T10:34:01.270Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.5.3\"},\"ecs\":{\"version\":\"8.0.0\"},\"host\":{\"name\":\"x01stsxxxx7a.vsi.uat.abc.com\"},\"agent\":{\"version\":\"8.5.3\",\"ephemeral_id\":\"76c0d20b-8087-4585-b625-70044a945678z4\",\"id\":\"ef7c7f39-97c9-4945-a331-16999999\",\"name\":\"x01stsxxxx7a.vsi.uat.abc.com\",\"type\":\"filebeat\"},\"log\":{\"offset\":2846928,\"file\":{\"path\":\"/tmp/csec/ds/ldap-access.audit1.json\"}},\"message\":\"{\\\"eventName\\\":\\\"DJ-LDAP\\\",\\\"client\\\":{\\\"ip\\\":\\\"10.52.64.166\\\",\\\"port\\\":43120},\\\"server\\\":{\\\"ip\\\":\\\"10.52.64.177\\\",\\\"port\\\":1636},\\\"request\\\":{\\\"protocol\\\":\\\"LDAPS\\\",\\\"operation\\\":\\\"SEARCH\\\",\\\"connId\\\":10,\\\"msgId\\\":41781,\\\"dn\\\":\\\"\\\",\\\"scope\\\":\\\"base\\\",\\\"filter\\\":\\\"(objectClass=*)\\\",\\\"attrs\\\":[\\\"1.1\\\"]},\\\"transactionId\\\":\\\"9109e171-b875-4f61-ab88-5984e8d952f6-5504997\\\",\\\"response\\\":{\\\"status\\\":\\\"SUCCESSFUL\\\",\\\"statusCode\\\":\\\"0\\\",\\\"elapsedTime\\\":0,\\\"elapsedTimeUnits\\\":\\\"MILLISECONDS\\\",\\\"nentries\\\":1},\\\"userId\\\":\\\"uid=am-identity-bind-account,ou=admins,ou=identities,dc=cybersecure,dc=abc,dc=com\\\",\\\"timestamp\\\":\\\"2023-07-12T05:52:26.168Z\\\",\\\"_id\\\":\\\"9109e171-b875-4f61-ab88-5984e8d952f6-5504999\\\"}\",\"metadata\":{\"application\":\"CSEC-DS-LDP-ACC\",\"timezone\":\"Asia/Asia\"},\"topic\":\"testtopic2006\",\"input\":{\"type\":\"filestream\"}}", ``` seems its not parsing all fields separately. ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [July 22, 2023, 12:58am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/14 "2023-07-22T00:58:50Z")
Grok is OK, however somewhere in .conf you are adding message from grok and other fields to the message field. Review your .conf, if is not clear, go step by step. The message filed inside the message field. ```auto { "eventName": "DJ-LDAP", "client": { "ip": "10.52.64.166", "port": 43120 }, "server": { "ip": "10.52.64.177", "port": 1636 }, "request": { "protocol": "LDAPS", "operation": "SEARCH", "connId": 10, "msgId": 41781, "dn": "", "scope": "base", "filter": "(objectClass=*)", "attrs": [ "1.1" ] }, "transactionId": "9109e171-b875-4f61-ab88-5984e8d952f6-5504997", "response": { "status": "SUCCESSFUL", "statusCode": "0", "elapsedTime": 0, "elapsedTimeUnits": "MILLISECONDS", "nentries": 1 }, "userId": "uid=am-identity-bind-account,ou=admins,ou=identities,dc=cybersecure,dc=abc,dc=com", "timestamp": "2023-07-12T05:52:26.168Z", "_id": "9109e171-b875-4f61-ab88-5984e8d952f6-5504999" } ``` ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 24, 2023, 3:35am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/15 "2023-07-24T03:35:09Z")
here is my .conf file attached. ```auto ![image|690x446](upload://qYR2ibiQyG2p5R4rQEJnHaOUXtY.png) ``` the problem is that i am seeking all my parsing coming into one block under the message fields. ```auto { "message" => "{\"@timestamp\":\"2023-07-24T03:21:27.404Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"8.5.3\"},\"host\":{\"name\":\"x01stxxxxxxx7a.vsi.uat.abc.com\"},\"agent\":{\"version\":\"8.5.3\",\"ephemeral_id\":\"b09569a6-f161-4b7c-8ffb-2d57f7xxxxx5\",\"id\":\"e5f672fc-b02d-4725-bf1f-22e20697436e\",\"name\":\"x01stxxxxxxx7a.vsi.uat.abc.com\",\"type\":\"filebeat\"},\"message\":\"{\\\"eventName\\\":\\\"DJ-LDAP\\\",\\\"client\\\":{\\\"ip\\\":\\\"10.52.64.17\\\",\\\"port\\\":38834},\\\"server\\\":{\\\"ip\\\":\\\"10.52.64.177\\\",\\\"port\\\":1636},\\\"request\\\":{\\\"protocol\\\":\\\"LDAPS\\\",\\\"operation\\\":\\\"SEARCH\\\",\\\"connId\\\":17324,\\\"msgId\\\":18237,\\\"dn\\\":\\\"\\\",\\\"scope\\\":\\\"base\\\",\\\"filter\\\":\\\"(objectClass=*)\\\",\\\"attrs\\\":[\\\"1.1\\\"]},\\\"transactionId\\\":\\\"9109e171-b875-4f61-ab88-5984e8d952f6-5499843\\\",\\\"response\\\":{\\\"status\\\":\\\"SUCCESSFUL\\\",\\\"statusCode\\\":\\\"0\\\",\\\"elapsedTime\\\":0,\\\"elapsedTimeUnits\\\":\\\"MILLISECONDS\\\",\\\"nentries\\\":1},\\\"userId\\\":\\\"uid=am-identity-bind-account,ou=admins,ou=identities,dc=cybersecure,dc=abc,dc=com\\\",\\\"timestamp\\\":\\\"2023-07-12T05:47:43.274Z\\\",\\\"_id\\\":\\\"9109e171-b875-4f61-ab88-5984e8d952f6-5499847\\\"}\",\"log\":{\"offset\":1885305,\"file\":{\"path\":\"/tmp/csec/ds/ldap-access.audit1.json\"}},\"input\":{\"type\":\"filestream\"},\"metadata\":{\"application\":\"CSEC-DS-LDP-ACC\",\"timezone\":\"Asia/Asia\"},\"topic\":\"testtopic2006\",\"ecs\":{\"version\":\"8.0.0\"}}", "@timestamp" => 2023-07-24T03:21:27.971946364Z, "@version" => "1" ``` ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 25, 2023, 1:40am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/16 "2023-07-25T01:40:53Z")
i think image was not uploaded clearly of my logstash config . i am sharing it with filter which i am using ```auto filter { ## Another application filtering if [application] in ["CSEC-ENT-HD-ACC"] { ruby { code => 'host = event.get("host") if host.is_a? Hash event.set("host", host.to_s) end' } grok { match => { "message" => '%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[access][user][identity]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HTTPDATE:timestamp}\] (%{HTTPDUSER:[user][name]})? \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int}bytes) \"(?:-|%{IPORHOST:[destination][address]})\" \"%{DATA:session}\" \[%{DATA:extra}\] (%{INT:time_taken:int})ms' } } } } output { stdout { codec => rubydebug } if "_jsonparsefailure" in [tags] { file { path => "/logs/mozu/logstash/events/csec_failed_events-%{+YYYY-MM-dd}.json" } elasticsearch { user => "logstash_output" password => "${LOGSTASH_OUTPUT_PWD}" id => "failed" hosts => ["https://x01sxxxx1a.vsi.uat.abc.com:9210", "https://x01sxxxx2a.vsi.uat.abc.com:9210", "https://x01sxxxxx3a.vsi.uat.abc.com:9210"] index => "csec_failed_events-%{+YYYY.MM}" ``` ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 31, 2023, 3:46am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/17 "2023-07-31T03:46:41Z")
Hello @Badger @Rios , mates. do you have any suggestions for me. Thanks in advance. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 31, 2023, 4:04am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/18 "2023-07-31T04:04:16Z")
In one place you say there are two types of logs, in another you suggest there are three. Whichever it is, please provide examples of each type. Then we can try to reproduce the issue. Also, what is the actual error you are trying to fix? It seems you understand the mapping exception, and also the json filter error parsing code 46, so what is your current question? ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [July 31, 2023, 4:37am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/19 "2023-07-31T04:37:21Z")
thank you for response. I am just newbie though in this ELK world. So, basically, i am creating a one pipeline where i am trying to capture the different type of logs in it. ie. httpd, OS, app logs which have different formats say .json and .log . so, in pipeline using codec=json is parsing but for the httpd logs parsing is happing using grok but its all-parsed data are going in message field only. i am not sure if its good idea to use one pipeline for different logs under one application (different logs) or is there any other better approach can be useful. ---
**Author:** ![shailendra1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shailendra1/32/122783_2.png) [@shailendra1](https://discuss.elastic.co/u/shailendra1)\ **Post date:** [August 7, 2023, 8:40am UTC](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535/20 "2023-08-07T08:40:27Z")
i think i fixed for a type of logs but my grok pattern is getting failed in the below logs formats since i have custom logs in my beats ```auto 10.52.245.67 - - [12/Jul/2023:08:08:52 +0800] uibau1a "GET /login/assets/fonts/OpenSans/OpenSans-SemiBold.woff HTTP/1.1" 200 18696bytes "10.168.224.18, 10.52.80.51" "ZK3vFHgceATC133RawX2NgAAAJE" [-] 0ms 10.52.245.67 - - [12/Jul/2023:08:08:59 +0800] app2a-bau "POST /employee HTTP/1.1" 200 215bytes "10.168.224.18, 10.52.80.51" "ZK3vGyGP-VpKu4Cf5HWLUwAAAMI" [-] 708ms ``` here with my logstash.conf which is giving me grokparsefailure for these logs., ```auto filter { #cleaning up the data # json { source => "[@metadata][body]" remove_field => ["message"] } ruby { code => ' if event event_hash = event.to_hash if event_hash.kind_of?(Hash) and event_hash != {} event_hash.each { |k, v| if v == "-" event.set(k, nil) end } end end' } ruby { code => 'event.set("@timestamp_logstash", event.get("@timestamp"))' } json { source => "message" } # CSEC Filtering the data grok { match => { "message" => "%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[access][user][identity]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HTTPDATE:eventtime}\] (%{HTTPDUSER:[user][name]})? \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int}bytes) \"(?:-|%{IPORHOST:[destination][address]})\" \"%{DATA:session}\" \[%{DATA:extra}\] (%{INT:time_taken:int})ms" } } ## date conversion ## date { match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "dd/MMM/yyyy:HH:mm:ss ZZZZ", "yyyy-MM-dd HH:mm:ss", "ISO8601"] timezone => "Asia/Singapore" #target => "parsed_times" } } output { stdout { codec => rubydebug } } ``` pl suggest if I am missing any format or i can use any other grok patterns for these logs. [Next page](https://discuss.elastic.co/t/combined-grok-pattern-for-customized-logs/338535.md?page=2)