# How to extract string from the log and create a new field and send to elastic search index **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 23, 2023, 8:03pm UTC](https://discuss.elastic.co/t/how-to-extract-string-from-the-log-and-create-a-new-field-and-send-to-elastic-search-index/336797 "2023-06-23T20:03:53Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![mbsarathchandra](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mbsarathchandra](https://discuss.elastic.co/u/mbsarathchandra)\ **Post date:** [June 23, 2023, 8:03pm UTC](https://discuss.elastic.co/t/how-to-extract-string-from-the-log-and-create-a-new-field-and-send-to-elastic-search-index/336797/1 "2023-06-23T20:03:53Z")
Hello Everyone, I am currently using Elastic Search Version 8.8.1 installed on RHEL os. Filebeat Version: 8.6.1 The logs are read from the Application server and pushed to Elasticsearch index using filebeat. Data structure in each log file: [2023-06-22 06:18:26.6566218] [24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms] The following fields are read and added to the payload: Timestamp LogthreadID LogLevel ClassName LogMessage Filebeat Input Type is Set as Logs Dissector from Filebeat.yml processors: # Following pattern dissects the message into fields and add it to the main payload. - dissect: tokenizer: '[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] %{LogMessage}' field: "message" target\_prefix: "" ignore\_failure: true trim\_values: left # Following pattern replaces any with a empty string - replace: fields: - field: "LogMessage" pattern: '[' replacement: "" - field: "LogMessage" pattern: ']' replacement: "" - field: "LogMessage" pattern: '-' replacement: "" ignore\_missing: true fail\_on\_error: false # Following pattern converts the log creation timestamp from text to datetime datatype - timestamp: field: TimeStamp timezone: America/Chicago ignore\_missing: true ignore\_failure: true layouts: - '2006-01-02T15:04:05Z' - '2006-01-02T15:04:05.999' - '2006-01-02T15:04:05.999Z' - '2006-01-02T15:04:05.000000' - '2006-01-02 15:04:05.0000000' test: - '2023-02-18T16:23:24.00Z' - '2023-02-18T16:23:24.000' - '2023-02-18T16:23:24.000Z' - '2023-02-18T16:23:24.000563' - '2023-02-18 16:23:24.0005633' Issue: I would like to apply a regex or any string function on the field LogMessage and look for a specific format , extract the string and create a new field and add it to the payload. Ex: Log Message: Finished OracleDBHelper.ExecuteQuery in 47 ms Extract String: OracleDBHelper.ExecuteQuery Create a New Field: LogMethod Add the string to the new field. If the regex fails, I want to add a default string as "None" What processor can be used for string extractions from a existing field using regex. Please let me know. Any help or suggestions greatly appreciated. Thanks ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [June 23, 2023, 10:46pm UTC](https://discuss.elastic.co/t/how-to-extract-string-from-the-log-and-create-a-new-field-and-send-to-elastic-search-index/336797/2 "2023-06-23T22:46:21Z")
You can use something like this: ```auto input { generator { message => "[2023-06-12 06:18:26.6566218] [24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]" count => 1 } } # input filter { dissect { mapping => { "message" => "[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] - [%{LogMessage}]%{+LogMessage}]" } } grok { match => { "LogMessage" => "%{WORD:status}%{SPACE}\[%{DATA:method}\]%{DATA}%{POSINT:exectime}%{SPACE}%{WORD:unit}" } } mutate { strip => ["ThreadId"] } date { match => ["TimeStamp", "yyyy-MM-dd HH:mm:ss.SSSSSSS"] timezone => "America/Chicago" target=> "TimeStamp" } } output { stdout { codec => rubydebug{}} } ``` Result ```auto { "message" => "[2023-06-12 06:18:26.6566218] [24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]", "unit" => "ms", "@timestamp" => 2023-06-23T22:44:08.412404800Z, "LogLevel" => "DEBUG", "TimeStamp" => 2023-06-12T11:18:26.656Z, "ThreadId" => "24", "LogMessage" => "Finished [OracleDBHelper.ExecuteQuery] in 10 ms", "exectime" => "10", "ClassName" => "CNP.AMS.Common.Utility.LogScope", "method" => "OracleDBHelper.ExecuteQuery", "status" => "Finished" } ``` ---
**Author:** ![mbsarathchandra](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mbsarathchandra](https://discuss.elastic.co/u/mbsarathchandra)\ **Post date:** [June 25, 2023, 1:02am UTC](https://discuss.elastic.co/t/how-to-extract-string-from-the-log-and-create-a-new-field-and-send-to-elastic-search-index/336797/3 "2023-06-25T01:02:46Z")
> [@Rios](#): > > `OracleDBHelper.ExecuteQuery"` Hi Rios, thanks for the update. I will try the Grok Processor on the Filebeat config and see if I could extract the string from the log message based on the inputs you have given. Thanks once again, really appreciate your help. Sarath ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 23, 2023, 3:02am UTC](https://discuss.elastic.co/t/how-to-extract-string-from-the-log-and-create-a-new-field-and-send-to-elastic-search-index/336797/4 "2023-07-23T03:02:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.