Hi,
Sample Log Data
com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\TBX-Dept-Quality-SU has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.242 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727304], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[TBX-ADMIN, corp.intusurg.com, SITE_ADMIN]]
2018-06-11 17:00:50.246 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\TBX-ADMIN has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.249 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727305], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[All_VPs_globally, corp.intusurg.com, VIEWER]]
2018-06-11 17:00:50.251 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\All_VPs_globally has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.253 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727306], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[TBX-Dept-Quality-EU, corp.intusurg.com, INTERACTOR]]
2018-06-11 17:00:50.256 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\TBX-Dept-Quality-EU has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.259 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727307], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[BI DevOps Team, corp.intusurg.com, INTERACTOR]]
2018-06-11 17:00:50.261 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\BI DevOps Team has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.264 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727308], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[TBX-DEPT-CRM-Contract-SU, corp.intusurg.com, PUBLISHER]]
2018-06-11 17:00:50.266 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\TBX-DEPT-CRM-Contract-SU has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.268 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727309], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[Revenue, corp.intusurg.com, PUBLISHER]]
2018-06-11 17:00:50.271 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\Revenue has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
2018-06-11 17:00:50.274 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.service.BackgroundJobService - Queued job: id[4727310], type[SyncActiveDirectoryGroup], site[Subscriptionsite], args[[sngsqldba, corp.intusurg.com, PUBLISHER]]
2018-06-11 17:00:50.276 -0700 (,,,,4727170,:enqueue_ad_groups_sync,-) pool-21-thread-1 : INFO com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\sngsqldba has been scheduled. Site id=[3], schedule_name= [internal_ad_sync_schedule]
I want to fetch fields like, nomenclature can vary:
(i) Environment details like UAT, Default
(ii) Ticket No. like 4727180
(iii) Status: success or failure
(iv) Time: total time taken to process
(v) Process Name
I am using file beat in input
input
{
beats {
port => 5042
codec => multiline
{
pattern => ",2.0"
what => "previous"
negate =>"true"
charset => "ISO-8859-1"
}
}
}
filter
{if "security" in [type]{
grok {
match => {"message" => ",(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),
(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]),
(?[^,]),(?[^,]),(?[^,]),(?[^,]),(?[^,]*),"}
}
mutate
{
,remove_tag => ["multiline"],
remove_field => ["m1","m4","m6","m2","m5","m22"]
strip => ["DateTime"]
}
date {
match => [ "DateTime", "YYYY MM dd HH:mm:ss:SSS" ]
timezone => "EST"
target => "DateTime"
}
}}
output {
if "security" in [type]{
elasticsearch {
hosts => ["10.103.23.224"]
index => "test"
}
},file{
, path=>"D:\systeminfolog2.txt"
,}
}
Please suggest me correct pattern to read log file.
input
{
file {
path => "D:\Intitutive\logfile\backgrounder-0.log"
}
}
filter
{
grok {match => {"message" => "(?<DateTime>.*?) \(\,\,\,\,(?<TicketNumber>%{INT}.*?)\,\:(?<TicketDetails>.*?)\,\-\) (?<ThreadDetails>[a-zA-Z0-9._-]+) \: (?<DebugLevel>%{WORD}) (?<Data>.*)"}}
}
output {
elasticsearch { hosts => ["10.103.23.224"] index => "nine" }file{
path=>"D:\systeminfolog9.txt"
}
}
I have used above pattern but it wont work .I can see my logstash stared successfully but neither my index created nor file copied in to specified path.