Why my logstash aggregate sometimes works, sometimes not?

maoxuguang · January 3, 2022, 10:57am

for example, sometimes I can get the tag [ "fordebug_%{job_name}" ], it indicates that the aggregate executed successfully, sometimes I can not get this tag. why? I struggled for days but no finding. I am exhausted. could you help me?

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => [ 
                 "Started by user %{GREEDYDATA:garbled}\[0m%{GREEDYDATA:trigger_by}",
                 "Running on %{GREEDYDATA:not_need}workspace/%{WORD:job_name}",
                 "Finished: %{WORD:job_result}" 
               ]
    }
  }
  
  if [job_name] {
    mutate {
      add_field => {"debug_job_name" => "%{job_name}_%{[log][file][path]}_'test'"} //for debug
    }
    ruby {
      code => "puts 'mxg'+event.get('[log][file][path]')" //for debug
    }
    aggregate {
      task_id => "%{[log][file][path]}"
      code => "puts 'hahaha'
               map['job_name'] = event.get('job_name')"
      map_action => "create"
      add_tag => [ "fordebug_%{job_name}" ]
    }
  }


  if [trigger_by] {
    mutate {
      add_field => {"debug_trigger_by" => "%{trigger_by}_%{[log][file][path]}_'test'"}
    }
    ruby {
      code => "puts 'mxg22'+event.get('[log][file][path]')"
    }

    aggregate {
      task_id => "%{[log][file][path]}"
      code => "puts 'hahaha2'
               map['trigger_by'] = event.get('trigger_by')"
      map_action => "create"
      add_tag => [ "fordebug_%{trigger_by}" ]
    }
  }
  if [job_result]{
    aggregate {
      task_id => "%{[log][file][path]}"
      code => "event.set('job_name', map['job_name'])
               event.set('trigger_by', map['trigger_by'])
               event.set('result', event.get('job_result'))"
      end_of_task => true
      add_tag => [ "fordebug_%{trigger_by}_%{job_name}" ]
    
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://x.x.x.x:9200"]
    index => "jenkins-test-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
  stdout { codec => rubydebug }
}

Badger · January 3, 2022, 5:19pm

Exactly, so the add_tag does not occur if

The task_id is nil or none of the sprintf substitutions occur
An aggregate exception occurs
map_action is "create" and the map already exists
map_action is "update" and the map does not exist

maoxuguang · January 5, 2022, 6:41am

Thanks for your help. when I delete following line

map_action => "create"

or replace it with:

map_action => "create_or_update"

it works. The debug tag was created. But I still have two questions:

every job log only have one line match "Running on %{GREEDYDATA:not_need}workspace/%{WORD:job_name}", The debug info code => "puts 'mxg'+event.get('[log][file][path]')" only occurs 1 or 0 times for one job, why map_action => "create is not ok?
even the aggregate for job_name and trigger_by were ok, sometimes event.set('job_name', map['job_name']) was not ok, i did not get field job_name, why? most of times it is ok. sometimes it went wrong, usually after I restart logstash. The two debug jobs only run less than 1 minute, so I think it is not timeout issue.

Badger · January 5, 2022, 4:56pm

Without seeing all of the input data I cannot suggest what the problem might be.

maoxuguang · January 6, 2022, 2:49am

In filebeat I only configured two jobs for debug. one is test_for_elk_trigger, here is the log of it:

Started by user ^[[8mha:////4H7fyrm5bxQG052fIU6vSOnHcHxgpop3ONd2SfKAwxJZAAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyFEgZu/dLi1CL9itLcxHwADV5VrMAAAAA=^[[0mMao, Xuguang (XXX - XX/XXXXXXX)
Running in Durability level: MAX_SURVIVABILITY
^[[8mha:////4B2qx+efg7CWwFWY2xTHrGxyLnOH0bH3HnBvrif7ncNcAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==^[[0m[Pipeline] Start of Pipeline
^[[8mha:////4EssAPmau1BUq9ByzFAZZzH47erpvSDyTfjBqQMTh8YoAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==^[[0m[Pipeline] node
Running on ^[[8mha:////4A0sqggNTVzkWT0y6yuhvFX4jtPzhql6rV2A2sGTQYGUAAAApB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzeEgYJ/eT83ILSktQi/Yzy4uQS3eKiXDNdAxN9APLsa0/NAAAA^[[0mhwsct-srm6-04 in /home/ca_RFSW_C_CH/jenkinsSlave/workspace/test_for_elk_trigger
^[[8mha:////4IuW672shnIJoQTMHtwuqjLOjDvTOMiBQCPqkldloO3tAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==^[[0m[Pipeline] {
^[[8mha:////4BBGFx4A8yiARlRQXYuu9hKy3riSRGdrscpLYICsJh4cAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==^[[0m[Pipeline] stage
^[[8mha:////4NHJRbOFAPOMOom5h7ZApqYfYQm33BmhK698O4jmCeDmAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==^[[0m[Pipeline] { (test)
^[[8mha:////4PCfXYgwv0nkBqeBy6X5JF6b+MU4Pw6oTL4taplCZDCfAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gwExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jcChmMxwAAAAA==^[[0m[Pipeline] script
^[[8mha:////4MthFXvFIUxd96C7TMbeiGo/JQF18iCLiM9zfXpB7LEPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0BAkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfpX/cvwAAAAA==^[[0m[Pipeline] {
^[[8mha:////4BP9oMp7JOOd0xDXMimgIVdJnK3VcVA4/Ff6MuM8c85QAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaoyVLaHxBMgiwpL/WWDdrTyRV/MOEjexcpJJ5k3zXm9Y5wQnSpZ7g73DrB2PYbR18YlSfws0cU9XrgkzBcOlmSR15rygpGJgCWtgJWBjUAfKDm2BrfDqodqg0LaXkup3FNC4boAnMFHNRaXyhTnB7j52mfDn+SuAORZg+9pD/AAeoCAqvAAAAA==^[[0m[Pipeline] build
Scheduling project: ^[[8mha:////4NREmu5cFdAPB2TupCS990Rmjky+NGncluyTrld6JBq1AAAAnR+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAweoBn6WflJ+iWpxSXxaflF8ak52foAXzXXUccAAAA=^[[0mtest_for_elk
Starting building: ^[[8mha:////4F6AUKk3xwL27saE3PoVlWGjmay8PF5OlymnJ9ngS1h0AAAAoh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzBEgYx/az8JP2S1OKS+LT8ovjUnGx9QzMjfQD5XeFeywAAAA==^[[0mtest_for_elk #162
^[[8mha:////4CMtnfiiK59q8syMON3TLxItNS0BZ/prH08G8PuvJc0FAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMKBDglR0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8Brmfz270AAAA=^[[0m[Pipeline] }
^[[8mha:////4HYmobU6fhUpojOBPZKLgYHVJZHryQmZrJoEksxXLVwfAAAAox+LCAAAAAAAAP9tjTESgjAQRT84FraWHiIMlYVjZZuh8QQRYgxkdjFZhMoTeTXvICMzVv7qv9e81xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0gL0tBtv8Aj1Msdr0AAAA=^[[0m[Pipeline] // script
^[[8mha:////4I76xG7OMtD44SWf+OG8r1xfT2ylYWON5eTUr9g6/tfJAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPyRmLRr0AAAA=^[[0m[Pipeline] }
^[[8mha:////4JlmLZ82OCmblWGyCtovpwbJQyWaGG/cUjUFgUgz0imIAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPw2Hi4r0AAAA=^[[0m[Pipeline] // stage
^[[8mha:////4BZhF5r/fZ3+wng2wqlxUDaA1STMWHdDjAdudt8P6hUWAAAAox+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6LEhM04uTYsfkGFWgvNe9g+hMkv8tf8B4kkTt7pnrOc1xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0g35eCrPwARY3FJ70AAAA=^[[0m[Pipeline] }
^[[8mha:////4EAdGTjRtNZAgBL+bYNzRV57pfYXzN9+s9X0FDwXS8NXAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPVjHBhL0AAAA=^[[0m[Pipeline] // node
^[[8mha:////4KVEe2i9DY/inRkY0XKEXobRaEdoc3qSwfbx2je+8Ed3AAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPEHtmtL0AAAA=^[[0m[Pipeline] End of Pipeline
Finished: SUCCESS

Another one is test_for_elk, here is the log of it:

Started by upstream project "^[[8mha:////4N1/xOgK+T9Td6VjTwOneTFBtt5x7FQnYNboln669TAiAAAApR+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyREgYp/az8JP2S1OKS+LT8ovjUnOx4oEnp6alF+gC9Ar1qzwAAAA==^[[0mtest_for_elk_trigger" build number ^[[8mha:////4MVgz9XBjuf0VaMoP0Y8fx0xQIo17kGFPq850pVwAYbeAAAApx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAymEgYZ/az8JP2S1OKS+LT8ovjUnOx4oEnp6alF+pbGAHiMn4rRAAAA^[[0m93
originally caused by:
Started by user ^[[8mha:////4H7fyrm5bxQG052fIU6vSOnHcHxgpop3ONd2SfKAwxJZAAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyFEgZu/dLi1CL9itLcxHwADV5VrMAAAAA=^[[0mMao, Xuguang (XXX - XX/XXXXXX)
Running in Durability level: MAX_SURVIVABILITY
^[[8mha:////4B2qx+efg7CWwFWY2xTHrGxyLnOH0bH3HnBvrif7ncNcAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==^[[0m[Pipeline] Start of Pipeline
^[[8mha:////4EssAPmau1BUq9ByzFAZZzH47erpvSDyTfjBqQMTh8YoAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==^[[0m[Pipeline] node
Running on ^[[8mha:////4GbBzFxigEI/sveYQD+IWZopfD0PTIEYSFDRvdIjb/xwAAAApB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzeEgYJ/eT83ILSktQi/Yzy4uQS3eKiXDNdA0t9AL+SxfrNAAAA^[[0mhwsct-srm6-09 in /home/ca_RFSW_C_CH/jenkinsSlave/workspace/test_for_elk
^[[8mha:////4IuW672shnIJoQTMHtwuqjLOjDvTOMiBQCPqkldloO3tAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==^[[0m[Pipeline] {
^[[8mha:////4BBGFx4A8yiARlRQXYuu9hKy3riSRGdrscpLYICsJh4cAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==^[[0m[Pipeline] stage
^[[8mha:////4NHJRbOFAPOMOom5h7ZApqYfYQm33BmhK698O4jmCeDmAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==^[[0m[Pipeline] { (test)
^[[8mha:////4PCfXYgwv0nkBqeBy6X5JF6b+MU4Pw6oTL4taplCZDCfAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gwExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jcChmMxwAAAAA==^[[0m[Pipeline] script
^[[8mha:////4MthFXvFIUxd96C7TMbeiGo/JQF18iCLiM9zfXpB7LEPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0BAkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfpX/cvwAAAAA==^[[0m[Pipeline] {
^[[8mha:////4BP9oMp7JOOd0xDXMimgIVdJnK3VcVA4/Ff6MuM8c85QAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaoyVLaHxBMgiwpL/WWDdrTyRV/MOEjexcpJJ5k3zXm9Y5wQnSpZ7g73DrB2PYbR18YlSfws0cU9XrgkzBcOlmSR15rygpGJgCWtgJWBjUAfKDm2BrfDqodqg0LaXkup3FNC4boAnMFHNRaXyhTnB7j52mfDn+SuAORZg+9pD/AAeoCAqvAAAAA==^[[0m[Pipeline] echo
just for elk test aaaaaa
^[[8mha:////4CMtnfiiK59q8syMON3TLxItNS0BZ/prH08G8PuvJc0FAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMKBDglR0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8Brmfz270AAAA=^[[0m[Pipeline] }
^[[8mha:////4HYmobU6fhUpojOBPZKLgYHVJZHryQmZrJoEksxXLVwfAAAAox+LCAAAAAAAAP9tjTESgjAQRT84FraWHiIMlYVjZZuh8QQRYgxkdjFZhMoTeTXvICMzVv7qv9e81xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0gL0tBtv8Aj1Msdr0AAAA=^[[0m[Pipeline] // script
^[[8mha:////4I76xG7OMtD44SWf+OG8r1xfT2ylYWON5eTUr9g6/tfJAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPyRmLRr0AAAA=^[[0m[Pipeline] }
^[[8mha:////4JlmLZ82OCmblWGyCtovpwbJQyWaGG/cUjUFgUgz0imIAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPw2Hi4r0AAAA=^[[0m[Pipeline] // stage
^[[8mha:////4BZhF5r/fZ3+wng2wqlxUDaA1STMWHdDjAdudt8P6hUWAAAAox+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6LEhM04uTYsfkGFWgvNe9g+hMkv8tf8B4kkTt7pnrOc1xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0g35eCrPwARY3FJ70AAAA=^[[0m[Pipeline] }
^[[8mha:////4EAdGTjRtNZAgBL+bYNzRV57pfYXzN9+s9X0FDwXS8NXAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPVjHBhL0AAAA=^[[0m[Pipeline] // node
^[[8mha:////4KVEe2i9DY/inRkY0XKEXobRaEdoc3qSwfbx2je+8Ed3AAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPEHtmtL0AAAA=^[[0m[Pipeline] End of Pipeline
Finished: SUCCESS

Badger · January 6, 2022, 5:26pm

With both those logs the filters work as expected and I get

  "job_name" => "test_for_elk",
    "result" => "SUCCESS",
"trigger_by" => "Mao, Xuguang (XXX - XX/XXXXXX)\r",

for the third message. You do have --pipeline.workers set to 1, right? And pipeline.ordered is auto?

maoxuguang · January 7, 2022, 9:13am

Thank you. pipeline.workers is comment off. When I set it to 1, it seems fine. and could I set pipeline.workers more than 1, and the lines of same log.file.path also handled in order?

And could you also reply my first question?

every job log only have one line match "Running on %{GREEDYDATA:not_need}workspace/%{WORD:job_name}", The debug info code => "puts 'mxg'+event.get('[log][file][path]')" only occurs 1 or 0 times for one job, why map_action => "create is not ok?

Badger · January 7, 2022, 4:46pm

If you have an aggregate filter then all of the lines have to go through the same instance of that filter, so pipeline.workers must be set to one. There is no way to route events to a pipeline worker based on the task_id value, which is what you would need to say that events from different files should be routed to different worker threads.

If the trigger_by event is processed before the job_name event (because event order can change) then the map will already have been created, so the job_name aggregate will not process the event.

system · February 4, 2022, 4:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.