# Why my logstash aggregate sometimes works, sometimes not?

**URL:** <https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342>\
**Category:** Logstash\
**Created:** [January 3, 2022, 10:57am UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342 "2022-01-03T10:57:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![maoxuguang](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@maoxuguang](https://discuss.elastic.co/u/maoxuguang)\
**Post date:** [January 3, 2022, 10:57am UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/1 "2022-01-03T10:57:29Z")

</div>

for example, sometimes I can get the tag ["fordebug\_%{job\_name}"], it indicates that the aggregate executed successfully, sometimes I can not get this tag. why? I struggled for days but no finding. I am exhausted. could you help me?

```auto
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => [ 
                 "Started by user %{GREEDYDATA:garbled}\[0m%{GREEDYDATA:trigger_by}",
                 "Running on %{GREEDYDATA:not_need}workspace/%{WORD:job_name}",
                 "Finished: %{WORD:job_result}" 
               ]
    }
  }
  
  if [job_name] {
    mutate {
      add_field => {"debug_job_name" => "%{job_name}_%{[log][file][path]}_'test'"} //for debug
    }
    ruby {
      code => "puts 'mxg'+event.get('[log][file][path]')" //for debug
    }
    aggregate {
      task_id => "%{[log][file][path]}"
      code => "puts 'hahaha'
               map['job_name'] = event.get('job_name')"
      map_action => "create"
      add_tag => ["fordebug_%{job_name}"]
    }
  }

  if [trigger_by] {
    mutate {
      add_field => {"debug_trigger_by" => "%{trigger_by}_%{[log][file][path]}_'test'"}
    }
    ruby {
      code => "puts 'mxg22'+event.get('[log][file][path]')"
    }

    aggregate {
      task_id => "%{[log][file][path]}"
      code => "puts 'hahaha2'
               map['trigger_by'] = event.get('trigger_by')"
      map_action => "create"
      add_tag => ["fordebug_%{trigger_by}"]
    }
  }
  if [job_result]{
    aggregate {
      task_id => "%{[log][file][path]}"
      code => "event.set('job_name', map['job_name'])
               event.set('trigger_by', map['trigger_by'])
               event.set('result', event.get('job_result'))"
      end_of_task => true
      add_tag => ["fordebug_%{trigger_by}_%{job_name}"]
    
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://x.x.x.x:9200"]
    index => "jenkins-test-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
  stdout { codec => rubydebug }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 3, 2022, 5:19pm UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/2 "2022-01-03T17:19:58Z")

</div>

> [@maoxuguang](#):
>
> it indicates that the aggregate executed successfully

Exactly, so the add\_tag does not occur if

- The task\_id is nil or none of the sprintf substitutions occur
- An aggregate exception occurs
- map\_action is "create" and the map already exists
- map\_action is "update" and the map does not exist

---

<div class="post-metadata">

**Author:** ![maoxuguang](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@maoxuguang](https://discuss.elastic.co/u/maoxuguang)\
**Post date:** [January 5, 2022, 6:41am UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/3 "2022-01-05T06:41:12Z")

</div>

Thanks for your help. when I delete following line

> map\_action =\> "create"

or replace it with:

> map\_action =\> "create\_or\_update"

it works. The debug tag was created. But I still have two questions:

1. every job log only have one line match "Running on %{GREEDYDATA:not\_need}workspace/%{WORD:job\_name}", The debug info code =\> "puts 'mxg'+event.get('[log][file][path]')" only occurs 1 or 0 times for one job, why map\_action =\> "create is not ok?
2. even the aggregate for job\_name and trigger\_by were ok, sometimes event.set('job\_name', map['job\_name']) was not ok, i did not get field job\_name, why? most of times it is ok. sometimes it went wrong, usually after I restart logstash. The two debug jobs only run less than 1 minute, so I think it is not timeout issue.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 5, 2022, 4:56pm UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/4 "2022-01-05T16:56:44Z")

</div>

Without seeing all of the input data I cannot suggest what the problem might be.

---

<div class="post-metadata">

**Author:** ![maoxuguang](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@maoxuguang](https://discuss.elastic.co/u/maoxuguang)\
**Post date:** [January 6, 2022, 2:49am UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/5 "2022-01-06T02:49:31Z")

</div>

In filebeat I only configured two jobs for debug. one is test\_for\_elk\_trigger, here is the log of it:

> Started by user ^[[8mha:////4H7fyrm5bxQG052fIU6vSOnHcHxgpop3ONd2SfKAwxJZAAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyFEgZu/dLi1CL9itLcxHwADV5VrMAAAAA=^[[0mMao, Xuguang (XXX - XX/XXXXXXX)  
> Running in Durability level: MAX\_SURVIVABILITY  
> ^[[8mha:////4B2qx+efg7CWwFWY2xTHrGxyLnOH0bH3HnBvrif7ncNcAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==^[[0m[Pipeline] Start of Pipeline  
> ^[[8mha:////4EssAPmau1BUq9ByzFAZZzH47erpvSDyTfjBqQMTh8YoAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==^[[0m[Pipeline] node  
> Running on ^[[8mha:////4A0sqggNTVzkWT0y6yuhvFX4jtPzhql6rV2A2sGTQYGUAAAApB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzeEgYJ/eT83ILSktQi/Yzy4uQS3eKiXDNdAxN9APLsa0/NAAAA^[[0mhwsct-srm6-04 in /home/ca\_RFSW\_C\_CH/jenkinsSlave/workspace/test\_for\_elk\_trigger  
> ^[[8mha:////4IuW672shnIJoQTMHtwuqjLOjDvTOMiBQCPqkldloO3tAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==^[[0m[Pipeline] {  
> ^[[8mha:////4BBGFx4A8yiARlRQXYuu9hKy3riSRGdrscpLYICsJh4cAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==^[[0m[Pipeline] stage  
> ^[[8mha:////4NHJRbOFAPOMOom5h7ZApqYfYQm33BmhK698O4jmCeDmAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==^[[0m[Pipeline] { (test)  
> ^[[8mha:////4PCfXYgwv0nkBqeBy6X5JF6b+MU4Pw6oTL4taplCZDCfAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gwExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jcChmMxwAAAAA==^[[0m[Pipeline] script  
> ^[[8mha:////4MthFXvFIUxd96C7TMbeiGo/JQF18iCLiM9zfXpB7LEPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0BAkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfpX/cvwAAAAA==^[[0m[Pipeline] {  
> ^[[8mha:////4BP9oMp7JOOd0xDXMimgIVdJnK3VcVA4/Ff6MuM8c85QAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaoyVLaHxBMgiwpL/WWDdrTyRV/MOEjexcpJJ5k3zXm9Y5wQnSpZ7g73DrB2PYbR18YlSfws0cU9XrgkzBcOlmSR15rygpGJgCWtgJWBjUAfKDm2BrfDqodqg0LaXkup3FNC4boAnMFHNRaXyhTnB7j52mfDn+SuAORZg+9pD/AAeoCAqvAAAAA==^[[0m[Pipeline] build  
> Scheduling project: ^[[8mha:////4NREmu5cFdAPB2TupCS990Rmjky+NGncluyTrld6JBq1AAAAnR+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAweoBn6WflJ+iWpxSXxaflF8ak52foAXzXXUccAAAA=^[[0mtest\_for\_elk  
> Starting building: ^[[8mha:////4F6AUKk3xwL27saE3PoVlWGjmay8PF5OlymnJ9ngS1h0AAAAoh+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzBEgYx/az8JP2S1OKS+LT8ovjUnGx9QzMjfQD5XeFeywAAAA==^[[0mtest\_for\_elk #162  
> ^[[8mha:////4CMtnfiiK59q8syMON3TLxItNS0BZ/prH08G8PuvJc0FAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMKBDglR0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8Brmfz270AAAA=^[[0m[Pipeline] }  
> ^[[8mha:////4HYmobU6fhUpojOBPZKLgYHVJZHryQmZrJoEksxXLVwfAAAAox+LCAAAAAAAAP9tjTESgjAQRT84FraWHiIMlYVjZZuh8QQRYgxkdjFZhMoTeTXvICMzVv7qv9e81xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0gL0tBtv8Aj1Msdr0AAAA=^[[0m[Pipeline] // script  
> ^[[8mha:////4I76xG7OMtD44SWf+OG8r1xfT2ylYWON5eTUr9g6/tfJAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPyRmLRr0AAAA=^[[0m[Pipeline] }  
> ^[[8mha:////4JlmLZ82OCmblWGyCtovpwbJQyWaGG/cUjUFgUgz0imIAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPw2Hi4r0AAAA=^[[0m[Pipeline] // stage  
> ^[[8mha:////4BZhF5r/fZ3+wng2wqlxUDaA1STMWHdDjAdudt8P6hUWAAAAox+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6LEhM04uTYsfkGFWgvNe9g+hMkv8tf8B4kkTt7pnrOc1xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0g35eCrPwARY3FJ70AAAA=^[[0m[Pipeline] }  
> ^[[8mha:////4EAdGTjRtNZAgBL+bYNzRV57pfYXzN9+s9X0FDwXS8NXAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPVjHBhL0AAAA=^[[0m[Pipeline] // node  
> ^[[8mha:////4KVEe2i9DY/inRkY0XKEXobRaEdoc3qSwfbx2je+8Ed3AAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPEHtmtL0AAAA=^[[0m[Pipeline] End of Pipeline  
> Finished: SUCCESS

Another one is test\_for\_elk, here is the log of it:

> Started by upstream project "^[[8mha:////4N1/xOgK+T9Td6VjTwOneTFBtt5x7FQnYNboln669TAiAAAApR+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyREgYp/az8JP2S1OKS+LT8ovjUnOx4oEnp6alF+gC9Ar1qzwAAAA==^[[0mtest\_for\_elk\_trigger" build number ^[[8mha:////4MVgz9XBjuf0VaMoP0Y8fx0xQIo17kGFPq850pVwAYbeAAAApx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAymEgYZ/az8JP2S1OKS+LT8ovjUnOx4oEnp6alF+pbGAHiMn4rRAAAA^[[0m93  
> originally caused by:  
> Started by user ^[[8mha:////4H7fyrm5bxQG052fIU6vSOnHcHxgpop3ONd2SfKAwxJZAAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAyFEgZu/dLi1CL9itLcxHwADV5VrMAAAAA=^[[0mMao, Xuguang (XXX - XX/XXXXXX)  
> Running in Durability level: MAX\_SURVIVABILITY  
> ^[[8mha:////4B2qx+efg7CWwFWY2xTHrGxyLnOH0bH3HnBvrif7ncNcAAAAoh+LCAAAAAAAAP9tjTEOwjAQBM8BClpKHuFItIiK1krDC0x8GCfWnbEdkooX8TX+gCESFVvtrLSa5wtWKcKBo5UdUu8otU4GP9jS5Mixv3geZcdn2TIl9igbHBs2eJyx4YwwR1SwULBGaj0nRzbDRnX6rmuvydanHMu2V1A5c4MHCFXMWcf8hSnC9jqYxPTz/BXAFEIGsfuclm8zQVqFvQAAAA==^[[0m[Pipeline] Start of Pipeline  
> ^[[8mha:////4EssAPmau1BUq9ByzFAZZzH47erpvSDyTfjBqQMTh8YoAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycohUghExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jduZBmjwAAAAA==^[[0m[Pipeline] node  
> Running on ^[[8mha:////4GbBzFxigEI/sveYQD+IWZopfD0PTIEYSFDRvdIjb/xwAAAApB+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzeEgYJ/eT83ILSktQi/Yzy4uQS3eKiXDNdA0t9AL+SxfrNAAAA^[[0mhwsct-srm6-09 in /home/ca\_RFSW\_C\_CH/jenkinsSlave/workspace/test\_for\_elk  
> ^[[8mha:////4IuW672shnIJoQTMHtwuqjLOjDvTOMiBQCPqkldloO3tAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gA0xsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfoP95RwAAAAA==^[[0m[Pipeline] {  
> ^[[8mha:////4BBGFx4A8yiARlRQXYuu9hKy3riSRGdrscpLYICsJh4cAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gQkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jc09154wAAAAA==^[[0m[Pipeline] stage  
> ^[[8mha:////4NHJRbOFAPOMOom5h7ZApqYfYQm33BmhK698O4jmCeDmAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0ggUxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jek7ggRwAAAAA==^[[0m[Pipeline] { (test)  
> ^[[8mha:////4PCfXYgwv0nkBqeBy6X5JF6b+MU4Pw6oTL4taplCZDCfAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0gwExsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jcChmMxwAAAAA==^[[0m[Pipeline] script  
> ^[[8mha:////4MthFXvFIUxd96C7TMbeiGo/JQF18iCLiM9zfXpB7LEPAAAApR+LCAAAAAAAAP9tjTEOwjAUQ3+KOrAycoh0BAkxsUZZOEFIQkgb/d8mKe3EibgadyBQiQlLlmxL1nu+oE4RjhQdby12HpP2vA+jK4lPFLtroIm3dOGaMFGwXNpJkrGnpUrKFhaxClYC1hZ1oOTRZdiIVt1VExS65pxj2Q4CKm8GeAAThZxVzN8yR9jeRpMIf5y/AJj7DGxXvP/86jfpX/cvwAAAAA==^[[0m[Pipeline] {  
> ^[[8mha:////4BP9oMp7JOOd0xDXMimgIVdJnK3VcVA4/Ff6MuM8c85QAAAAoh+LCAAAAAAAAP9tjTEOAiEURD9rLGwtPQTbaoyVLaHxBMgiwpL/WWDdrTyRV/MOEjexcpJJ5k3zXm9Y5wQnSpZ7g73DrB2PYbR18YlSfws0cU9XrgkzBcOlmSR15rygpGJgCWtgJWBjUAfKDm2BrfDqodqg0LaXkup3FNC4boAnMFHNRaXyhTnB7j52mfDn+SuAORZg+9pD/AAeoCAqvAAAAA==^[[0m[Pipeline] echo  
> just for elk test aaaaaa  
> ^[[8mha:////4CMtnfiiK59q8syMON3TLxItNS0BZ/prH08G8PuvJc0FAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMKBDglR0VppeIFJjHFi3QX7QlLxIr7GH4iIRMVWO9PM641lijhydKqx1HpKlVdd6N301MCxvQYeVMMXVTElDlaVdii5tqcZSxaLeVmOhcbKUhU4eXKCtW7MwxTBkCvOEid30Mh9fccTmZ7KYqJ8YYzY3Po6Mf06fwMYu06Q77aCbP8Brmfz270AAAA=^[[0m[Pipeline] }  
> ^[[8mha:////4HYmobU6fhUpojOBPZKLgYHVJZHryQmZrJoEksxXLVwfAAAAox+LCAAAAAAAAP9tjTESgjAQRT84FraWHiIMlYVjZZuh8QQRYgxkdjFZhMoTeTXvICMzVv7qv9e81xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0gL0tBtv8Aj1Msdr0AAAA=^[[0m[Pipeline] // script  
> ^[[8mha:////4I76xG7OMtD44SWf+OG8r1xfT2ylYWON5eTUr9g6/tfJAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPyRmLRr0AAAA=^[[0m[Pipeline] }  
> ^[[8mha:////4JlmLZ82OCmblWGyCtovpwbJQyWaGG/cUjUFgUgz0imIAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsTZPsPw2Hi4r0AAAA=^[[0m[Pipeline] // stage  
> ^[[8mha:////4BZhF5r/fZ3+wng2wqlxUDaA1STMWHdDjAdudt8P6hUWAAAAox+LCAAAAAAAAP9tjbEOgjAURS8YB1dHP6LEhM04uTYsfkGFWgvNe9g+hMkv8tf8B4kkTt7pnrOc1xvrFHHk6FRrqfOUaq/6MLj5qZFjdw08qpYvqmZKHKyq7FhxY08LViwWy7IcK42NpTpw8uQEW92ahymCIVecJc7uoJH75o4nMj2XxUT5whSxuw1NYvp1/gYw9b0g35eCrPwARY3FJ70AAAA=^[[0m[Pipeline] }  
> ^[[8mha:////4EAdGTjRtNZAgBL+bYNzRV57pfYXzN9+s9X0FDwXS8NXAAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRQlSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPVjHBhL0AAAA=^[[0m[Pipeline] // node  
> ^[[8mha:////4KVEe2i9DY/inRkY0XKEXobRaEdoc3qSwfbx2je+8Ed3AAAAox+LCAAAAAAAAP9tjTEOwjAQBDdBFLSUPMIRFDSIitZKwwtMYowT6y7YF5KKF/E1/kBEJCq22plmXm8sU8SRo1ONpdZTqrzqQu+mpwaO7TXwoBq+qIopcbCqtEPJtT3NWLJYzMtyLDRWlqrAyZMTrHVjHqYIhlxxlji5g0bu6zueyPRUFhPlC2PE5tbXienX+RvA2HWCfLsXZLsPEHtmtL0AAAA=^[[0m[Pipeline] End of Pipeline  
> Finished: SUCCESS

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 6, 2022, 5:26pm UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/6 "2022-01-06T17:26:19Z")

</div>

With both those logs the filters work as expected and I get

```
  "job_name" => "test_for_elk",
    "result" => "SUCCESS",
"trigger_by" => "Mao, Xuguang (XXX - XX/XXXXXX)\r",

```

for the third message. You do have [--pipeline.workers set to 1](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-description), right? And pipeline.ordered is auto?

---

<div class="post-metadata">

**Author:** ![maoxuguang](https://avatars.discourse-cdn.com/v4/letter/m/97f17d/32.png) [@maoxuguang](https://discuss.elastic.co/u/maoxuguang)\
**Post date:** [January 7, 2022, 9:13am UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/7 "2022-01-07T09:13:12Z")

</div>

Thank you. pipeline.workers is comment off. When I set it to 1, it seems fine. and could I set pipeline.workers more than 1, and the lines of same log.file.path also handled in order?

And could you also reply my first question?

1. every job log only have one line match "Running on %{GREEDYDATA:not\_need}workspace/%{WORD:job\_name}", The debug info code =\> "puts 'mxg'+event.get('[log][file][path]')" only occurs 1 or 0 times for one job, why map\_action =\> "create is not ok?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 7, 2022, 4:46pm UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/8 "2022-01-07T16:46:26Z")

</div>

If you have an aggregate filter then all of the lines have to go through the same instance of that filter, so pipeline.workers must be set to one. There is no way to route events to a pipeline worker based on the task\_id value, which is what you would need to say that events from different files should be routed to different worker threads.

If the trigger\_by event is processed before the job\_name event (because event order can change) then the map will already have been created, so the job\_name aggregate will not process the event.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 4, 2022, 4:47pm UTC](https://discuss.elastic.co/t/why-my-logstash-aggregate-sometimes-works-sometimes-not/293342/9 "2022-02-04T16:47:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.