# Logstash log says "Pushing flushed events" but I can't see any output in ES

**URL:** <https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988>\
**Category:** Logstash\
**Created:** [September 27, 2017, 1:38pm UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988 "2017-09-27T13:38:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [September 27, 2017, 1:38pm UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/1 "2017-09-27T13:38:38Z")

</div>

Hi,

I am getting the following log in Logstash, but from some reason in Elasticsearch I can't see any output. How can this be?

> [2017-09-27T16:38:40,961][DEBUG][logstash.pipeline] Pushing flushed events {:event=\>2017-09-27T13:38:40.956Z %{host} %{message}}  
> [2017-09-27T16:38:40,962][DEBUG][logstash.pipeline] output received {"event"=\>{"msgSentToJMS\_MT"=\>"1506519335020", "reason"=\>"", "lnpInfoGot"=\>"1506519334976", "MedGotMsgFromApi"=\>"1506519334971", "medSentRespToApi"=\>"1506519335021", "blackListCBGot"=\>"1506519334978", "FSM\_got"=\>"1506519508985", "transactionId"=\>"10529843-c76d-4f20-b367-b97286e2a34f", "accountInfoGot"=\>"1506519334974", "SRI\_sent"=\>"1506519505363", "tags"=\>["aggregate", "\_dateparsefailure"], "concatenatedFromTotal"=\>"", "medGotAck\_MT"=\>"1506519341405", "SRI\_got"=\>"1506519507287", "@timestamp"=\>2017-09-27T13:38:40.956Z, "authCBGot"=\>"1506519334973", "terminationCause"=\>"QUEUED", "cdrType"=\>"aggregated", "@version"=\>"1", "FSM\_sent"=\>"1506519507510"}}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 27, 2017, 8:08pm UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/2 "2017-09-27T20:08:15Z")

</div>

Show your configuration.

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [September 28, 2017, 9:00am UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/5 "2017-09-28T09:00:24Z")

</div>

Hi @magnusbaeck disregard the config file I sent since:

When checking the logstash output by:

```
output {
	stdout {
		codec => rubydebug
	}
}

```

I see that the logstash does post info. So my guess is that since my ES is in a yellow state from somereason it does not read these certain logs.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 28, 2017, 10:14am UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/6 "2017-09-28T10:14:56Z")

</div>

A yellow state shouldn't be a problem. If ES doesn't accept what Logstash is sending there should be stuff in the log about that.

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [September 28, 2017, 11:33am UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/7 "2017-09-28T11:33:44Z")

</div>

so here is my config file, maybe there I have a problem?:

```
filter {
  json{
        source => "message"
  }

  aggregate {
		task_id => "%{transactionId}"
		code => "

			if (map['authCBGot'])
				event.set('authCBGot', (map['authCBGot']))
			else
				if (event.get('authCBGot').eql? '0')
					# do nothing
				else
					map['authCBGot'] = event.get('authCBGot')
				end
			end
			if (map['blackListCBGot'])
				event.set('blackListCBGot', (map['blackListCBGot']))
			else
				if (event.get('blackListCBGot').eql? '0')
					# do nothing
				else
					map['blackListCBGot'] = event.get('blackListCBGot')
				end
			end
			if (map['medGotFromUser_MO'])
				event.set('medGotFromUser_MO', (map['medGotFromUser_MO']))
			else
				if (event.get('medGotFromUser_MO').eql? '0')
					# do nothing
				else
					map['medGotFromUser_MO'] = event.get('medGotFromUser_MO')
				end
			end
			if (map['MedGotMsgFromApi'])
				event.set('MedGotMsgFromApi', (map['MedGotMsgFromApi']))
			else
				if (event.get('MedGotMsgFromApi').eql? '0')
					# do nothing
				else
					map['MedGotMsgFromApi'] = event.get('MedGotMsgFromApi')
				end
			end
			if (map['lnpInfoGot'])
				event.set('lnpInfoGot', (map['lnpInfoGot']))
			else
				if (event.get('lnpInfoGot').eql? '0')
					# do nothing
				else
					map['lnpInfoGot'] = event.get('lnpInfoGot')
				end
			end
			if (map['msgSentToJMS_MT'])
				event.set('msgSentToJMS_MT', (map['msgSentToJMS_MT']))
			else
				if (event.get('msgSentToJMS_MT').eql? '0')
					# do nothing
				else
					map['msgSentToJMS_MT'] = event.get('msgSentToJMS_MT')
				end
			end
			if (map['medGotAck_MT'])
				event.set('medGotAck_MT', (map['medGotAck_MT']))
			else
				if (event.get('medGotAck_MT').eql? '0')
					# do nothing
				else
					map['medGotAck_MT'] = event.get('medGotAck_MT')
				end
			end
			if (map['medSentToJMS_MO'])
				event.set('medSentToJMS_MO', (map['medSentToJMS_MO']))
			else
				if (event.get('medSentToJMS_MO').eql? '0')
					# do nothing
				else
					map['medSentToJMS_MO'] = event.get('medSentToJMS_MO')
				end
			end
			if (map['medSentDLR_MT'])
				event.set('medSentDLR_MT', (map['medSentDLR_MT']))
			else
				if (event.get('medSentDLR_MT').eql? '0')
					# do nothing
				else
					map['medSentDLR_MT'] = event.get('medSentDLR_MT')
				end
			end
			if (map['accountInfoGot'])
				event.set('accountInfoGot', (map['accountInfoGot']))
			else
				if (event.get('accountInfoGot').eql? '0')
					# do nothing
				else
					map['accountInfoGot'] = event.get('accountInfoGot')
				end
			end
			if (map['smscGotAck_MO'])
				event.set('smscGotAck_MO', (map['smscGotAck_MO']))
			else
				if (event.get('smscGotAck_MO').eql? '0')
					# do nothing
				else
					map['smscGotAck_MO'] = event.get('smscGotAck_MO')
				end
			end
			if (map['SMPP_got'])
				event.set('SMPP_got', (map['SMPP_got']))
			else
				if (event.get('SMPP_got').eql? '0')
					# do nothing
				else
					map['SMPP_got'] = event.get('SMPP_got')
				end
			end
			if (map['SMPP_sent'])
				event.set('SMPP_sent', (map['SMPP_sent']))
			else
				if (event.get('SMPP_sent').eql? '0')
					# do nothing
				else
					map['SMPP_sent'] = event.get('SMPP_sent')
				end
			end
			if (map['medGotMsg_MO'])
				event.set('medGotMsg_MO', (map['medGotMsg_MO']))
			else
				if (event.get('medGotMsg_MO').eql? '0')
					# do nothing
				else
					map['medGotMsg_MO'] = event.get('medGotMsg_MO')
				end
			end
			if (map['smscSentAck_MO'])
				event.set('smscSentAck_MO', (map['smscSentAck_MO']))
			else
				if (event.get('smscSentAck_MO').eql? '0')
					# do nothing
				else
					map['smscSentAck_MO'] = event.get('smscSentAck_MO')
				end
			end
			if (map['medSentRespToApi'])
				event.set('medSentRespToApi', (map['medSentRespToApi']))
			else
				if (event.get('medSentRespToApi').eql? '0')
					# do nothing
				else
					map['medSentRespToApi'] = event.get('medSentRespToApi')
				end
			end
			if (map['medGotDLR_MT'])
				event.set('medGotDLR_MT', (map['medGotDLR_MT']))
			else
				if (event.get('medGotDLR_MT').eql? '0')
					# do nothing
				else
					map['medGotDLR_MT'] = event.get('medGotDLR_MT')
				end
			end
			if (map['medSentToUser_MO'])
				event.set('medSentToUser_MO', (map['medSentToUser_MO']))
			else
				if (event.get('medSentToUser_MO').eql? '0')
					# do nothing
				else
					map['medSentToUser_MO'] = event.get('medSentToUser_MO')
				end
			end
		    if (map['SRI_sent'])
				event.set('SRI_sent', (map['SRI_sent']))
			else
				if (event.get('SRI_sent').eql? '0')
					# do nothing
				else
					map['SRI_sent'] = event.get('SRI_sent')
				end
			end
			if (map['SRI_got'])
				event.set('SRI_got', (map['SRI_got']))
			else
				if (event.get('SRI_got').eql? '0')
					# do nothing
				else
					map['SRI_got'] = event.get('SRI_got')
				end
			end
			if (map['FSM_sent'])
				event.set('FSM_sent', (map['FSM_sent']))
			else
				if (event.get('FSM_sent').eql? '0')
					# do nothing
				else
					map['FSM_sent'] = event.get('FSM_sent')
				end
			end
			if (map['FSM_got'])
				event.set('FSM_got', (map['FSM_got']))
			else
				if (event.get('FSM_got').eql? '0')
					# do nothing
				else
					map['FSM_got'] = event.get('FSM_got')
				end
			end
      if (map['terminationCause'])
          event.set('terminationCause', (map['terminationCause']))
      else
				if (event.get('terminationCause').eql? '0')
					# do nothing
				else
					map['terminationCause'] = event.get('terminationCause')
				end
			end
      if (map['reason'])
          event.set('reason', (map['reason']))
      else
				if (event.get('reason').eql? '0')
					# do nothing
				else
					map['reason'] = event.get('reason')
				end
			end

			"
		push_map_as_event_on_timeout => true
    timeout_task_id_field => "transactionId"
		timeout => 3600
    inactivity_timeout => 2
    timeout_tags => ["aggregate"]
		timeout_code => "event.set('cdrType', 'aggregated')"

}
if "aggregate" not in [tags] {
		drop{}
	}
```

---

<div class="post-metadata">

**Author:** ![tomer](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@tomer](https://discuss.elastic.co/u/tomer)\
**Post date:** [September 28, 2017, 11:35am UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/8 "2017-09-28T11:35:48Z")

</div>

and the second part that looks a lot less important, since if I erase the "drop" part I will get logs to ES, but not the aggregated ones.

May I mention that this worked for someone else on his ELK

```
  date {
      match => ["eventTimestamp", "UNIX"]
      target => "eventTimestamp"
    }
    date {
      match => ["FSM_got", "UNIX"]
      target => "FSM_got"
    }
    date {
      match => ["FSM_sent", "UNIX"]
      target => "FSM_sent"
    }
    date {
      match => ["authCBGot", "UNIX"]
      target => "authCBGot"
    }
    date {
      match => ["MedGotMsgFromApi", "UNIX"]
      target => "MedGotMsgFromApi"
    }
    date {
      match => ["accountInfoGot", "UNIX"]
      target => "accountInfoGot"
    }
    date {
      match => ["lnpInfoGot", "UNIX"]
      target => "lnpInfoGot"
    }
    date {
      match => ["blackListCBGot", "UNIX"]
      target => "blackListCBGot"
    }
    date {
      match => ["msgSentToJMS_MT", "UNIX"]
      target => "msgSentToJMS_MT"
    }
    date {
      match => ["alertSC_got", "UNIX"]
      target => "alertSC_got"
    }
    date {
      match => ["SRI_sent", "UNIX"]
      target => "SRI_sent"
    }
    date {
      match => ["SRI_got", "UNIX"]
      target => "SRI_got"
    }
    date {
      match => ["medGotMsg_MO", "UNIX"]
      target => "medGotMsg_MO"
    }
    date {
      match => ["medSentToUser_MO", "UNIX"]
      target => "medSentToUser_MO"
    }
    date {
      match => ["medGotDLR_MT", "UNIX"]
      target => "medGotDLR_MT"
    }
    date {
      match => ["medSentDLR_MT", "UNIX"]
      target => "medSentDLR_MT"
    }
    date {
      match => ["medGotAck_MT", "UNIX"]
      target => "medGotAck_MT"
    }
    date {
      match => ["SMPP_sent", "UNIX"]
      target => "SMPP_sent"
    }
    date {
      match => ["SMPP_got", "UNIX"]
      target => "SMPP_got"
    }
    date {
      match => ["medGotFromUser_MO", "UNIX"]
      target => "medGotFromUser_MO"
    }
    date {
      match => ["medSentToJMS_MO", "UNIX"]
      target => "medSentToJMS_MO"
    }
    date {
      match => ["smscGotAck_MO", "UNIX"]
      target => "smscGotAck_MO"
    }
    date {
      match => ["smscSentAck_MO", "UNIX"]
      target => "smscSentAck_MO"
    }
    date {
      match => ["medSentRespToApi", "UNIX"]
      target => "medSentRespToApi"
    }
     date {
      match => ["asrFrom", "UNIX"]
      target => "asrFrom"
    }

}
#output {
#	stdout {
# codec => rubydebug
#	}
#}
output {
    elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "aggregated"
      document_type => "aggregatedtype"
    }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2017, 11:36am UTC](https://discuss.elastic.co/t/logstash-log-says-pushing-flushed-events-but-i-cant-see-any-output-in-es/101988/9 "2017-10-26T11:36:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.