# Logstash azure log analytics output - Failed to flush outgoing items - block in start\_workers **URL:** \ **Category:** Logstash\ **Created:** [May 18, 2022, 1:54pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078 "2022-05-18T13:54:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![pavank](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@pavank](https://discuss.elastic.co/u/pavank)\ **Post date:** [May 18, 2022, 1:54pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/1 "2022-05-18T13:54:56Z")
Hi, I am using the redhat openshift logging operator provided fluentd forward to forward my logs to a Logstash instance, and using the azure log analytics output plugin to forward the logs to an Azure Log Analytics Workspace. I am seeing the following Warning continuously in my Logstash logs: ```auto May 18 09:48:53 cbxrelaysrvsit1 logstash[39503]: [2022-05-18T09:48:53,219][WARN][logstash.outputs.azureloganalytics][main][43328c4a9749fe314828ca147d3b05e72c47d0cd340a51d239257c16fca6eac1] Failed to flush outgoing items {:outgoing_count=>1, :exception=>"JSON::GeneratorError", :backtrace=>["json/ext/GeneratorMethods.java:79:in `to_json'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/logAnalyticsClient/logStashAutoResizeBuffer.rb:41:in `flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/logAnalyticsClient/logStashAutoResizeBuffer.rb:28:in `add_event_document'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/outputs/microsoft-logstash-output-azure-loganalytics.rb:85:in `block in multi_receive'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/microsoft-logstash-output-azure-loganalytics-1.0.0/lib/logstash/outputs/microsoft-logstash-output-azure-loganalytics.rb:78:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:295:in `block in start_workers'"]} ``` I am also using Filebeat in some other VMs, and am able to aggregate those logs to Azure Log Analytics, but facing this issue after using fluent input. The following is my Logstash pipeline.conf: ```auto input { beats { port => "5044" } tcp { codec => fluent port => 4000 } } filter { mutate { copy => { "json_log_s" => "Message" } } } output { if "PTE" in [tags]{ microsoft-logstash-output-azure-loganalytics { workspace_id => # workspace_key => # custom_log_table_name => "PTE_OCP" } } else if "test" in [tags]{ microsoft-logstash-output-azure-loganalytics { workspace_id => # workspace_key => # custom_log_table_name => "TEST" } } else if "SIT2" in [tags] { microsoft-logstash-output-azure-loganalytics { workspace_id => # workspace_key => # custom_log_table_name => "SIT2_OCP" } } else if "SIT" in [tags]{ microsoft-logstash-output-azure-loganalytics { workspace_id => # workspace_key => # custom_log_table_name => "SIT_OCP" } } else if "UAT" in [tags]{ microsoft-logstash-output-azure-loganalytics { workspace_id => # workspace_key => # custom_log_table_name => "UAT_OCP" } } else { microsoft-logstash-output-azure-loganalytics { workspace_id => # workspace_key => # custom_log_table_name => "remrep_ocp" } } } ``` Could you please help me understand this issue? Best Regards, Pavan ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 18, 2022, 4:58pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/2 "2022-05-18T16:58:11Z")
> [@pavank](#): > > Could you please help me understand this issue? The to\_json is finding something in the message that it does not know how to encode. JSON is unicode. If your message has some other character encoding then it may contain characters (or NaN) that are not valid Unicode, in which case this exception will occur. You need to find out what encoding the source of your events is using. You can force the encoding to UTF-8 in a ruby filter once you know that. [This](https://stackoverflow.com/questions/18067203/ruby-to-json-issue-with-error-illegal-malformed-utf-8) may help you understand the problem. ---
**Author:** ![pavank](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@pavank](https://discuss.elastic.co/u/pavank)\ **Post date:** [May 18, 2022, 5:38pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/3 "2022-05-18T17:38:21Z")
Hi @Badger Thank you very much for that reply. I think you've pushed me in the right direction. I will check the encoding of my logs. I have 3 questions from your response: 1. From the solution in stackoverflow link, I would need to identify the corresponding for **ISO-8859-1** that I am using in my logs, correct? ```auto "iPhone\xAE".force_encoding("ISO-8859-1").encode("UTF-8") #=> "iPhone®" ``` 1. Would I apply this on each one of the characters that are not encoded in JSON or can I do it for the message field in it's entirety? 2. Where exactly do I need to apply this ruby filter? Best Regards, Pavan ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 18, 2022, 6:58pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/4 "2022-05-18T18:58:26Z")
Yes, you need to find what encoding your message is in. The ruby filter would be something like ``` ruby { code => ' event.set("message", event.get("message").force_encoding("xxx").encode("UTF-8") ' } ``` ---
**Author:** ![pavank](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@pavank](https://discuss.elastic.co/u/pavank)\ **Post date:** [May 19, 2022, 1:56pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/5 "2022-05-19T13:56:41Z")
@Badger I am also seeing this in the start up of my Logstash logs: ```auto [2022-05-19T09:53:32,292][ERROR][logstash.codecs.fluent][main][4fe2598517a75ecc0668a0a6e7a9f6e9c056628fcac106ac2ef5213729b2555f] Fluent parse error, original data now in message field {:error=>#, :data=>102} [2022-05-19T09:53:32,303][ERROR][logstash.codecs.fluent][main][4fe2598517a75ecc0668a0a6e7a9f6e9c056628fcac106ac2ef5213729b2555f] Fluent parse error, original data now in message field {:error=>#, :data=>"pod_ip"} [2022-05-19T09:53:32,309][ERROR][logstash.codecs.fluent][main][4fe2598517a75ecc0668a0a6e7a9f6e9c056628fcac106ac2ef5213729b2555f] Fluent parse error, original data now in message field {:error=>#, :data=>"host"} ``` Are these the characters that are causing me trouble? Best Regards, Pavan ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 19, 2022, 3:17pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/6 "2022-05-19T15:17:50Z")
You would need to look at the [message] field (as the error message says) to determine what the data looks like, and why the codec cannot process it. ---
**Author:** ![pavank](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@pavank](https://discuss.elastic.co/u/pavank)\ **Post date:** [May 24, 2022, 12:29pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/7 "2022-05-24T12:29:11Z")
Hey @Badger , I saw this [GIT post](https://github.com/loomltd/fluent-plugin-out_tcp_secure/issues/1) that said to remove the codec = fluent from my logstash inputs. When I did that change, I am able to receive the logs, but it has all the metadata that fluent sends in the Message field itself. Attaching my logstash inputs section: ```auto input { beats { port => "5044" } tcp { port => 4000 } } ``` I am sending this the log to Azure Log Analytics workspace output. I am seeing the Fluentd's metadata as well in the Message itself. I have highlighted the actual message in the below log snippet. A sample log message that I received that was previously throwing the parse error: ```auto \x93هkubernetes.var.log.pods.cibc-iso-sit1_digital-registry-3-nt8xg_83abd5ef-a808-461f-bb83-e543bc8693a1.digital-registry-java-service.0.log\xDB\u0000\u000E\u0006̒\xCEb\x86K\x8C\x8B\xA6docker\x81\xACcontainer_id\xD9@9c8f7f00e4c2ae853111a36fe7ae915593345ac5082c73c717bc3cbfac4e7740\xAAkubernetes\x8C\xAEcontainer_name\xBDdigital-registry-java-service\xAEnamespace_name\xADcibc-iso-sit1\xA8pod_name\xB8digital-registry-3-nt8xg\xAFcontainer_image\xD9Uimage-registry.openshift-image-registry.svc:5000/cibc-iso-sit1/digital-registry:8.0.2\xB2container_image_idٗimage-registry.openshift-image-registry.svc:5000/cibc-iso-sit1/digital-registry@sha256:f7c87983cc55520e0d40f5d19109f3933ffcaaa5464980f174a4755394096cb3\xA6pod_id\xD9$83abd5ef-a808-461f-bb83-e543bc8693a1\xA6pod_ip\xAC10.131.2.185\xA4host\xACisositwrk001\xA6labels\x83\xAAdeployment\xB2digital-registry-3\xB0deploymentconfig\xB0digital-registry\xA4name\xBBdigital-registry-deployment\xAAmaster_url\xBEhttps://kubernetes.default.svc\xACnamespace_id\xD9$5cd3f450-e872-4599-8eee-9c5ca31508e3\xB0namespace_labels\x81\xBBkubernetes_io/metadata_name\xADcibc-iso-sit1\xA7 **message** \xC4\xE32022-05-19 13:52:12,617 DEBUG [http-nio-8761-exec-1] com.netflix.eureka.registry.AbstractInstanceRegistry (getApplicationsFromMultipleRegions:737) - Fetching applications registry with remote regions: false, Regions argument []\xA5level\xA5debug\xA8hostname\xACisositwrk001\xB1pipeline_metadata\x81\xA9collector\x85\xA7ipaddr4\xAC172.16.24.37\xA9inputname\xB5fluent-plugin-systemd\xA4name\xA7fluentd\xABreceived_at\xD9 2022-05-19T13:52:12.620695+00:00\xA7version\xAC1.14.5 1.6.0\xA9openshift\x81\xA6labels\x81\xA9clusterId\xAEremrep-nonprod\xAA ``` Is there a way I can parse this with Logstash itself? Best Regards, Pavan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 21, 2022, 12:30pm UTC](https://discuss.elastic.co/t/logstash-azure-log-analytics-output-failed-to-flush-outgoing-items-block-in-start-workers/305078/8 "2022-06-21T12:30:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.