Logstash Servers Stopping Each Others Logging

Hi all,

Currently I am troubleshooting a configuration that is working in our prod environment but not in our development environment. We are running multiple servers in Dev so I am testing changes on one while I let the other continue to run. I currently am seeing that every time I tweak something in the configuration of the one server and stop the logstash service on the working server, logstash breaks. Upon turning on the other servers service, logs will continue not to parse until I stop the service on the server I am testing the new config on. Aren't these supposed to be parsing individually and not working in tandam?

Can you explain better what you are doing? it is not clear what you are doing and what is not working.

Describe your architecture and share the configuration you are testing and what you changed that make it stop working, also share logs from your logstash server.

If you have multiple Logstash servers, they are independent from each other and have no knowledge about the existence of other servers, so it makes no sense a configuration change in one server will impact another server, but you need to share your configurations and also the logs.

We are just trying to use a standard config for both prod and dev. The current dev config is the same as prod but without cloning the data and then sorting the output. Below is the config that is in prod but breaking dev. It looks like in theory it should work but then when I tail the logs, messages will stop going through for this config:

input {
  azure_event_hubs {
    event_hub_connections => ["Endpoint=sb://#{eventhub_host}/;SharedAccessKeyName=#{eventhub_SharedAccessKeyName};SharedAccessKey=#{eventhub_SharedAccessKey};EntityPath=eventlogging"]
    threads => 10
    decorate_events => false
    consumer_group => "#{eventhub_consumer_group}"
    storage_connection => "DefaultEndpointsProtocol=https;AccountName=#{eventhub_storageAccount};AccountKey=#{eventhub_storageAccountkey};EndpointSuffix=core.windows.net"
    tags => ["#{eventhub_tags}"]
    storage_container => ["#{eventhub_container}"]
    checkpoint_interval => 0
  }
}

filter {
  if ("#{eventhub_tags}" in [tags]) {

    json { source => "message" }

    #Remove all TN 2FA cookie fields that contain an extremely long random GUID field name.
    #This long field name causes the Kibana UI to become unreadable.
    #The values for this GUID can be found in value for other fields
    ruby {
      code => '
        tn2FAkey = event.get("message").scan(/(?:\"((?:[0-9A-Z]{2}\-){63}[0-9A-Z]{2})\": \"[a-zA-Z0-9]+\")/)

        tn2FAkey.each do |match|
          event.remove("[Request][Cookies]" + "[" +  match[0] + "]")
        end
      '
    }

    mutate { remove_field => [********S] }

    # If Session field contains the word SignupInfo then parse the portal user specific data out of the session
    if ("SignupInfo" in [Session]) {

      grok { match => { "Session" => "(*******).*?(?=<br \/>))" } }

      mutate { add_field => { ******* } }

    }

    # If Session field contains the word PortalUserInfo then parse the portal user specific data out of the session
    else if ("PortalUserInfo" in [Session]) {

      grok { match => { "Session" => "(*******).*?(?=<br \/>))" } }

      mutate { add_field => { "*******" } }

    }
    
    # If Session field contains the word UserInfo then parse the user specific data out of the session
    else if ("UserInfo" in [Session]) {

      grok { match => { "Session" => "(*******).*?(?=<br \/>))" } }

      mutate { add_field => { "*******" } }

    }

    mutate { replace => { "LoggerName" => "tn-%{LoggerName}" } }

    # Creates a clone of the current log and adds a type field with the name of the clone
    clone { clones => ['Clone_No_PHI'] }

    # Remove any PHI from the message clone
    if [type] == 'Clone_No_PHI' {
       mutate { remove_field => [
        "Session", "[Request][Form]", "[Request][Data][HTTP_COOKIE]", "[Request][Data][ALL_RAW]",
        "[Request][Cookies][ASP.NET_SessionId]", "[Request][Cookies][SessionlessIdentity]" , "[Request][Headers]"] }
    }
  }
}

output {
  if ("#{eventhub_tags}" in [tags]) {
    # Send the orginal log and clone to their own indices in order to allow for index level permissions to be added.
    if [type] == 'Clone_No_PHI' {
        elasticsearch {
          hosts => #{elastic_hosts}
          index => "%{[LoggerName]}"
          truststore => "#{elastic.truststore}"
          truststore_password => ""
          user => "#{elastic_user}"
          password => "#{elastic_password}"
          ssl_certificate_verification => false
        }
    }
    else {
        elasticsearch {
          hosts => #{elastic_hosts}
          index => "priv-%{[LoggerName]}"
          truststore => "#{elastic.truststore}"
          truststore_password => ""
          user => "#{elastic_user}"
          password => "#{elastic_password}"
          ssl_certificate_verification => false
        }
        }
  }
}

I still doesn't understand what is breaking, you didn't share any logs from Logstash.

Can you replicate the issue and share the Logstash logs?

Also, since you are using the Event Hub input, Are you using different consumer groups and storage containers for the prod and dev environments?

Sorry about that, here are the logs here once I stop the logstash service on the server with the working config. Yes we have prod automatically writing to the default storage_container eventlogging and then specify dev. They also are using different consumer_groups as we define them in octopus variables for these servers.

[2022-01-10T15:48:22,673][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] messagingFactory[MF_149a69_1641847702525], hostName[deveventsthnlogging-ncus.servicebus.windows.
net], getting a session.
[2022-01-10T15:48:22,674][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_16987b_1641847702535], entityName[cbs-session], sessionIncCapa
city[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,674][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionLocalOpen connectionId[MF_149a69_1641847702525], entityName[eventlogging/ConsumerGroups/d
evelkstack/Partitions/5], condition[Error{condition=null, description='null', info=null}]
[2022-01-10T15:48:22,670][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] messagingFactory[MF_985f72_1641847702536], hostName[deveventsthnlogging-ncus.servicebus.windows.
net], getting a session.
[2022-01-10T15:48:22,675][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 7: Opening EH receiver with epoch 6834 at l
ocation offset[16428265188768], sequenceNumber[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,675][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionLocalOpen connectionId[MF_985f72_1641847702536], entityName[eventlogging/ConsumerGroups/d
evelkstack/Partitions/6], condition[Error{condition=null, description='null', info=null}]
[2022-01-10T15:48:22,676][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] clientId[PR_c6219b_1641847702676_MF_ea6b7a_1641847702524-InternalReceiver], path[eventlogging/Con
sumerGroups/develkstack/Partitions/7], operationTimeout[PT1M], creating a receive link
[2022-01-10T15:48:22,675][INFO ][com.microsoft.azure.eventhubs.impl.SendLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen senderName[cbs], linkName[cbs:sender], remoteTarget[Target{address='$cbs', durab
le=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
[2022-01-10T15:48:22,677][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] messagingFactory[MF_ea6b7a_1641847702524], hostName[deveventsthnlogging-ncus.servicebus.windows.
net], getting a session.
[2022-01-10T15:48:22,679][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_985f72_1641847702536], entityName[eventlogging/ConsumerGroups/
develkstack/Partitions/6], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,679][INFO ][com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[16376740350736], sequence
Number[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,680][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[PR_8d82a3_1641847702642_MF_985f72_1641847702536-InternalReceiver]
, linkName[LN_6e0fd5_1641847702680_83b_G22], localSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/6', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={a
pache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '16376740350736'}}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,681][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionLocalOpen connectionId[MF_ea6b7a_1641847702524], entityName[cbs-session], condition[Error
{condition=null, description='null', info=null}]
[2022-01-10T15:48:22,682][INFO ][com.microsoft.azure.eventhubs.impl.SendLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen senderName[cbs], linkName[cbs:sender], localTarget[Target{address='$cbs', durable
=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
[2022-01-10T15:48:22,683][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[cbs], linkName[cbs:receiver], localSource[Source{address='$cbs', 
durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,673][INFO ][com.microsoft.azure.eventhubs.impl.RequestResponseOpener][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] requestResponseChannel.onOpen complete clientId[MF_84e0d2_1641847702510], session[cbs-sessi
on], link[cbs], endpoint[$cbs]
[2022-01-10T15:48:22,686][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_8d82a3_1641847702642_MF_985f72_1641847702536-InternalReceiver
], linkName[LN_6e0fd5_1641847702680_83b_G22], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/6', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@4d43e1c3}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,687][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_8d82a3_1641847702642_MF_985f72_1641847702536-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/6], linkName[LN_6e0fd5_1641847702680_83b_G22], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,688][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_ea6b7a_1641847702524], entityName[cbs-session], sessionIncCapa
city[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,688][INFO ][com.microsoft.azure.eventhubs.impl.SendLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen senderName[cbs], linkName[cbs:sender], remoteTarget[Target{address='$cbs', durab
le=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, capabilities=null}]
[2022-01-10T15:48:22,689][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[cbs], linkName[cbs:receiver], remoteSource[Source{address='$cbs'
, durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,690][INFO ][com.microsoft.azure.eventhubs.impl.RequestResponseOpener][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] requestResponseChannel.onOpen complete clientId[MF_ea6b7a_1641847702524], session[cbs-sessi
on], link[cbs], endpoint[$cbs]
[2022-01-10T15:48:22,691][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 6: EH client and receiver creation finished
[2022-01-10T15:48:22,692][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] messagingFactory[MF_84e0d2_1641847702510], hostName[deveventsthnlogging-ncus.servicebus.windows.
net], getting a session.
[2022-01-10T15:48:22,692][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionLocalOpen connectionId[MF_84e0d2_1641847702510], entityName[eventlogging/ConsumerGroups/d
evelkstack/Partitions/9], condition[Error{condition=null, description='null', info=null}]
[2022-01-10T15:48:22,678][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_149a69_1641847702525], entityName[eventlogging/ConsumerGroups/
develkstack/Partitions/5], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,694][INFO ][com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[16372439173776], sequence
Number[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,699][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[PR_ca4f7c_1641847702641_MF_149a69_1641847702525-InternalReceiver]
, linkName[LN_4e5199_1641847702699_d20_G26], localSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/5', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={a
pache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '16372439173776'}}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,699][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_84e0d2_1641847702510], entityName[eventlogging/ConsumerGroups/
develkstack/Partitions/9], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,700][INFO ][com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[16385329770640], sequence
Number[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,696][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] messagingFactory[MF_ea6b7a_1641847702524], hostName[deveventsthnlogging-ncus.servicebus.windows.
net], getting a session.
[2022-01-10T15:48:22,702][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionLocalOpen connectionId[MF_ea6b7a_1641847702524], entityName[eventlogging/ConsumerGroups/d
evelkstack/Partitions/7], condition[Error{condition=null, description='null', info=null}]
[2022-01-10T15:48:22,678][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[cbs], linkName[cbs:receiver], remoteSource[Source{address='$cbs'
, durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=null, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,703][INFO ][com.microsoft.azure.eventhubs.impl.RequestResponseOpener][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] requestResponseChannel.onOpen complete clientId[MF_16987b_1641847702535], session[cbs-sessi
on], link[cbs], endpoint[$cbs]
[2022-01-10T15:48:22,705][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_ca4f7c_1641847702641_MF_149a69_1641847702525-InternalReceiver
], linkName[LN_4e5199_1641847702699_d20_G26], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/5', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@7b1af800}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,705][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_ca4f7c_1641847702641_MF_149a69_1641847702525-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/5], linkName[LN_4e5199_1641847702699_d20_G26], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,706][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_ea6b7a_1641847702524], entityName[eventlogging/ConsumerGroups/
develkstack/Partitions/7], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,701][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[PR_19af22_1641847702660_MF_84e0d2_1641847702510-InternalReceiver]
, linkName[LN_e6436b_1641847702701_cae0_G4], localSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/9', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={a
pache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '16385329770640'}}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,738][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_19af22_1641847702660_MF_84e0d2_1641847702510-InternalReceiver
], linkName[LN_e6436b_1641847702701_cae0_G4], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/9', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@7c0b8b3b}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,740][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_19af22_1641847702660_MF_84e0d2_1641847702510-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/9], linkName[LN_e6436b_1641847702701_cae0_G4], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,707][INFO ][com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[16428265188768], sequence
Number[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,740][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 5: EH client and receiver creation finished
[2022-01-10T15:48:22,708][INFO ][com.microsoft.azure.eventhubs.impl.MessagingFactory][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] messagingFactory[MF_16987b_1641847702535], hostName[deveventsthnlogging-ncus.servicebus.windows.
net], getting a session.
[2022-01-10T15:48:22,741][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[PR_c6219b_1641847702676_MF_ea6b7a_1641847702524-InternalReceiver]
, linkName[LN_ffcfde_1641847702741_e2c_G12], localSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/7', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={a
pache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '16428265188768'}}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,742][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionLocalOpen connectionId[MF_16987b_1641847702535], entityName[eventlogging/ConsumerGroups/d
evelkstack/Partitions/8], condition[Error{condition=null, description='null', info=null}]
[2022-01-10T15:48:22,743][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 9: EH client and receiver creation finished
[2022-01-10T15:48:22,757][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_16987b_1641847702535], entityName[eventlogging/ConsumerGroups/
develkstack/Partitions/8], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,759][INFO ][com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[16449753903048], sequence
Number[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,759][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[PR_2b0355_1641847702654_MF_16987b_1641847702535-InternalReceiver]
, linkName[LN_f9cf66_1641847702759_04e2_G8], localSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/8', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={a
pache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '16449753903048'}}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,761][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_c6219b_1641847702676_MF_ea6b7a_1641847702524-InternalReceiver
], linkName[LN_ffcfde_1641847702741_e2c_G12], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/7', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@31f53228}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,772][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_2b0355_1641847702654_MF_16987b_1641847702535-InternalReceiver
], linkName[LN_f9cf66_1641847702759_04e2_G8], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/8', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@1b4323b5}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,772][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_2b0355_1641847702654_MF_16987b_1641847702535-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/8], linkName[LN_f9cf66_1641847702759_04e2_G8], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,762][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_c6219b_1641847702676_MF_ea6b7a_1641847702524-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/7], linkName[LN_ffcfde_1641847702741_e2c_G12], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,774][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 7: EH client and receiver creation finished
[2022-01-10T15:48:22,786][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 8: EH client and receiver creation finished
^[[D^C
[root@DEVLSH04 logstash]# tail -f /var/log/logstash/logstash-plain.log
[2022-01-10T15:48:22,743][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 9: EH client and receiver creation finished
[2022-01-10T15:48:22,757][INFO ][com.microsoft.azure.eventhubs.impl.SessionHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onSessionRemoteOpen connectionId[MF_16987b_1641847702535], entityName[eventlogging/ConsumerGroups/
develkstack/Partitions/8], sessionIncCapacity[0], sessionOutgoingWindow[2147483647]
[2022-01-10T15:48:22,759][INFO ][com.microsoft.azure.eventhubs.impl.PartitionReceiverImpl][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] receiverPath[RECEIVER IS NULL], action[createReceiveLink], offset[16449753903048], sequence
Number[null], enqueuedTime[null], inclusiveFlag[false]
[2022-01-10T15:48:22,759][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkLocalOpen receiverName[PR_2b0355_1641847702654_MF_16987b_1641847702535-InternalReceiver]
, linkName[LN_f9cf66_1641847702759_04e2_G8], localSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/8', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter={a
pache.org:selector-filter:string=UnknownDescribedType{descriptor=apache.org:selector-filter:string, described=amqp.annotation.x-opt-offset > '16449753903048'}}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,761][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_c6219b_1641847702676_MF_ea6b7a_1641847702524-InternalReceiver
], linkName[LN_ffcfde_1641847702741_e2c_G12], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/7', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@31f53228}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,772][INFO ][com.microsoft.azure.eventhubs.impl.ReceiveLinkHandler][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onLinkRemoteOpen receiverName[PR_2b0355_1641847702654_MF_16987b_1641847702535-InternalReceiver
], linkName[LN_f9cf66_1641847702759_04e2_G8], remoteSource[Source{address='eventlogging/ConsumerGroups/develkstack/Partitions/8', durable=NONE, expiryPolicy=SESSION_END, timeout=0, dynamic=false, dynamicNodeProperties=null, distributionMode=null, filter=
{apache.org:selector-filter:string=org.apache.qpid.proton.codec.DecoderImpl$UnknownDescribedType@1b4323b5}, defaultOutcome=null, outcomes=null, capabilities=null}]
[2022-01-10T15:48:22,772][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_2b0355_1641847702654_MF_16987b_1641847702535-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/8], linkName[LN_f9cf66_1641847702759_04e2_G8], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,762][INFO ][com.microsoft.azure.eventhubs.impl.MessageReceiver][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] onOpenComplete - clientId[PR_c6219b_1641847702676_MF_ea6b7a_1641847702524-InternalReceiver], rece
iverPath[eventlogging/ConsumerGroups/develkstack/Partitions/7], linkName[LN_ffcfde_1641847702741_e2c_G12], updated-link-credit[300], sentCredits[300]
[2022-01-10T15:48:22,774][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 7: EH client and receiver creation finished
[2022-01-10T15:48:22,786][INFO ][com.microsoft.azure.eventprocessorhost.PartitionPump][main][6505ec297fde135a342b783981cd85372cb775d363fb64ea56491cf8ba248d01] host logstash-3df826a5-b00f-4d62-a8a4-5c4dcd8f5d55: 8: EH client and receiver creation finished

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.