Codec in gelf input plugin not working

Elastic Stack Logstash
1

I'm trying to use codec for logs coming from gelf driver of another container to aggregate multiline logs but they are turning into events for each line in message, So multiline is not working at all.

input{
  gelf{
    port => 514
    
    codec => multiline {
        	     
          pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY}"
          negate => "true"
          what => "previous"
	}
  }
}

What am I doing wrong as multiline doesn't seem to be working for any type of pattern for gelf?

2

Please share examples of the messages you are receiving and also an example of your multiline event.

3

2022-06-23 13:53:19.820 WARN [fmpGateway,9ec322a35a8ee222,9ec322a35a8ee222,false] 1 --- [ XNIO-1 task-1] o.s.c.n.zuul.web.ZuulHandlerMapping : No routes found from RouteLocator
In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, SCOPE_address, SCOPE_email, SCOPE_jhipster, SCOPE_microprofile-jwt, SCOPE_offline_access, SCOPE_openid, SCOPE_phone, SCOPE_profile]
In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, ROLE_ADMIN]
**********************@@@authorities::[ROLE_USER, SCOPE_address, SCOPE_email, SCOPE_jhipster, SCOPE_microprofile-jwt, SCOPE_offline_access, SCOPE_openid, SCOPE_phone, SCOPE_profile]
***************attributes:::{sub=4c973896-5761-41fc-8217-07c5d13a004b, email_verified=true, address={}, roles=[ROLE_USER, offline_access, ROLE_ADMIN, uma_authorization], iss=http://keycloak:9080/auth/realms/jhipster, groups=["ROLE_USER","offline_access","ROLE_ADMIN","uma_authorization"], typ=ID, preferred_username=admin, given_name=Admin, nonce=mZhSheicHPK02ZB_Brxruc07HCKxIxCMX88S3aTiHHg, aud=[web_app], acr=1, upn=admin, nbf=Thu Jan 01 00:00:00 GMT 1970, azp=web_app, auth_time=2022-06-23T13:55:51Z, name=Admin Administrator, exp=2022-06-23T14:00:51Z, session_state=33c30859-9e3b-4f89-9f5a-5433fa313140, family_name=Administrator, iat=2022-06-23T13:55:51Z, email=admin@localhost, jti=ef9df142-9abd-40cb-b59c-46982309f3b2}
******************attributes keys:: [sub, email_verified, address, roles, iss, groups, typ, preferred_username, given_name, nonce, aud, acr, upn, nbf, azp, auth_time, name, exp, session_state, family_name, iat, email, jti]
%%%%%%%%authority%%%%ROLE_USER
%%%%%%%%authority%%%%SCOPE_address
%%%%%%%%authority%%%%SCOPE_email
%%%%%%%%authority%%%%SCOPE_jhipster
%%%%%%%%authority%%%%SCOPE_microprofile-jwt

{
"hostname" => "aman-HP-ProBook-440-G8-Notebook-PC",
"@timestamp" => 2022-06-23T13:55:51.622Z,
"command" => "./entrypoint.sh",
"message" => "In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, SCOPE_address, SCOPE_email, SCOPE_jhipster, SCOPE_microprofile-jwt, SCOPE_offline_access, SCOPE_openid, SCOPE_phone, SCOPE_profile]",
"image_id" => "sha256:d3f1e9dc3eca3a7d75eb8cf2b89dfa58e2d533ec9f38e41f81e7714ef708736a",
"tag" => "71239affab45",
"created" => "2022-06-19T17:38:28.723336637Z",
"image_name" => "api:latest",
"container_id" => "71239affab45e8ceebab404c3593cfdd891021ab440b34c57ba07f4e57de807e",
"data" => "{"log":"In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, SCOPE_address, SCOPE_email, SCOPE_jhipster, SCOPE_microprofile-jwt, SCOPE_offline_access, SCOPE_openid, SCOPE_phone, SCOPE_profile]"}",
"log" => "In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, SCOPE_address, SCOPE_email, SCOPE_jhipster, SCOPE_microprofile-jwt, SCOPE_offline_access, SCOPE_openid, SCOPE_phone, SCOPE_profile]",
"hash" => "SharedKey 90c5ca8f-3f89-490e-9a54-4781f4b5ebd1:erEMBDMGTOHxkuyUURER4/mculSANfmCjBNDBbWmgYs=",
"level" => 6,
"@version" => "1",
"container_name" => "docker_fmpgateway-app_1",
"source_host" => "172.18.0.1",
"version" => "1.1",
"x-ms-date" => "Thu, 23 Jun 2022 13:55:51 GMT",
"content-length" => "204",
"string" => "POST\n204\napplication/json\nx-ms-date:Thu, 23 Jun 2022 13:55:51 GMT\n/api/logs"
}

{
"hostname" => "aman-HP-ProBook-440-G8-Notebook-PC",
"@timestamp" => 2022-06-23T13:55:51.624Z,
"command" => "./entrypoint.sh",
"message" => "In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, ROLE_ADMIN]",
"image_id" => "sha256:d3f1e9dc3eca3a7d75eb8cf2b89dfa58e2d533ec9f38e41f81e7714ef708736a",
"tag" => "71239affab45",
"created" => "2022-06-19T17:38:28.723336637Z",
"image_name" => "api:latest",
"container_id" => "71239affab45e8ceebab404c3593cfdd891021ab440b34c57ba07f4e57de807e",
"data" => "{"log":"In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, ROLE_ADMIN]"}",
"log" => "In SecurityConfiguration : UserAuthoritiesMapper:: [ROLE_USER, ROLE_ADMIN]",
"hash" => "SharedKey 90c5ca8f-3f89-490e-9a54-4781f4b5ebd1:PHRlpCWzGk//+MWER0AmxvpUXDDBFI/4ZcaipLOAUsE=",
"level" => 6,
"@version" => "1",
"container_name" => "docker_fmpgateway-app_1",
"source_host" => "172.18.0.1",
"version" => "1.1",
"x-ms-date" => "Thu, 23 Jun 2022 13:55:51 GMT",
"content-length" => "84",
"string" => "POST\n84\napplication/json\nx-ms-date:Thu, 23 Jun 2022 13:55:51 GMT\n/api/logs"
}

4

An input has to have a codec option, but it does not have to use it. By my reading the gelf input ignores it. GELF messages are always JSON encoded objects, so the plugin parses them that way.

5

What could be the way to do this? if I use udp input plugin, it still doesn't work and render bytes.

6

You might be able to do it using the deprecated multiline filter. Source here.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.