Filebeat support required

Elastic Stack Beats
1

getting error :

Jan 14 21:30:07 ndc3vmappelk05 filebeat[2374]: 2023-01-14T21:30:07.888+0530        ERROR        pipeline/output.go:100        Failed to connect to backoff(async(tcp://10.94.241.145:5043)): dial tcp 10.94.241.145:5043: connect: connection refused

Below is netstat output

Jan 14 21:29:54 ndc3vmappelk05 filebeat[2374]: 2023-01-14T21:29:54.585+0530        ERROR        pipeline/output.go:100        Failed to connect to backoff(async(tcp://10.94.241.145:5043)): dial tcp 10.94.241.145:5043: connect: connection refused
Jan 14 21:29:54 ndc3vmappelk05 filebeat[2374]: 2023-01-14T21:29:54.585+0530        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(async(tcp://10.94.241.145:5043)) with 3 reconnect attempt(s)
Jan 14 21:30:07 ndc3vmappelk05 filebeat[2374]: 2023-01-14T21:30:07.888+0530        ERROR        pipeline/output.go:100        Failed to connect to backoff(async(tcp://10.94.241.145:5043)): dial tcp 10.94.241.145:5043: connect: connection refused
Jan 14 21:30:07 ndc3vmappelk05 filebeat[2374]: 2023-01-14T21:30:07.888+0530        INFO        pipeline/output.go:93        Attempting to reconnect to backoff(async(tcp://10.94.241.145:5043)) with 4 reconnect attempt(s)
Jan 14 21:30:11 ndc3vmappelk05 filebeat[2374]: 2023-01-14T21:30:11.964+0530        INFO        [monitoring]        log/log.go:145        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":60,"time":{"ms":64}},"total":{"ticks":200,"time":{"ms":210},"value":200},"user":{"ticks":140,"time":{"ms":146}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":15},"info":{"ephemeral_id":"6f932a40-caf4-4c24-87fe-7fd013f80e36","uptime":{"ms":30035}},"memstats":{"gc_next":33845072,"memory_alloc":18556112,"memory_total":42003712,"rss":55070720},"runtime":{"goroutines":75}},"filebeat":{"events":{"active":4126,"added":4171,"done":45},"harvester":{"files":{"781f58a1-b1ef-4895-b9f7-83ce73a3e723":{"last_event_published_time":"2023-01-14T21:29:44.966Z","last_event_timestamp":"2023-01-14T21:29:44.966Z","name":"/var/log/vmware-vmtoolsd-root.log","read_offset":5132379,"size":5132379,"start_time":"2023-01-14T21:29:42.134Z"},"8f8e0855-9c11-48bc-97f7-db56ceb2e105":{"last_event_published_time":"2023-01-14T21:29:44.974Z","last_event_timestamp":"2023-01-14T21:29:44.974Z","name":"/var/log/maillog","read_offset":57877,"size":57877,"start_time":"2023-01-14T21:29:44.963Z"},"97d522c7-2ef6-4a70-87f6-6c96f382cc2a":{"last_event_published_time":"2023-01-14T21:29:45.018Z","last_event_timestamp":"2023-01-14T21:29:45.018Z","name":"/var/log/messages","read_offset":62021813,"size":107044035,"start_time":"2023-01-14T21:29:44.959Z"},"dd9d7e78-f82c-4f6e-a6d4-fb06c693b2a9":{"last_event_published_time":"2023-01-14T21:29:45.018Z","last_event_timestamp":"2023-01-14T21:29:45.018Z","name":"/tmp/SIMEX1/SIMEX_1App04/ndc3vpaieaiapp79_SIMEX_1App04_SystemOut.log","read_offset":518881,"size":2670235,"start_time":"2023-01-14T21:29:42.129Z"},"df3d58fb-ad1b-45fe-b39f-1756a7e16c90":{"last_event_published_time":"2023-01-14T21:29:44.968Z","last_event_timestamp":"2023-01-14T21:29:44.968Z","name":"/var/log/errlog","read_offset":81085414,"size":81085414,"start_time":"2023-01-14T21:29:42.129Z"},"e870f4a5-900b-4f52-8ebe-15279f3fe9f7":{"last_event_published_time":"","last_event_timestamp":"","name":"/var/log/lastlog","read_offset":296966,"size":297256,"start_time":"2023-01-14T21:29:44.959Z"},"effc792b-f386-4972-8c4c-87f0ba7e81a8":{"last_event_published_time":"","last_event_timestamp":"","name":"/var/log/logstash/logstash-plain.log","size":3770,"start_time":"2023-01-14T21:29:52.126Z"},"f722b6cb-4665-4934-a21d-db7f1f2b3d4b":{"last_event_published_time":"","last_event_timestamp":"","name":"/var/log/tallylog","read_offset":65271,"size":65728,"start_time":"2023-01-14T21:29:44.964Z"}},"open_files":8,"running":8,"started":8}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":2},"output":{"type":"logstash"},"pipeline":{"clients":3,"events":{"active":4118,"filtered":50,"published":4116,"retry":6144,"total":4168}}},"registrar":{"states":{"current":42,"update":45},"writes":{"success":45,"total":45}},"system":{"cpu":{"cores":6},"load":{"1":3.18,"15":3.3,"5":3.2,"norm":{"1":0.53,"15":0.55,"5":0.5333}}}}}}
[root@ndc3vmappelk05 ~]#
You have new mail in /var/spool/mail/root
[root@ndc3vmappelk05 ~]#
[root@ndc3vmappelk05 ~]#
[root@ndc3vmappelk05 ~]#
[root@ndc3vmappelk05 ~]# netstat -an | grep 5043
[root@ndc3vmappelk05 ~]# netstat -an | grep 5044
[root@ndc3vmappelk05 ~]# netstat -an 5044
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:2223            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:20048           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:44114           0.0.0.0:*               LISTEN
tcp       17      0 0.0.0.0:38194           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:43861         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:37978           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:14206           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:10110           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:6014            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:53183         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:44865         0.0.0.0:*               LISTEN
tcp        0      0 10.94.241.145:5601      0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:199           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:56743         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:55916           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:1837            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:1581            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:36110         0.0.0.0:*               LISTEN
tcp        0      0 10.94.241.145:35306     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35346     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:43576     10.94.104.184:80        ESTABLISHED
tcp        0      0 10.94.241.145:49520     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:46240     172.18.172.11:9200      ESTABLISHED
tcp        0      0 10.94.241.145:48706     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:36118     10.94.241.146:9200      ESTABLISHED
tcp        0     64 10.94.241.145:22        10.94.240.195:65292     ESTABLISHED
tcp        0      0 127.0.0.1:44865         127.0.0.1:48660         ESTABLISHED
tcp        0      0 10.94.241.145:34850     10.94.241.146:9200      ESTABLISHED
tcp        0      0 127.0.0.1:37803         127.0.0.1:56743         ESTABLISHED
tcp        0      0 10.94.241.145:35300     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38912     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38308     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:59354     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38946     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:59340     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:36116     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:43066     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:36100     10.94.241.146:9200      ESTABLISHED
tcp        0      0 127.0.0.1:56743         127.0.0.1:37803         ESTABLISHED
tcp        0      0 10.94.241.145:60792     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:514       10.94.77.118:39284      ESTABLISHED
tcp        0      0 10.94.241.145:38278     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:32854     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:46082     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38928     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:48726     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:34870     10.94.104.184:80        ESTABLISHED
tcp        0      0 10.94.241.145:35268     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:60912     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38958     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:56128     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38954     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:60968     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:5601      10.94.240.196:52090     ESTABLISHED
tcp        0      0 10.94.241.145:35338     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38938     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35312     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:60810     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:43072     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38316     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:44102     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:44158     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38940     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35004     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35254     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38194     10.94.77.118:43524      ESTABLISHED
tcp        0      0 10.94.241.145:49502     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38312     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:48702     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38924     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:44106     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38296     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:34866     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:34862     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:60906     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:59458     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:49338     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:49522     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:48692     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:49488     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:36102     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:34028     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:56140     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:34854     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35250     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:60910     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:44096     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35214     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:48712     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:59996     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38306     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38314     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35316     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38918     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:1581      10.94.65.62:38398       ESTABLISHED
tcp        0      0 10.94.241.145:38284     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:44112     10.94.241.146:9200      ESTABLISHED

tcp        0      0 10.94.241.145:36080     172.18.172.11:9200      ESTABLISHED
tcp        0      0 10.94.241.145:5601      10.94.240.196:60568     TIME_WAIT

tcp        0      0 10.94.241.145:49494     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:36112     172.18.172.11:9200      ESTABLISHED
tcp        0      0 10.94.241.145:60828     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35262     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38288     10.94.241.146:9200      ESTABLISHED
tcp        0      0 127.0.0.1:53183         127.0.0.1:33925         ESTABLISHED
tcp        0      0 10.94.241.145:44108     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:43548     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35332     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:59336     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35320     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:47978     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:43110     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:42966     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38273     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:32932     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:46080     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:33282     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:5601      10.94.240.195:63325     ESTABLISHED
tcp        0      0 10.94.241.145:47700     10.94.104.184:80        ESTABLISHED
tcp        0      0 10.94.241.145:33270     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:32794     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:35298     10.94.241.146:9200      ESTABLISHED
tcp        0      0 10.94.241.145:42842     172.18.172.11:9200      ESTABLISHED
tcp        0      0 10.94.241.145:38290     10.94.241.146:9200      ESTABLISHED
tcp6       0      0 :::111                  :::*                    LISTEN
tcp6       0      0 :::20048                :::*                    LISTEN
tcp6       0      0 :::43571                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::4118                 :::*                    LISTEN
tcp6       0      0 ::1:25                  :::*                    LISTEN
tcp6       0      0 :::14206                :::*                    LISTEN
tcp6       0      0 :::10110                :::*                    LISTEN
tcp6       0      0 :::6014                 :::*                    LISTEN
tcp6       0      0 :::2049                 :::*                    LISTEN
tcp6       0      0 :::514                  :::*                    LISTEN
tcp6       0      0 :::49611                :::*                    LISTEN
tcp6       0      0 :::15948                :::*                    LISTEN
tcp6       0      0 :::11852                :::*                    LISTEN
unix  2      [ ]         STREAM     CONNECTED     25057765 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     72536714
unix  2      [ ]         DGRAM                    174885759
unix  3      [ ]         STREAM     CONNECTED     265402515 /tmp/__7dea7cf5_123183957016580.sock
unix  3      [ ]         STREAM     CONNECTED     174876155
unix  3      [ ]         STREAM     CONNECTED     24268
unix  2      [ ]         DGRAM                    16190
unix  3      [ ]         STREAM     CONNECTED     24294
unix  3      [ ]         STREAM     CONNECTED     72245855 @sym_config_ipc
unix  3      [ ]         STREAM     CONNECTED     72245848 @sym_config_ipc
unix  2      [ ]         STREAM     CONNECTED     15832320 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     265399029 /tmp/__7dea7cf5_123183957016578.sock
unix  3      [ ]         STREAM     CONNECTED     24255
unix  3      [ ]         STREAM     CONNECTED     18243    /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     174876810
unix  3      [ ]         STREAM     CONNECTED     24285
unix  3      [ ]         STREAM     CONNECTED     18173
unix  3      [ ]         STREAM     CONNECTED     481404189 /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     174888132 /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     97025919
unix  3      [ ]         STREAM     CONNECTED     18092    /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     24264
unix  3      [ ]         STREAM     CONNECTED     24240
unix  2      [ ]         DGRAM                    22034
unix  3      [ ]         STREAM     CONNECTED     72246700 @sym_config_ipc
unix  2      [ ]         STREAM     CONNECTED     38375686 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     24298
unix  2      [ ]         DGRAM                    481368313
unix  3      [ ]         SEQPACKET  CONNECTED     265399489
unix  2      [ ]         DGRAM                    21385
unix  3      [ ]         STREAM     CONNECTED     265398087
unix  3      [ ]         STREAM     CONNECTED     481280625
unix  3      [ ]         STREAM     CONNECTED     72538618 @sym_config_ipc
unix  2      [ ]         DGRAM                    174876821
unix  3      [ ]         STREAM     CONNECTED     24273
unix  3      [ ]         STREAM     CONNECTED     466731950
unix  3      [ ]         STREAM     CONNECTED     265398019
unix  2      [ ]         STREAM     CONNECTED     41137245 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     16118    /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    72250598
unix  3      [ ]         STREAM     CONNECTED     174887031 /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     24247
unix  3      [ ]         STREAM     CONNECTED     17045
unix  2      [ ]         STREAM     CONNECTED     481415688
unix  3      [ ]         STREAM     CONNECTED     174886331
unix  3      [ ]         STREAM     CONNECTED     471445303
unix  3      [ ]         STREAM     CONNECTED     97029342 @sym_config_ipc
unix  3      [ ]         STREAM     CONNECTED     19754
unix  3      [ ]         STREAM     CONNECTED     24277
unix  2      [ ]         DGRAM                    18207
unix  3      [ ]         STREAM     CONNECTED     72249753 @sym_config_ipc
unix  3      [ ]         STREAM     CONNECTED     24226
unix  3      [ ]         STREAM     CONNECTED     429160847 /run/systemd/journal/stdout
unix  2      [ ]         STREAM     CONNECTED     41131369 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     24234
unix  2      [ ]         DGRAM                    481281649
unix  3      [ ]         STREAM     CONNECTED     265398834 /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     24243
unix  3      [ ]         STREAM     CONNECTED     438966188 /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     265396095
unix  3      [ ]         STREAM     CONNECTED     20026    /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     265398090
unix  3      [ ]         STREAM     CONNECTED     20742    /run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    2475466
unix  3      [ ]         STREAM     CONNECTED     24229
unix  3      [ ]         STREAM     CONNECTED     24295
unix  3      [ ]         STREAM     CONNECTED     24280
unix  3      [ ]         STREAM     CONNECTED     481415684
unix  2      [ ]         DGRAM                    174886212
unix  3      [ ]         STREAM     CONNECTED     265398092
unix  3      [ ]         STREAM     CONNECTED     24265
unix  3      [ ]         STREAM     CONNECTED     24250
unix  3      [ ]         STREAM     CONNECTED     2475463
unix  3      [ ]         STREAM     CONNECTED     254690782 @sym_config_ipc
unix  2      [ ]         DGRAM                    16172
unix  2      [ ]         STREAM     CONNECTED     41018118 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     17185    /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     265398088
unix  3      [ ]         STREAM     CONNECTED     24261
unix  3      [ ]         STREAM     CONNECTED     24246
unix  2      [ ]         DGRAM                    20737
unix  2      [ ]         DGRAM                    481369222
unix  3      [ ]         STREAM     CONNECTED     471444743
unix  3      [ ]         STREAM     CONNECTED     174886227 /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    17051
unix  2      [ ]         STREAM     CONNECTED     24015
unix  3      [ ]         STREAM     CONNECTED     16170
unix  3      [ ]         STREAM     CONNECTED     265396094
unix  3      [ ]         STREAM     CONNECTED     18927
unix  2      [ ]         DGRAM                    72246624
unix  3      [ ]         STREAM     CONNECTED     15588    /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    174888150
unix  3      [ ]         SEQPACKET  CONNECTED     265399488
unix  3      [ ]         STREAM     CONNECTED     481405032
unix  3      [ ]         STREAM     CONNECTED     72246681
unix  3      [ ]         STREAM     CONNECTED     17225    /run/gssproxy.sock
unix  3      [ ]         STREAM     CONNECTED     16114    /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     481280626
unix  3      [ ]         STREAM     CONNECTED     429159386
unix  3      [ ]         STREAM     CONNECTED     19755
unix  3      [ ]         STREAM     CONNECTED     24291
unix  3      [ ]         STREAM     CONNECTED     24276
unix  3      [ ]         STREAM     CONNECTED     72246626 @sym_config_ipc
unix  3      [ ]         STREAM     CONNECTED     254693389
unix  3      [ ]         STREAM     CONNECTED     265399602 /tmp/__7dea7cf5_123192546951169.sock
unix  3      [ ]         STREAM     CONNECTED     20717
unix  3      [ ]         STREAM     CONNECTED     17053
unix  3      [ ]         STREAM     CONNECTED     265397072 /tmp/__7dea7cf5_123183957016579.sock
unix  3      [ ]         STREAM     CONNECTED     24253
unix  2      [ ]         STREAM     CONNECTED     25137797 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         DGRAM                    174876846
unix  3      [ ]         STREAM     CONNECTED     24279
unix  3      [ ]         STREAM     CONNECTED     471444742
unix  3      [ ]         STREAM     CONNECTED     265399005 /tmp/__7dea7cf5_123192546951170.sock
unix  3      [ ]         STREAM     CONNECTED     19913
unix  3      [ ]         STREAM     CONNECTED     16110    /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    481404191
unix  3      [ ]         STREAM     CONNECTED     24270
unix  2      [ ]         DGRAM                    23858
unix  3      [ ]         STREAM     CONNECTED     265400429
unix  3      [ ]         STREAM     CONNECTED     72295461
unix  3      [ ]         STREAM     CONNECTED     72249793
unix  3      [ ]         STREAM     CONNECTED     265402514
unix  3      [ ]         STREAM     CONNECTED     24249
unix  3      [ ]         STREAM     CONNECTED     174875382 /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     24283
unix  3      [ ]         STREAM     CONNECTED     18174
unix  3      [ ]         STREAM     CONNECTED     174885519
unix  2      [ ]         DGRAM                    174888026
unix  2      [ ]         DGRAM                    17210
unix  3      [ ]         STREAM     CONNECTED     24288
unix  3      [ ]         STREAM     CONNECTED     265398086
unix  3      [ ]         STREAM     CONNECTED     24227
unix  3      [ ]         STREAM     CONNECTED     481414953 /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     72249805
unix  2      [ ]         DGRAM                    21352
unix  2      [ ]         DGRAM                    471445298
unix  2      [ ]         STREAM     CONNECTED     35876    /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     24271
unix  3      [ ]         STREAM     CONNECTED     24262
unix  3      [ ]         STREAM     CONNECTED     24236
unix  3      [ ]         STREAM     CONNECTED     174882798
unix  3      [ ]         STREAM     CONNECTED     72537660
unix  3      [ ]         STREAM     CONNECTED     24292
unix  3      [ ]         STREAM     CONNECTED     265398093
unix  3      [ ]         STREAM     CONNECTED     24241
unix  3      [ ]         STREAM     CONNECTED     265400376 /tmp/__7dea7cf5_123183957016577.sock
unix  3      [ ]         STREAM     CONNECTED     72248746 @sym_config_ipc
unix  2      [ ]         DGRAM                    20707
unix  3      [ ]         STREAM     CONNECTED     18981
unix  2      [ ]         DGRAM                    174884275
unix  2      [ ]         STREAM     CONNECTED     14981823 /var/opt/ds_agent/event_socket_am
unix  3      [ ]         STREAM     CONNECTED     18206
unix  3      [ ]         STREAM     CONNECTED     72246625
unix  3      [ ]         STREAM     CONNECTED     466733547 /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     24258

[root@ndc3vmappelk05 filebeat]# cat filebeat.yml

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages
    - /var/log/auth.log
    - /var/log/syslog
    - /var/log/faillog
    - /var/log/lastlog
    - /var/log/sa
    - /var/log/*.log
    - /var/log/*log
    - /var/adm/commandlog
    - /tmp/SIMEX1/SIMEX_1App04/ndc3vpaieaiapp79_SIMEX_1App04_SystemOut.log
#  fields:
#    app_id: LINUX_OS_10.94.241.145

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  # multiline.pattern: '^\[(\d+\/\d+\/\d+ \d+:\d+:\d+:\d+ \w+)\]|^\[\d{2}\-\d{2}\-\d{2}-\d{2}:\d{2}:\d{2}] \[\w+( )*\]|^(\w{3} \w{3} \d+ \d+:\d+:\d+ \w+ \d{4})|^(\w{3} \w{3}  \d+ \d+:\d+:\d+ \w+ \d{4})'
#  multiline.pattern: '^\[\d+\/\d+|\d+\/\d+\/\d+\s+\d+\:\d+\:\d+'

  # Defines if the pattern set under pattern should be negated or not. Default is false.
 #multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
 # multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

#============================= Elastic Cloud ==================================

# These settings simplify using filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.94.241.145:5043"]
  #index: "WAS_Filebeat-%{+YYYY.MM.dd}"
  # user: "logstash_system"
  #password: "logstash"
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

#====Processor===
processors:
- add_cloud_metadata:

need help to setup Analyzing WebSphere Application Server logs with Elastic Stack - IBM Documentation

2

Is it Logstash on the same IP? There no listener on port 5043, only 9200 - Elasticsearch.
Either is wrong IP or Logstash has not been started.

3

logstash is running on same node

4

Nothing on that port according to netstat

5

[elkadmin@ndc3vmappelk02 ~]$ netstat -an | grep 5043
tcp6 0 0 10.94.241.146:9200 10.94.241.150:50438 ESTABLISHED
[elkadmin@ndc3vmappelk02 ~]$ telnet
telnet> ^C
You have new mail in /var/spool/mail/elkadmin
[elkadmin@ndc3vmappelk02 ~]$ telnet 10.94.241.146 5043
Trying 10.94.241.146...
telnet: connect to address 10.94.241.146: Connection refused
[elkadmin@ndc3vmappelk02 ~]$ telnet 10.94.241.146 50438
Trying 10.94.241.146...
telnet: connect to address 10.94.241.146: Connection refused
[elkadmin@ndc3vmappelk02 ~]$ telnet 10.94.241.146 5044
Trying 10.94.241.146...
telnet: connect to address 10.94.241.146: Connection refused
[elkadmin@ndc3vmappelk02 ~]$ telnet localhost 5044
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

Yes , even if it is on other host , it is not allowing telnet

6

also , will it help, The URL of the SOCKS5 proxy to use when connecting to the Logstash servers. The value must be a URL with a scheme of socks5://. The protocol used to communicate to Logstash is not based on HTTP so a web-proxy cannot be used.

If the SOCKS5 proxy server requires client authentication, then a username and password can be embedded in the URL as shown in the example.

7

After changing port to 9600

Jan 15 00:40:20 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:20.102+0530 ERROR pipeline/output.go:121 Failed to publish events: client is not connected
Jan 15 00:40:20 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:20.102+0530 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://10.94.241.146:9600))
Jan 15 00:40:20 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:20.103+0530 INFO pipeline/output.go:105 Connection to backoff(async(tcp://10.94.241.146:9600)) established
Jan 15 00:40:20 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:20.131+0530 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp 10.94.241.145:58956->10.94.241.146:9600: write: connection reset by peer
Jan 15 00:40:21 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:21.969+0530 ERROR pipeline/output.go:121 Failed to publish events: write tcp 10.94.241.145:58956->10.94.241.146:9600: write: connection reset by peer
Jan 15 00:40:21 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:21.969+0530 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://10.94.241.146:9600))
Jan 15 00:40:21 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:21.970+0530 INFO pipeline/output.go:105 Connection to backoff(async(tcp://10.94.241.146:9600)) established
Jan 15 00:40:21 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:21.997+0530 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp 10.94.241.145:58962->10.94.241.146:9600: write: connection reset by peer

also check

[root@ndc3vmappelk05 filebeat]# cd /etc/logstash/conf.d/was_logstash.conf
-bash: cd: /etc/logstash/conf.d/was_logstash.conf: Not a directory
[root@ndc3vmappelk05 filebeat]# cat /etc/logstash/conf.d/was_logstash.conf
input {
beats {
port => "5044"
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => [ "https://ndc3vmappelk05.inroot.in:9200", "https://ndc3vmappelk06.inroot.in:9200", "https://ndc3vmappelk02.inroot.in:9200", "https://ndc3vmappelk03.inroot.in:9200", "https://ndc3vmappelk04.inroot.in:9200" ]
user => elastic
password => elastic
index => 'WAS_Filebeat-%{+YYYY.MM.dd}'
#cacert => '/etc/logstash/certs/ca.crt'
cacert => '/etc/logstash/cert_blog_new/ca/ca.crt'
}
}

8

@Sagar_Naik Please take to 10 seconds and format your code just select it and press the </> button! :slight_smile: I did it for you in your first post, many of us won't even try to answer questions that are so hard to read.

Now filebeat needs to point to the correct port.

output.logstash:
  # The Logstash hosts
  hosts: ["10.94.241.145:5044"] <!------ HERE put the correct port
  #index: "WAS_Filebeat-%{+YYYY.MM.dd}"
  # user: "logstash_system"
  #password: "logstash"
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

to match

input {
  beats {
    port => "5044"
  }
}

Notice how lovely it works when you format your code :slight_smile:

9

</>
Jan 15 00:40:20 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:20.103+0530 INFO pipeline/output.go:105 Connection to backoff(async(tcp://10.94.241.146:9600)) established
Jan 15 00:40:20 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:20.131+0530 ERROR logstash/async.go:256 Failed to publish events caused by: write tcp 10.94.241.145:58956->10.94.241.146:9600: write: connection reset by peer
Jan 15 00:40:21 ndc3vmappelk05 filebeat[10524]: 2023-01-15T00:40:21.969+0530 ERROR pipeline/output.go:121 Failed to publish events: write tcp 10.94.241.145:58956->10.94.241.146:9600: write: connection reset by peer</>

is 9600 port correct as logstash is working on it

10

9600 is for LS monitoring.
Follow stephenb instructions.

11

Hi ,

I have checked using </> but not working , do i need to press 3 buttons after < / > , also please check as i am using it in filebeat.yml for me logstash is output so logstash port i gave 9600 , and in logstash we need pnly input as below given. here i tried 5043,5044 but both are not working , any file i can grep to get correct port

input {
beats {
port => "5044"
}
}

12

Hi @Sagar_Naik

9600 is logstash management port not the beats port which is 5044

What I showed you above is correct...

You asked a question and we tell you the answer and the do not try/ follow...

Not much more we can do.

Try what a suggested...

Those settings are also in the docs.

And it showing in detail in the official documentation

13

Hi,

I changed it back to 5044 .
Its telnet not working on remote host 10.94.241.146 on port 5044 ,Is this telnet is the actual cause , do we need to get firewall open as self telnet is also not working and hence I did not installed filebeat on 146 node.

elkadmin@ndc3vmappelk02 ~]$ telnet 10.94.241.146 5044
Trying 10.94.241.146...
telnet: connect to address 10.94.241.146: Connection refused
[elkadmin@ndc3vmappelk02 ~]$ telnet localhost 5044
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

14

If you start logstash...Can you telnet to 5044 on the same server that you are running logstash?

The logstash logs should show successfully opening the port... Did you look?

If you can connect to 5044 on the same server running logstash... But can not connect from another server that would indicate a connectivity/ firewall issue.

Here is my simple logstash config that listens to beats it is the sample in the config directory

$ cat config/logstash-sample.conf 
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

It will start and you should see log lines like

[2023-01-14T22:06:33,067][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.4}
[2023-01-14T22:06:33,079][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-01-14T22:06:33,087][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-01-14T22:06:33,142][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-01-14T22:06:33,156][INFO ][org.logstash.beats.Server][main][445fc32757e644ee24a7914982f2ee4ac7f879da469d77c75ea576f4809bdc43] Starting server on port: 5044

Then telnet connects from the same server with localhost, 127.0.0.1 or the server IP

$ telnet 192.168.86.40 5044
Trying 192.168.86.40...
Connected to 192.168.86.40.
Escape character is '^]'.

BTW to format code or log lines select it and the press

Screen Shot 2023-01-14 at 10.05.56 PM
Screen Shot 2023-01-14 at 10.05.56 PM1530×1248 157 KB

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.