# Probleme de synchronisation entre logstash et kibana **URL:** \ **Category:** Discussions en français\ **Created:** [February 12, 2021, 10:52am UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107 "2021-02-12T10:52:26Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![hugoboubou](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Post date:** [February 12, 2021, 10:52am UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/1 "2021-02-12T10:52:26Z")
Bonjour à tous, Je souhaite analyser différents types de logs. Qui proviennent de plusieurs agents beats. Actuellement j'ai deux patterns grok pour les traiter. Les voici : match =\> { "message" =\> "(?\%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}) %{WORD:VCS} %{LOGLEVEL:logLevel} %{GREEDYDATA:logMessage}"} match =\> { "message" =\> "%{DATE\_EU:mytimestamp} %{TIME:temps} %{WORD:serveur} %{GREEDYDATA:logMessage}"} Les groks sont justes, je les ai testé sur grok debugger 🙂 **Mon probleme est le suivant :** Au debut de mon projet j'avais simplement un pattern, et dans kibana je pouvais filtrer les données et faire un graphique en fonction de la variable "logLevel". Cela me permettait de compter les logs "error" par exemple. Actuellement j'aimerai faire de meme avec la variable "serveur" mais celle ci n'est pas crée dans kibana, c'est comme ci elle n'existait pas. Alors qu'elle est dans mon grok... 😔 Pensez vous pouvoir m'aider ? Je me tiens disponible si vous avez besoin de plus d'informations 😃 Hugo ---
**Author:** ![alejandrosl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alejandrosl/32/83830_2.png) [@alejandrosl](https://discuss.elastic.co/u/alejandrosl)\ **Post date:** [February 12, 2021, 11:31am UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/2 "2021-02-12T11:31:18Z")
> [@hugoboubou](#): > > %{WORD:serveur} Hello Hugo !!! Could you please provide the logstash pipeline config? I ask you that because when you use two different grok patterns to match, you have to use array for example: ```auto grok { match => {"message" => [ "%{TIMESTAMP_ISO8601:timestampIn} %{TIMESTAMP_ISO8601:timestampOut}%{SPACE}%{IP:IPV4}", "%{TIMESTAMP_ISO8601:timestampIn} %{TIMESTAMP_ISO8601:timestampOut}%{SPACE}(?[a-zA-Z0-9._-]+)%{SPACE}%{IP:IPV4}" ] } } ``` and you can check on your kibana discovery sections, filtering the docs, to see if is there any tag "\_grokparsefailure" that add when the grok is not working. With that we can start helping you. Kind regards Ale ---
**Author:** ![hugoboubou](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Post date:** [February 12, 2021, 12:10pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/3 "2021-02-12T12:10:34Z")
> [@alejandrosl](#): > > \_grokparsefailure Thanks for your reply, i have change my logstash filter with your proposition and in my kibana I have had a filter : tags is tags "\_grokparsefailure". I got a lot of results. So I think there is a problem with my grok, but it compiles in the grok debugger... This is my pipelines.yml : - pipeline.id: main path.config: "/etc/logstash/conf.d/\*.conf" and this is my logstash.conf : ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a6468012688afc00162785568ff8df9252301277.png) ---
**Author:** ![alejandrosl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alejandrosl/32/83830_2.png) [@alejandrosl](https://discuss.elastic.co/u/alejandrosl)\ **Post date:** [February 12, 2021, 12:16pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/4 "2021-02-12T12:16:50Z")
This grok are to match two kind of messages. Could you please share one example document that have the "\_grokparsefailure" ? Thats means the match you use in the grok filtering is not working, I mean you have another kind of message thats no match ... ---
**Author:** ![hugoboubou](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Post date:** [February 12, 2021, 12:33pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/5 "2021-02-12T12:33:45Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/e/d/edb575b45db84df3c01bc3056581a3c64c130b8a.png) but my problem is... why logstash parse this file ? logs in gc.log is like : ` > [2021-02-12T12:27:32.922+0000][7923][safepoint] Safepoint "Cleanup", Time since last: 2000532842 ns, Reaching safepoint: 180616 ns, At safepoint: 9000 ns, Total: 189616 ns ` and the file that i would like parse is : > ``` > 2017/07/27 18:02:37 VCS INFO V-16-6-15015 (rsxlXXXX) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed > 2017/07/27 18:02:38 VCS INFO V-16-1-50159 User fired command: hares -clear XXXXXXXX_edatdbe from node rsxlXXXXX > 02/11/2021 03:18:54 ANSXXXXX TDPO Linux86-64 ANXXXXX TDP for Oracle: (XXXXX): =>() AXXXXXX The object /rXXXXXXXX_centric/ /k7vmsjds_1_1.bck was not found on the IXX Spectrum Protect Server > 02/11/2021 03:21:54 XXXXXXXXI DIAG: sessSendVerb: Error sending Verb, rc: -50 > 02/11/2021 03:21:54 AXXXXXXE Session rejected: TCP/IP connection failure. > 02/11/2021 03:21:54 XXXXXXXX Session rejected: TCP/IP connection failure. > > ``` I parse this file with logstash.conf. I showed it to you earlier ---
**Author:** ![alejandrosl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alejandrosl/32/83830_2.png) [@alejandrosl](https://discuss.elastic.co/u/alejandrosl)\ **Post date:** [February 12, 2021, 12:52pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/6 "2021-02-12T12:52:05Z")
I try the groks pattern and the only one that works is te second, so try to leve only that before... and I get some confuse with the gc.log (garbage collector) of elasticsearch, could you please show all the message, when a \_grokparsefailure happen the doc on kibana have a full "message" field with the raw message ---
**Author:** ![hugoboubou](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Post date:** [February 12, 2021, 12:55pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/7 "2021-02-12T12:55:50Z")
> [@alejandrosl](#): > > I try the groks pattern and the only one that works is te second, so try to leve only that before... and > I get some confuse with the gc.log (garbage collector) of elasticsearch, could you please show all the message, when a \_grokparsefailure happen the doc on kibana have a full "message" field with the raw message for kibana message : ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ff6435459bebb94ae87d9236f62414c166376d0d.png) ---
**Author:** ![hugoboubou](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Post date:** [February 12, 2021, 1:06pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/8 "2021-02-12T13:06:33Z")
and im confused because in grok debugger my first pattern work on this ` > 2017/07/27 18:02:37 VCS INFO V-16-6-15015 (rsxlXXXX) hatrigger:/opt/VRTSvcs/bin/triggers/resfault is not a trigger scripts directory or can not be executed ` and my second pattern work on this kind of log `> 02/11/2021 03:21:54 AXXXXXXE Session rejected: TCP/IP connection failure` ---
**Author:** ![alejandrosl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alejandrosl/32/83830_2.png) [@alejandrosl](https://discuss.elastic.co/u/alejandrosl)\ **Post date:** [February 12, 2021, 2:30pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/9 "2021-02-12T14:30:55Z")
There is the problem, you are grokin some logs and try to grokin to the GC logs and this (as you can see in the message field) dont match with your match ---
**Author:** ![hugoboubou](https://avatars.discourse-cdn.com/v4/letter/h/ac8455/32.png) [@hugoboubou](https://discuss.elastic.co/u/hugoboubou)\ **Post date:** [February 12, 2021, 3:25pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/10 "2021-02-12T15:25:36Z")
I just understood one thing. I changed the index of my logstash.conf. I was created a new index pattern in kibana and kibana offered me to choose a time fields ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/0/70052bae1f2a09fce001f815a56d1987b89f4456.png) When i choose "REQ\_TIME" I got vcs filters, serveur filters and another informations from grok. but now, I can't find any more logs lol ! I have the filters but not the logs ---
**Author:** ![alejandrosl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alejandrosl/32/83830_2.png) [@alejandrosl](https://discuss.elastic.co/u/alejandrosl)\ **Post date:** [February 12, 2021, 4:39pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/11 "2021-02-12T16:39:07Z")
Well I think you need before all segment the origin of data. First you can take a look to this blog that teach you how to separate in different logstash config pipelines to adapt the filtering. why? because as I see before, you have almost 3 different origin, thats true? - Logs that start with YYYY/MM/DD (VCS) - Logs that start with MM/DD/YYYY - GC logs (garbage collector) > **[How to create maintainable and reusable Logstash pipelines](https://www.elastic.co/blog/how-to-create-maintainable-and-reusable-logstash-pipelines)** > > Logstash is a powerful processing and transformation pipeline, and some implementations may have many lines of code. Learn how to make your pipeline code more maintainable and reusable by creating modular components. More: If you are using just one filebeat to collect the data, as I know, Unfortunately, running multiple outputs in Filebeat is not supported, so one easy and good options are create a separated services to run the filebeat with different config files, for example ( **just in linux** ) ( **please adapt it** ): copy the original filebeat service into a new other: ```bash root@laboratory:~# cd /lib/systemd/system root@laboratory:/lib/systemd/system# cp -aR filebeat.service filebeat-vcs.service root@laboratory:/lib/systemd/system# ls -lhtra filebeat* -rw-r--r-- 1 root root 616 Jun 14 20:19 filebeat.service -rw-r--r-- 1 root root 637 Aug 10 09:23 filebeat-vcs.service ``` adapt the the file to run the filebeat with a new config file: ```bash [Unit] Description=Filebeat sends log files to Logstash or directly to Elasticsearch. Documentation=https://www.elastic.co/products/beats/filebeat Wants=network-online.target After=network-online.target [Service] Environment="BEAT_LOG_OPTS=" Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat-vcs.yml" Environment="BEAT_PATH_OPTS=-path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat/vcs -path.logs /var/log/filebeat/vcs" ExecStart=/usr/share/filebeat/bin/filebeat -environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS Restart=always [Install] WantedBy=multi-user.target ``` Now in the **/etc/filebeat/filebeat-vcs.yml** you can add another input file and modify the output to the new logstash pipeline. When you will get the source data separated, you can adapt better the filtering and the output to elastic in diferente index to make better troubleshooting and management of data. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 12, 2021, 4:39pm UTC](https://discuss.elastic.co/t/probleme-de-synchronisation-entre-logstash-et-kibana/264107/12 "2021-03-12T16:39:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.