# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=215 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 216 --- ## [Logstash stucks if Index is not available](https://discuss.elastic.co/t/logstash-stucks-if-index-is-not-available/277173)
**Author:** [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Replies:** 1\ **Last updated:** [June 29, 2021, 7:23am UTC](https://discuss.elastic.co/t/logstash-stucks-if-index-is-not-available/277173 "2021-06-29T07:23:08Z")
Hello Team I am using Elasticsearch version 7.8.0 and Logstash 7.8.0. I am trying to extract data from Elasticsearch index to csv file. I am using a script to perform the activity for various different indexes created… --- ## [Use Office 365 module in Logstash?](https://discuss.elastic.co/t/use-office-365-module-in-logstash/276528)
**Author:** [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Replies:** 3\ **Last updated:** [June 29, 2021, 7:21am UTC](https://discuss.elastic.co/t/use-office-365-module-in-logstash/276528 "2021-06-29T07:21:00Z")
Hello Is it possible to use the Office 365 module with Logstash? Thanks --- ## [File input doesn t read incoming Json file from web application but when modified everythnig is fine](https://discuss.elastic.co/t/file-input-doesn-t-read-incoming-json-file-from-web-application-but-when-modified-everythnig-is-fine/277284)
**Author:** [@hadii](https://discuss.elastic.co/u/hadii)\ **Replies:** 0\ **Last updated:** [June 29, 2021, 4:56am UTC](https://discuss.elastic.co/t/file-input-doesn-t-read-incoming-json-file-from-web-application-but-when-modified-everythnig-is-fine/277284 "2021-06-29T04:56:27Z")
I have to project in rails which have huge logs, I want to parse them in logstash and kibana, the problem is when logs do from web application logstash could not pars them but after modified them even very unused change … --- ## [Get datetime from logs](https://discuss.elastic.co/t/get-datetime-from-logs/277175)
**Author:** [@Rashmika\_Gamage](https://discuss.elastic.co/u/Rashmika_Gamage)\ **Replies:** 2\ **Last updated:** [June 29, 2021, 3:45am UTC](https://discuss.elastic.co/t/get-datetime-from-logs/277175 "2021-06-29T03:45:10Z")
I am trying to get logs to Kibana using logstash and filebeats. However kibana doesn't take data and time (which in the logs) as datetime. Instead in takes as string and cannot change format as well . Appreciate your he… --- ## [Logstash Create new field from existing field and remove character from new field](https://discuss.elastic.co/t/logstash-create-new-field-from-existing-field-and-remove-character-from-new-field/277268)
**Author:** [@Annette1](https://discuss.elastic.co/u/Annette1)\ **Replies:** 4\ **Last updated:** [June 28, 2021, 9:16pm UTC](https://discuss.elastic.co/t/logstash-create-new-field-from-existing-field-and-remove-character-from-new-field/277268 "2021-06-28T21:16:45Z")
I have a simple request but seem unable to be able execute it correctly. I have a field named logname that has several name forms. For instance are logname-app, logname-os, logname-access, and logname. The logname is … --- ## [Output Using CEF codec and to Kafka Server](https://discuss.elastic.co/t/output-using-cef-codec-and-to-kafka-server/277243)
**Author:** [@Wilks](https://discuss.elastic.co/u/Wilks)\ **Replies:** 4\ **Last updated:** [June 28, 2021, 6:34pm UTC](https://discuss.elastic.co/t/output-using-cef-codec-and-to-kafka-server/277243 "2021-06-28T18:34:01Z")
Hi, Is there a way to nest the Output statement so that I can first output using the CEF codec and then Output to a Kafka topic. The requirement is that I have to covert the data to CEF and then send to a Kafka topic. … --- ## [Can not purge Logstash](https://discuss.elastic.co/t/can-not-purge-logstash/277225)
**Author:** [@Vladimir](https://discuss.elastic.co/u/Vladimir)\ **Replies:** 2\ **Last updated:** [June 28, 2021, 5:44pm UTC](https://discuss.elastic.co/t/can-not-purge-logstash/277225 "2021-06-28T17:44:17Z")
Hi, can somebody help me please to remove logstash from Ubuntu? I had a broken Elastic + Kibana + Logstash + Filebeat installtion so I decided to remove it completely and start installation again. All packages are gone … --- ## [Grok pattern not working with spaces](https://discuss.elastic.co/t/grok-pattern-not-working-with-spaces/277254)
**Author:** [@kriss332](https://discuss.elastic.co/u/kriss332)\ **Replies:** 2\ **Last updated:** [June 28, 2021, 4:54pm UTC](https://discuss.elastic.co/t/grok-pattern-not-working-with-spaces/277254 "2021-06-28T16:54:40Z")
Hi all , I am trying to get a pattern for tbelow log in GrokDebugger site. Jun 27 00:00:12 location2-squid2 SQUID 1624732203.029 877 172.1.1.1 TCP\_MISS/200 70205 GET http://10.10.1.1/v1/loading/summary? - HIER\_DIRECT… --- ## [Problem install logstash-7.13.2-x86](https://discuss.elastic.co/t/problem-install-logstash-7-13-2-x86/277249)
**Author:** [@mylogstash](https://discuss.elastic.co/u/mylogstash)\ **Replies:** 2\ **Last updated:** [June 28, 2021, 4:22pm UTC](https://discuss.elastic.co/t/problem-install-logstash-7-13-2-x86/277249 "2021-06-28T16:22:53Z")
On Centos 7.9 During upgrade of logstash-7.13.2-x86 the ownership /usr/share/logstash changes from logstash:logstash to root:root. Why does the installation changes the owner of /usr/share/logstash ? --- ## [Logstash ERROR with CVS parsing in a Pod](https://discuss.elastic.co/t/logstash-error-with-cvs-parsing-in-a-pod/277180)
**Author:** [@jsu](https://discuss.elastic.co/u/jsu)\ **Replies:** 2\ **Last updated:** [June 28, 2021, 8:41am UTC](https://discuss.elastic.co/t/logstash-error-with-cvs-parsing-in-a-pod/277180 "2021-06-28T08:41:13Z")
Hey there, I'm currently trying to automatically ingest a CVS when launching a Logstash pod connected to ECK and I have the following error: \[ERROR\] 2021-06-28 07:25:43.494 \[Converge PipelineAction::Create\\] agent… --- ## [How to run configured pipeline.yml in logstash](https://discuss.elastic.co/t/how-to-run-configured-pipeline-yml-in-logstash/276072)
**Author:** [@sudo-ranjith](https://discuss.elastic.co/u/sudo-ranjith)\ **Replies:** 9\ **Last updated:** [June 28, 2021, 6:27am UTC](https://discuss.elastic.co/t/how-to-run-configured-pipeline-yml-in-logstash/276072 "2021-06-28T06:27:41Z")
I have configured 2 logstash.conf file in pipelines.yml file, pipeline.id: sys\_log\_1 pipeline.workers: 2 path.config: "D:/Elastic/Logstash/7.12.1/config/logstash\_sys\_1.conf" pipeline.id: sys\_log\_2 pipeline.workers: … --- ## [Load balancer for multi logstash](https://discuss.elastic.co/t/load-balancer-for-multi-logstash/277071)
**Author:** [@alicango](https://discuss.elastic.co/u/alicango)\ **Replies:** 2\ **Last updated:** [June 27, 2021, 8:16pm UTC](https://discuss.elastic.co/t/load-balancer-for-multi-logstash/277071 "2021-06-27T20:16:17Z")
Hi, We have 2 Logstash servers. I want to put them behind a load balancer and push the data to the logstash over the HTTPS. But now, I am using the HTTP input plugin without a load balancer. Let me show you my HTTPS co… --- ## [XML Logstash](https://discuss.elastic.co/t/xml-logstash/277160)
**Author:** [@leggomyego](https://discuss.elastic.co/u/leggomyego)\ **Replies:** 1\ **Last updated:** [June 27, 2021, 7:21pm UTC](https://discuss.elastic.co/t/xml-logstash/277160 "2021-06-27T19:21:27Z")
Custom fields have been tricky because they are in an array. I can rename them fine and everything, but I am looking for a Ruby answer to my issue. For instance, if the field names are \[test\]\[test\]\[0\]\[test\], \[test\]\[test\]… --- ## [Logstash error](https://discuss.elastic.co/t/logstash-error/277142)
**Author:** [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\ **Replies:** 2\ **Last updated:** [June 27, 2021, 2:08pm UTC](https://discuss.elastic.co/t/logstash-error/277142 "2021-06-27T14:08:13Z")
Hello team, I am getting following error on logstash pipeline. can you please help me to identify exact issue. Logstash version: 7.9 C:\\Users\\mangeshsuresh.jadhav\\Downloads\\EKL\\logstash-7.9.0\>bin\\logstash -f wubsprod.… --- ## [Multiline parser](https://discuss.elastic.co/t/multiline-parser/277124)
**Author:** [@bz\_Os](https://discuss.elastic.co/u/bz_Os)\ **Replies:** 2\ **Last updated:** [June 27, 2021, 12:29pm UTC](https://discuss.elastic.co/t/multiline-parser/277124 "2021-06-27T12:29:46Z")
Hello, I am glad to joint the elastic stack community, to pars SQL logs I need to put together rows into one: juin 26 13:30:05 sqlpc mysql-slow: SELECT fk.pib, fk.id, fk.idAccount, fk.idVad, fk.somevalue, fk.somevalue… --- ## [Logstash twitter plugin parses wrong user id](https://discuss.elastic.co/t/logstash-twitter-plugin-parses-wrong-user-id/277135)
**Author:** [@Lukasz\_Geras](https://discuss.elastic.co/u/Lukasz_Geras)\ **Replies:** 2\ **Last updated:** [June 26, 2021, 9:52pm UTC](https://discuss.elastic.co/t/logstash-twitter-plugin-parses-wrong-user-id/277135 "2021-06-26T21:52:39Z")
Hi, quick question regarding parsing tweets using logstash official plugin. I had a problem parsing tweets, that comes from account other than mine. As you see, using a online converter, I identified my real twitter ID: … --- ## [Logstash not created indexes](https://discuss.elastic.co/t/logstash-not-created-indexes/276277)
**Author:** [@Ilya\_21bA](https://discuss.elastic.co/u/Ilya_21bA)\ **Replies:** 2\ **Last updated:** [June 26, 2021, 9:17pm UTC](https://discuss.elastic.co/t/logstash-not-created-indexes/276277 "2021-06-26T21:17:57Z")
Hi, pls tell me why new indexes not not displayed in Kibana logstash.json: input { file { type =\> "pikautotesttc4" path =\> "C:/Users/bezzbtsev/Desktop/pik2/RTS-PIK/dms-selenium-tests/TestSelenium/bin/Debug/Log… --- ## [Grok filter](https://discuss.elastic.co/t/grok-filter/276671)
**Author:** [@Sarthak\_Mishra](https://discuss.elastic.co/u/Sarthak_Mishra)\ **Replies:** 13\ **Last updated:** [June 26, 2021, 1:50pm UTC](https://discuss.elastic.co/t/grok-filter/276671 "2021-06-26T13:50:56Z")
Hello, I want to create a grok filter. e.g There is a column in my csv file with all the os system listed windows,ubuntu etc. If there is any typo in the column. I want logstash to create a new field with the name "is\_o… --- ## [Add more parameter to kafka input](https://discuss.elastic.co/t/add-more-parameter-to-kafka-input/277109)
**Author:** [@kannan\_raj](https://discuss.elastic.co/u/kannan_raj)\ **Replies:** 0\ **Last updated:** [June 26, 2021, 5:36am UTC](https://discuss.elastic.co/t/add-more-parameter-to-kafka-input/277109 "2021-06-26T05:36:44Z")
Hello, Logstash version is 7.9 and kafka version is 2.5.0 could you please confirm that can we change the below parameter in kafka input plugin? allow.auto.create.topics (default is true) group.instance.id … --- ## [Get rid of .keyword](https://discuss.elastic.co/t/get-rid-of-keyword/277105)
**Author:** [@svenvg93](https://discuss.elastic.co/u/svenvg93)\ **Replies:** 1\ **Last updated:** [June 25, 2021, 9:36pm UTC](https://discuss.elastic.co/t/get-rid-of-keyword/277105 "2021-06-25T21:36:00Z")
Hi, I'm still a ELK noob so bear with me here ;). I've setup Logstash to send the output of fping command to Elasticsearch and add some geoip data to it. Fping output; ec2.eu-west-1.amazonaws.com : xmt/rcv/%loss = 3… --- ## [Unreadable "Source: (byte\[\])" gives input\_coercion\_exception](https://discuss.elastic.co/t/unreadable-source-byte-gives-input-coercion-exception/277104)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 1\ **Last updated:** [June 25, 2021, 9:26pm UTC](https://discuss.elastic.co/t/unreadable-source-byte-gives-input-coercion-exception/277104 "2021-06-25T21:26:24Z")
I'm receiving the following warning in my Logstash container. I think it's because there may be some malformed/corrupt data coming through. \[WARN \] 2021-06-25 20:38:37.941 \[\[test\]\>worker0\] elasticsearch - Could not ind… --- ## [Logstash pipeline](https://discuss.elastic.co/t/logstash-pipeline/277082)
**Author:** [@Sarthak\_Mishra](https://discuss.elastic.co/u/Sarthak_Mishra)\ **Replies:** 2\ **Last updated:** [June 25, 2021, 5:40pm UTC](https://discuss.elastic.co/t/logstash-pipeline/277082 "2021-06-25T17:40:11Z")
My logstash does not ship the data. Please help input { file { path =\> "C:/Users/kumar/Desktop/grok.csv" start\_position =\> "beginning" sincedb\_path =\> "NULL" } } filter { csv { separator =\> "," columns =\> \["Na… --- ## [How to grok pattern match middle of a string?](https://discuss.elastic.co/t/how-to-grok-pattern-match-middle-of-a-string/276708)
**Author:** [@mohsin106](https://discuss.elastic.co/u/mohsin106)\ **Replies:** 2\ **Last updated:** [June 25, 2021, 4:51pm UTC](https://discuss.elastic.co/t/how-to-grok-pattern-match-middle-of-a-string/276708 "2021-06-25T16:51:04Z")
I have a string that looks like this: SITE;500400;PEER FLOW (AS 3535);;100G;MR;PP:10.02.01.91 97,98; I just want to extract "FLOW" into a field named peer. I tried doing this: (?\\\w+\\D\\d+\\D\\w+\\s(\\w+)) But tha… --- ## [Date filter problems after upgrade](https://discuss.elastic.co/t/date-filter-problems-after-upgrade/274841)
**Author:** [@Renato](https://discuss.elastic.co/u/Renato)\ **Replies:** 1\ **Last updated:** [June 25, 2021, 4:25pm UTC](https://discuss.elastic.co/t/date-filter-problems-after-upgrade/274841 "2021-06-25T16:25:19Z")
Hey, we are having problems after we upgraded elk to 7.12.1 (I'm not sure which version we previously used 7.xx). The date filter which we previously used is not working anymore. We don't get any logs in elastic. If I w… --- ## [Data Logstash filter not putting month into @timestamp](https://discuss.elastic.co/t/data-logstash-filter-not-putting-month-into-timestamp/277079)
**Author:** [@adwearys](https://discuss.elastic.co/u/adwearys)\ **Replies:** 2\ **Last updated:** [June 25, 2021, 3:46pm UTC](https://discuss.elastic.co/t/data-logstash-filter-not-putting-month-into-timestamp/277079 "2021-06-25T15:46:01Z")
Hi, I'm trying to take the date and time from a syslog message and put it into the @timestamp field (so the @timestamp field has the date and time that the source system created the message, rather than when it arrived … --- ## [Logstash error](https://discuss.elastic.co/t/logstash-error/277048)
**Author:** [@Ashwin\_Patil1](https://discuss.elastic.co/u/Ashwin_Patil1)\ **Replies:** 2\ **Last updated:** [June 25, 2021, 3:42pm UTC](https://discuss.elastic.co/t/logstash-error/277048 "2021-06-25T15:42:44Z")
I have installed logstash and the following is the .conf file input { beat { port =\> 5044 type =\> syslog } } filter { if \[type\] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} … --- ## [CSV File re-reading/re-parsing issue 7.10](https://discuss.elastic.co/t/csv-file-re-reading-re-parsing-issue-7-10/276940)
**Author:** [@Rajesh\_S](https://discuss.elastic.co/u/Rajesh_S)\ **Replies:** 3\ **Last updated:** [June 25, 2021, 3:36pm UTC](https://discuss.elastic.co/t/csv-file-re-reading-re-parsing-issue-7-10/276940 "2021-06-25T15:36:07Z")
I am trying to re-read a csv file from the beginning using sincedb set to '/dev/null'. logstash is started using --config.reload.automatic. All entries are blank when the file is re-read. This is not consistent, some tim… --- ## [How to make filter in beat on the logstash](https://discuss.elastic.co/t/how-to-make-filter-in-beat-on-the-logstash/276992)
**Author:** [@Juan\_David\_Jaramillo](https://discuss.elastic.co/u/Juan_David_Jaramillo)\ **Replies:** 3\ **Last updated:** [June 25, 2021, 2:48pm UTC](https://discuss.elastic.co/t/how-to-make-filter-in-beat-on-the-logstash/276992 "2021-06-25T14:48:04Z")
Hi, I need to create a filter to avoid duplicating the information that I have from one index to another, in the configuration of the logstash file where the data of the first index that I upload to elastic are, I alread… --- ## [Logstash Pipeline which works fine for file input but not working when set with filebeats](https://discuss.elastic.co/t/logstash-pipeline-which-works-fine-for-file-input-but-not-working-when-set-with-filebeats/277068)
**Author:** [@MuskanBeig](https://discuss.elastic.co/u/MuskanBeig)\ **Replies:** 1\ **Last updated:** [June 25, 2021, 1:42pm UTC](https://discuss.elastic.co/t/logstash-pipeline-which-works-fine-for-file-input-but-not-working-when-set-with-filebeats/277068 "2021-06-25T13:42:12Z")
My logstash pipeline works fine with the input as file or generator plugin But the same pipeline is not working when i use beats as input plugin the filebeat is working properly --- ## [Logstash with Point to Point](https://discuss.elastic.co/t/logstash-with-point-to-point/277047)
**Author:** [@Michael\_Dylan\_McAloo](https://discuss.elastic.co/u/Michael_Dylan_McAloo)\ **Replies:** 0\ **Last updated:** [June 25, 2021, 11:03am UTC](https://discuss.elastic.co/t/logstash-with-point-to-point/277047 "2021-06-25T11:03:07Z")
In Kibana I try to use the "Point to Point" map but it doesn't show me anything because I only have two fields called "geoip.location" and "location" and I don't have the destination field to filter and "geoip.location" … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=214) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=216)