# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=960 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 961 --- ## [Not Able to Create Geopoint data](https://discuss.elastic.co/t/not-able-to-create-geopoint-data/89597)
**Author:** [@wlin](https://discuss.elastic.co/u/wlin)\ **Replies:** 5\ **Last updated:** [June 21, 2017, 8:26pm UTC](https://discuss.elastic.co/t/not-able-to-create-geopoint-data/89597 "2017-06-21T20:26:25Z")
Hi I'm trying to make some map based on some ip address which are private, so I cannot use geoip filter here. Currently I'm implementing some naive method to assign the (lat, lon) coordinates using some if else condition… --- ## [Crashing Logstash S3 output - NotImplementedError fstat error at startup in 5.2+ on Alpine](https://discuss.elastic.co/t/crashing-logstash-s3-output-notimplementederror-fstat-error-at-startup-in-5-2-on-alpine/90338)
**Author:** [@timecode](https://discuss.elastic.co/u/timecode)\ **Replies:** 0\ **Last updated:** [June 21, 2017, 5:48pm UTC](https://discuss.elastic.co/t/crashing-logstash-s3-output-notimplementederror-fstat-error-at-startup-in-5-2-on-alpine/90338 "2017-06-21T17:48:05Z")
OS: Alpine 3.6 Logstash 5.2+ Plugin: logstash-output-s3 Java: OpenJDK-1.8.0.131-b11 Prior to Logstash 5.2, context.http\_request.body (as seen in /usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/… --- ## [Update counter while reading logs](https://discuss.elastic.co/t/update-counter-while-reading-logs/90352)
**Author:** [@THILAK\_thilz](https://discuss.elastic.co/u/THILAK_thilz)\ **Replies:** 0\ **Last updated:** [June 21, 2017, 7:23pm UTC](https://discuss.elastic.co/t/update-counter-while-reading-logs/90352 "2017-06-21T19:23:15Z")
I am trying to update document while reading logs, If I get a same log again the document field counter should be update by 1. I need the syntax for the conf file. Please help --- ## [Logstash encoding (previously in Kibana encoding)](https://discuss.elastic.co/t/logstash-encoding-previously-in-kibana-encoding/90141)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 9\ **Last updated:** [June 21, 2017, 6:43pm UTC](https://discuss.elastic.co/t/logstash-encoding-previously-in-kibana-encoding/90141 "2017-06-21T18:43:34Z")
Is it possible to change the charset that Kibana reports in? I am having issues where Kibana reports the replacement character when it is supposed to be an oslash. I have changed the encoding charset in both the input fi… --- ## [Deserializing a json string and make it a name in an index to ES](https://discuss.elastic.co/t/deserializing-a-json-string-and-make-it-a-name-in-an-index-to-es/90328)
**Author:** [@EigoCogito](https://discuss.elastic.co/u/EigoCogito)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 6:31pm UTC](https://discuss.elastic.co/t/deserializing-a-json-string-and-make-it-a-name-in-an-index-to-es/90328 "2017-06-21T18:31:33Z")
Hello People, I'm relatively new to the ELK stack and have made progress in getting data into ES, however I would really appreciate some pointers on the following: My setup is sending data to kafka and then to ES thr… --- ## [Logstash missing logs from s3 bucket](https://discuss.elastic.co/t/logstash-missing-logs-from-s3-bucket/90333)
**Author:** [@sarthak2016](https://discuss.elastic.co/u/sarthak2016)\ **Replies:** 0\ **Last updated:** [June 21, 2017, 4:59pm UTC](https://discuss.elastic.co/t/logstash-missing-logs-from-s3-bucket/90333 "2017-06-21T16:59:51Z")
Hello All, I am importing around 500 CSV's into "test" bucket every hourly, But when i verify data/hits into Kibana, Half of logs are missing, When i checked CSV's into S3, All CSV's are exits, Also Logstash script is r… --- ## [Logstash S3 input + Looking Path object](https://discuss.elastic.co/t/logstash-s3-input-looking-path-object/57843)
**Author:** [@sarthak2016](https://discuss.elastic.co/u/sarthak2016)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 5:00pm UTC](https://discuss.elastic.co/t/logstash-s3-input-looking-path-object/57843 "2017-06-21T17:00:47Z")
Hello All, I have replace my old input with Amazon S3 input because i Need to read CSVs from S3 bucket instead of my local drive. Old Input syntax (Read CSVs from my local) input { file { sincedb\_path =\> '/… --- ## [How to uniquely identify data from different sources](https://discuss.elastic.co/t/how-to-uniquely-identify-data-from-different-sources/90151)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 12\ **Last updated:** [June 21, 2017, 3:40pm UTC](https://discuss.elastic.co/t/how-to-uniquely-identify-data-from-different-sources/90151 "2017-06-21T15:40:25Z")
I have different sources of data that provide the same type of logs. I would like to be able to uniquely identify where each log file came from. Here is an example of what I am talking about: There are three sources na… --- ## [Logstash service doesn't output logs, command line does](https://discuss.elastic.co/t/logstash-service-doesnt-output-logs-command-line-does/90294)
**Author:** [@leoponton](https://discuss.elastic.co/u/leoponton)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 3:28pm UTC](https://discuss.elastic.co/t/logstash-service-doesnt-output-logs-command-line-does/90294 "2017-06-21T15:28:12Z")
If I start the logstash service using service or systemctl, it starts and reports as running but produces no output that is seen by elasticsearch. On the other hand, if I run logstash using ./bin/logstash -f /etc/log… --- ## [Match and extract last part of log in pipe delimited log](https://discuss.elastic.co/t/match-and-extract-last-part-of-log-in-pipe-delimited-log/90086)
**Author:** [@Pranav1](https://discuss.elastic.co/u/Pranav1)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 2:54pm UTC](https://discuss.elastic.co/t/match-and-extract-last-part-of-log-in-pipe-delimited-log/90086 "2017-06-21T14:54:19Z")
Logstash is receving following log - \> Timestamp|Field1|Field2|...|Field n|Log message The number of fields is varaible and not fixed. My goal is to extract just the Timestamp and Log message and add it as a new key v… --- ## [Best approach to parse key value inside groups](https://discuss.elastic.co/t/best-approach-to-parse-key-value-inside-groups/90221)
**Author:** [@Laurent\_Barbisan](https://discuss.elastic.co/u/Laurent_Barbisan)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 1:31pm UTC](https://discuss.elastic.co/t/best-approach-to-parse-key-value-inside-groups/90221 "2017-06-21T13:31:29Z")
Hi, We are starting using logstash intensively and now we have a very specific line of log to parse : 2017-06-16 13:17:59.156 \[26\] \[Statistics\] (INFO) category1:subcategory1\[load=7%, physicalfree=11487488, swapfree… --- ## [Xml processing using logstash 5.4](https://discuss.elastic.co/t/xml-processing-using-logstash-5-4/90261)
**Author:** [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 12:19pm UTC](https://discuss.elastic.co/t/xml-processing-using-logstash-5-4/90261 "2017-06-21T12:19:24Z")
I have one xml file to process: e.g. \ \ \ \xxx\ \ \1\ \<\*\*firstView\*\*\> \6604\ \ \ \0\ \X… --- ## [How to use multiple csv files in logstash ,](https://discuss.elastic.co/t/how-to-use-multiple-csv-files-in-logstash/90200)
**Author:** [@vijjun123](https://discuss.elastic.co/u/vijjun123)\ **Replies:** 9\ **Last updated:** [June 21, 2017, 12:10pm UTC](https://discuss.elastic.co/t/how-to-use-multiple-csv-files-in-logstash/90200 "2017-06-21T12:10:25Z")
I want to use multiple csv files in logstash hence please guide me . now using single csv file as per the below. remaining file names are different ex : file1,file2,file3 input { file { path =\> "/tmp/AWSDiscove… --- ## [ELK Satelit and Master Installation](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312)
**Author:** [@DanMa](https://discuss.elastic.co/u/DanMa)\ **Replies:** 14\ **Last updated:** [June 21, 2017, 12:00pm UTC](https://discuss.elastic.co/t/elk-satelit-and-master-installation/41312 "2017-06-21T12:00:51Z")
Hello together, i want to make a big ELK Installation. At the one site we want have logstash Redis and Logstash Forwarder (Satelit) at the other site we want have Logstash Redis Logstash Elasticsearch Kibana (Master)… --- ## [Lostash - Command line - logstash -e 'input { stdin { } } output { stdout {} }' does nothing!](https://discuss.elastic.co/t/lostash-command-line-logstash-e-input-stdin-output-stdout-does-nothing/90229)
**Author:** [@ianmac66](https://discuss.elastic.co/u/ianmac66)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 11:22am UTC](https://discuss.elastic.co/t/lostash-command-line-logstash-e-input-stdin-output-stdout-does-nothing/90229 "2017-06-21T11:22:32Z")
Hi - I'm new to Logstash and elasticssearch in general - so hopefully one of you guys can help me. System is a Openstack VM - Centos 7 New installation of Elasticsearch - both ES and Kibana are working fine, i.e can … --- ## [Extract the value from CSV field and add new field](https://discuss.elastic.co/t/extract-the-value-from-csv-field-and-add-new-field/89974)
**Author:** [@mussa572](https://discuss.elastic.co/u/mussa572)\ **Replies:** 6\ **Last updated:** [June 21, 2017, 10:57am UTC](https://discuss.elastic.co/t/extract-the-value-from-csv-field-and-add-new-field/89974 "2017-06-21T10:57:34Z")
Dear Team Wondering if you can help please . We are parsing the CSV using the below code , One of the field (exec) in CSV file can have either one of below values (T= some integer value |D= some integer value |N=s… --- ## [Adding quotes to variables](https://discuss.elastic.co/t/adding-quotes-to-variables/89943)
**Author:** [@orhiee](https://discuss.elastic.co/u/orhiee)\ **Replies:** 4\ **Last updated:** [June 21, 2017, 10:37am UTC](https://discuss.elastic.co/t/adding-quotes-to-variables/89943 "2017-06-21T10:37:45Z")
hello, so i have a elasticsearch filter added to my logstash and I am trying to search all the IP address i get, but i get an ipv6 it messes up, due to the ":" in the address: so my query is: query =\> "type:XYZ A… --- ## ["Couldn't find any filter plugin named 'multiline'. Are you sure this is correct](https://discuss.elastic.co/t/couldnt-find-any-filter-plugin-named-multiline-are-you-sure-this-is-correct/90207)
**Author:** [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 9:39am UTC](https://discuss.elastic.co/t/couldnt-find-any-filter-plugin-named-multiline-are-you-sure-this-is-correct/90207 "2017-06-21T09:39:29Z")
I am migrated to Logstash 5.4 and running the conf with multiline plugin. its not working. filter { if \[message\] =~ "^\<\\?xml .\*" { drop {} } multiline { pattern =\> "^\<\\/response\>" negate =\> true w… --- ## [Logstash-grok filter pattern](https://discuss.elastic.co/t/logstash-grok-filter-pattern/90251)
**Author:** [@blackOcean](https://discuss.elastic.co/u/blackOcean)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 9:27am UTC](https://discuss.elastic.co/t/logstash-grok-filter-pattern/90251 "2017-06-21T09:27:59Z")
this is my sample.log 2017-01-03 05:40:49.693 INFO main --- com.getsentry.raven.DefaultRavenFactory : Using an HTTP connection to Sentry. 2017-01-03 05:40:49.935 INFO background-preinit --- org.hibernate.validator.in… --- ## [Exception in thread "\[main\]\>worker4" java.lang.ClassCastException: org.logstash.bivalues.StringBiValue cannot be cast to java.lang.String: is this a bug in logstash?](https://discuss.elastic.co/t/exception-in-thread-main-worker4-java-lang-classcastexception-org-logstash-bivalues-stringbivalue-cannot-be-cast-to-java-lang-string-is-this-a-bug-in-logstash/88638)
**Author:** [@filebeater](https://discuss.elastic.co/u/filebeater)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 9:18am UTC](https://discuss.elastic.co/t/exception-in-thread-main-worker4-java-lang-classcastexception-org-logstash-bivalues-stringbivalue-cannot-be-cast-to-java-lang-string-is-this-a-bug-in-logstash/88638 "2017-06-21T09:18:19Z")
hello, I consistently got an exception when running logstash. the exception is copied at the end. Any pointer is appreciated! Version: logstash-5.4.1 Operating System: Redhat Enterprise R6.6(Santiago), i started a kafk… --- ## [\_node/stats - duration\_in\_millis vs queue\_push\_duration\_in\_millis](https://discuss.elastic.co/t/-node-stats-duration-in-millis-vs-queue-push-duration-in-millis/90000)
**Author:** [@bhatch](https://discuss.elastic.co/u/bhatch)\ **Replies:** 3\ **Last updated:** [June 21, 2017, 8:35am UTC](https://discuss.elastic.co/t/-node-stats-duration-in-millis-vs-queue-push-duration-in-millis/90000 "2017-06-21T08:35:37Z")
We recently upgraded Logstash to 5.4.1 and have been having some performance issues. In order to try drilling down into what is going on I have been drilling into the node stats API. However not all of the fields are wel… --- ## [Problem parsing large XML with logstash?](https://discuss.elastic.co/t/problem-parsing-large-xml-with-logstash/90122)
**Author:** [@Yaswanth](https://discuss.elastic.co/u/Yaswanth)\ **Replies:** 3\ **Last updated:** [June 21, 2017, 8:30am UTC](https://discuss.elastic.co/t/problem-parsing-large-xml-with-logstash/90122 "2017-06-21T08:30:36Z")
Logstash 2.4.1 I want to parse a XML file : \ \ \ \Empire Burlesque\ \Bob Dylan\ \ **Author:** [@Benjamin\_Peere](https://discuss.elastic.co/u/Benjamin_Peere)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 8:28am UTC](https://discuss.elastic.co/t/good-configuration-for-logstash/90219 "2017-06-21T08:28:57Z") Hello, i am new to use logstash, i've just installed it and configured like in the Getting started video, i have also installed elasticsearch and when i try to make the hello word (like in the video), i run logstash with… --- ## [Logstash plugin unit tests, strip output and input blocks of config](https://discuss.elastic.co/t/logstash-plugin-unit-tests-strip-output-and-input-blocks-of-config/90170)
**Author:** [@Conall\_Prendergast](https://discuss.elastic.co/u/Conall_Prendergast)\ **Replies:** 2\ **Last updated:** [June 21, 2017, 8:28am UTC](https://discuss.elastic.co/t/logstash-plugin-unit-tests-strip-output-and-input-blocks-of-config/90170 "2017-06-21T08:28:31Z")
In my unit tests for my custom logstash plugin, I load in a config file that has input, filter, and output blocks, as normal. Is there an easy way to somehow load only the filter section of this file, perhaps using the… --- ## [CSV and double qoutes](https://discuss.elastic.co/t/csv-and-double-qoutes/90201)
**Author:** [@Mario\_van\_Gemert](https://discuss.elastic.co/u/Mario_van_Gemert)\ **Replies:** 4\ **Last updated:** [June 21, 2017, 7:14am UTC](https://discuss.elastic.co/t/csv-and-double-qoutes/90201 "2017-06-21T07:14:13Z")
I am trying to import a CSV file by using the CSV filter in Logstash. This works fine, but in the description field I sometime have special characters, like " (double quotes), so this gives a parse failure. How can I sol… --- ## [Scroll format in Elasticsearch input plugin](https://discuss.elastic.co/t/scroll-format-in-elasticsearch-input-plugin/90205)
**Author:** [@Marc\_Aragones](https://discuss.elastic.co/u/Marc_Aragones)\ **Replies:** 0\ **Last updated:** [June 21, 2017, 7:10am UTC](https://discuss.elastic.co/t/scroll-format-in-elasticsearch-input-plugin/90205 "2017-06-21T07:10:52Z")
Hi, In the Logstash's Elasticsearch input plugin there is the scroll field with "1m" as a default value. Documentation says: This parameter controls the keepalive time in seconds of the scrolling request and initiates… --- ## [Logstash not parsing logs](https://discuss.elastic.co/t/logstash-not-parsing-logs/90148)
**Author:** [@vgupta0308](https://discuss.elastic.co/u/vgupta0308)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 6:15am UTC](https://discuss.elastic.co/t/logstash-not-parsing-logs/90148 "2017-06-21T06:15:51Z")
Hello, We have created the Logstash config for parsing the bro ids logs which is not working. Below is the output of log file: \[2017-06-20T17:25:16,966\]\[WARN \]\[logstash.outputs.elasticsearch\] Restored connection to … --- ## [Filebeat v/s Logstash file Path](https://discuss.elastic.co/t/filebeat-v-s-logstash-file-path/90149)
**Author:** [@vgupta0308](https://discuss.elastic.co/u/vgupta0308)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 5:37am UTC](https://discuss.elastic.co/t/filebeat-v-s-logstash-file-path/90149 "2017-06-21T05:37:37Z")
Hi, During reading the posts, I have found most of the people are using filebeat even we can input the file path for parsing the logs. Can anyone please explain the difference in between both options? What are the pros… --- ## [Ruby exception occured \`+'](https://discuss.elastic.co/t/ruby-exception-occured/90192)
**Author:** [@Xavier\_Clavien](https://discuss.elastic.co/u/Xavier_Clavien)\ **Replies:** 1\ **Last updated:** [June 21, 2017, 5:33am UTC](https://discuss.elastic.co/t/ruby-exception-occured/90192 "2017-06-21T05:33:37Z")
Hi guys Got strange exception because the result is correct, but then all the code after the one who caused the exception is not working anymore. Here is the code : event.set( '\[ln\_resistance\_B\_inversion\]', ( event.ge… --- ## [Logstash \_grokparsefailure error](https://discuss.elastic.co/t/logstash--grokparsefailure-error/89989)
**Author:** [@sslv](https://discuss.elastic.co/u/sslv)\ **Replies:** 20\ **Last updated:** [June 21, 2017, 5:31am UTC](https://discuss.elastic.co/t/logstash--grokparsefailure-error/89989 "2017-06-21T05:31:48Z")
Hi, I am trying to parse a simple log file to understand how logstash works. This is my log format: 2017-06-14 11:17:48 \[ad8880\] INFO: blah blah blah And I have built the following grok regex using grok Constructor %… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=959) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=961)