# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=982 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 983 --- ## [Logstash needs to be config'd to append only in elastic](https://discuss.elastic.co/t/logstash-needs-to-be-configd-to-append-only-in-elastic/86104)
**Author:** [@lgee](https://discuss.elastic.co/u/lgee)\ **Replies:** 1\ **Last updated:** [May 17, 2017, 12:56pm UTC](https://discuss.elastic.co/t/logstash-needs-to-be-configd-to-append-only-in-elastic/86104 "2017-05-17T12:56:17Z")
Is it possible to configure logstash so that it can only append to an index in elastic, but without being able to create indexes, and delete or overwrite data? We need this so that the stack can be accredited for use i… --- ## [Logstash conf file not working](https://discuss.elastic.co/t/logstash-conf-file-not-working/86069)
**Author:** [@RustuGurkan](https://discuss.elastic.co/u/RustuGurkan)\ **Replies:** 16\ **Last updated:** [May 17, 2017, 12:51pm UTC](https://discuss.elastic.co/t/logstash-conf-file-not-working/86069 "2017-05-17T12:51:41Z")
Hi, I have a problem with the conf file of logstash it doesn't use the grok patterns I've passed. I'm trying to get the IP address so geoip can show me where it came from. It's working with apache but it doesn't with the… --- ## [Building logstash from source fails as the http\_proxy env setting is not getting passed in](https://discuss.elastic.co/t/building-logstash-from-source-fails-as-the-http-proxy-env-setting-is-not-getting-passed-in/85920)
**Author:** [@pkempia](https://discuss.elastic.co/u/pkempia)\ **Replies:** 10\ **Last updated:** [May 17, 2017, 12:19pm UTC](https://discuss.elastic.co/t/building-logstash-from-source-fails-as-the-http-proxy-env-setting-is-not-getting-passed-in/85920 "2017-05-17T12:19:26Z")
I am trying to build logstash 5.3.2 from source. Below are the steps I followed. But the "rake bootstrap" command fails with the "Errno::EHOSTUNREACH: No route to host - connect(2)" This is been done on a empty docker … --- ## [How to parse docker logs (nested serialized JSON)](https://discuss.elastic.co/t/how-to-parse-docker-logs-nested-serialized-json/85940)
**Author:** [@hunterchris](https://discuss.elastic.co/u/hunterchris)\ **Replies:** 1\ **Last updated:** [May 17, 2017, 12:14pm UTC](https://discuss.elastic.co/t/how-to-parse-docker-logs-nested-serialized-json/85940 "2017-05-17T12:14:34Z")
Hello, I have an AWS Beanstalk environment with docker and I'm sending different logs to logstash, however I'm finding problems parsing the /var/lib/docker/containers//-json.log which has the following format: {"log":"… --- ## [Conditional IF in logstash's filter](https://discuss.elastic.co/t/conditional-if-in-logstashs-filter/85779)
**Author:** [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\ **Replies:** 2\ **Last updated:** [May 17, 2017, 12:07pm UTC](https://discuss.elastic.co/t/conditional-if-in-logstashs-filter/85779 "2017-05-17T12:07:57Z")
Hello, i work on conditional filter in logstash. Currently it's not working :frowning: Here a few example of line in my file : 20170415020000;Nb\_RQ\_live-C-FWK-WRK-DM-2;server5;2 20170415020000;Nb\_RQ\_live-C-FWK-WR… --- ## [Remove unnecessary documents Logstash](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768)
**Author:** [@Sam67000](https://discuss.elastic.co/u/Sam67000)\ **Replies:** 9\ **Last updated:** [May 17, 2017, 11:54am UTC](https://discuss.elastic.co/t/remove-unnecessary-documents-logstash/85768 "2017-05-17T11:54:09Z")
Hello everyOne, I work on several log files that I process with logstash. I divide them into several documents (multiline) and then I extract the information I want. The problem is that I find myself in the end with… --- ## [Message field failed to store more than 500 lines](https://discuss.elastic.co/t/message-field-failed-to-store-more-than-500-lines/84660)
**Author:** [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\ **Replies:** 2\ **Last updated:** [May 17, 2017, 11:51am UTC](https://discuss.elastic.co/t/message-field-failed-to-store-more-than-500-lines/84660 "2017-05-17T11:51:44Z")
HI, In my log one event contain more than 2000 line(XML) when i process the log, i could see first 500 line of the event in kibana. I am using file input plugin and wanted to know if Logstash has a size limit for each… --- ## [Logstash GROK parse for syslog](https://discuss.elastic.co/t/logstash-grok-parse-for-syslog/86077)
**Author:** [@engineering87](https://discuss.elastic.co/u/engineering87)\ **Replies:** 3\ **Last updated:** [May 17, 2017, 11:33am UTC](https://discuss.elastic.co/t/logstash-grok-parse-for-syslog/86077 "2017-05-17T11:33:04Z")
Hi all, I have a problem with the logstash grok configuration. This is my current logstash configuration (logstash is running on a windows machine): input { tcp { port =\> 5000 type =\> syslog } udp { … --- ## [Logstash and dynamic fields in logs](https://discuss.elastic.co/t/logstash-and-dynamic-fields-in-logs/86091)
**Author:** [@jmccormack](https://discuss.elastic.co/u/jmccormack)\ **Replies:** 0\ **Last updated:** [May 17, 2017, 11:27am UTC](https://discuss.elastic.co/t/logstash-and-dynamic-fields-in-logs/86091 "2017-05-17T11:27:02Z")
Hey All So I have a log file that has dynamic content depending on the specific machine it is running on. For example: 0170517 00:07:20 name 00 2 3 6 0 5 name1 00 0 2 3 6 0 5 name2 00 0 2 3 6 0 5 0170517 00:07:30 n… --- ## [How to write filter part in my conf file for json file as input](https://discuss.elastic.co/t/how-to-write-filter-part-in-my-conf-file-for-json-file-as-input/86075)
**Author:** [@Gopi\_K1](https://discuss.elastic.co/u/Gopi_K1)\ **Replies:** 1\ **Last updated:** [May 17, 2017, 11:24am UTC](https://discuss.elastic.co/t/how-to-write-filter-part-in-my-conf-file-for-json-file-as-input/86075 "2017-05-17T11:24:52Z")
Here is my conf file look like: input { file { codec =\> "json" path =\> "/ci/data/cirrus/report/\*/status.json" start\_position =\> "beginning" sincedb\_path =\> "… --- ## [Cluster ID "5d1f1d" - Logstash startup is super slow](https://discuss.elastic.co/t/cluster-id-5d1f1d-logstash-startup-is-super-slow/85963)
**Author:** [@vidagh](https://discuss.elastic.co/u/vidagh)\ **Replies:** 5\ **Last updated:** [May 17, 2017, 9:38am UTC](https://discuss.elastic.co/t/cluster-id-5d1f1d-logstash-startup-is-super-slow/85963 "2017-05-17T09:38:19Z")
Hi, Logstash is taking hours to execute as the filter size is growing, and CPU usage goes to max. It works fine for a parser of around 2000 lines, but then it would slow down, almost stops, as it grows. CentOS ma… --- ## [Elasticsearch-output-statsd plugin](https://discuss.elastic.co/t/elasticsearch-output-statsd-plugin/86047)
**Author:** [@Ninny\_Harsha](https://discuss.elastic.co/u/Ninny_Harsha)\ **Replies:** 1\ **Last updated:** [May 17, 2017, 9:13am UTC](https://discuss.elastic.co/t/elasticsearch-output-statsd-plugin/86047 "2017-05-17T09:13:14Z")
statsd { host =\> "192.168.2.20" port =\> 8125 count =\> \[ "stats.gauges.success" , "%{REQUEST}" \] } i am using above plugin in output{ ... }. when i send my "LOG" to parser… --- ## [Timestamp =\> @timestamp](https://discuss.elastic.co/t/timestamp-timestamp/85763)
**Author:** [@Saplog](https://discuss.elastic.co/u/Saplog)\ **Replies:** 16\ **Last updated:** [May 17, 2017, 7:45am UTC](https://discuss.elastic.co/t/timestamp-timestamp/85763 "2017-05-17T07:45:58Z")
Hi , I have specific log in a file log: Tue May 2 03:54:22 2017 I use a grok filter to extract the timestamp into a field of its own called "datestamp" "datestamp" =\> "Tue May 2 03:57:00 2017 ", "@timestamp" =\> … --- ## [How to collect log and gathering log with elastic & kibana from WEBSITE](https://discuss.elastic.co/t/how-to-collect-log-and-gathering-log-with-elastic-kibana-from-website/86052)
**Author:** [@omid](https://discuss.elastic.co/u/omid)\ **Replies:** 1\ **Last updated:** [May 17, 2017, 7:40am UTC](https://discuss.elastic.co/t/how-to-collect-log-and-gathering-log-with-elastic-kibana-from-website/86052 "2017-05-17T07:40:07Z")
Hello I am using the Elasticsearch, Logstash, and Kibana collect log a website Elasticsearch, Logstash, and Kibana How does this ? The API uses of the website? Javascript code is? Or the other way? --- ## [Logstah optimization and data merge](https://discuss.elastic.co/t/logstah-optimization-and-data-merge/86055)
**Author:** [@JIELOGAN](https://discuss.elastic.co/u/JIELOGAN)\ **Replies:** 0\ **Last updated:** [May 17, 2017, 7:15am UTC](https://discuss.elastic.co/t/logstah-optimization-and-data-merge/86055 "2017-05-17T07:15:59Z")
hello everyone，i need help！please ！ problem1：how to optimize my code？ problem2：how to delete the logdata i don‘t want and merge my data，cause my data has lots of blank and duplicates. i want delete the invalid data，an… --- ## [Beginner question grok + pattern matching + different line format](https://discuss.elastic.co/t/beginner-question-grok-pattern-matching-different-line-format/85974)
**Author:** [@Goudal](https://discuss.elastic.co/u/Goudal)\ **Replies:** 1\ **Last updated:** [May 17, 2017, 7:08am UTC](https://discuss.elastic.co/t/beginner-question-grok-pattern-matching-different-line-format/85974 "2017-05-17T07:08:55Z")
Hello, I'm new to elk and I'm trying to figure out how the grok match keyword works. What I wonder is if I want to parse different line format, do I have to put multiple grok filters one after the other one and the firs… --- ## [Logstash keeps trying configuration that's already deleted](https://discuss.elastic.co/t/logstash-keeps-trying-configuration-thats-already-deleted/84470)
**Author:** [@ronakg](https://discuss.elastic.co/u/ronakg)\ **Replies:** 5\ **Last updated:** [May 17, 2017, 5:16am UTC](https://discuss.elastic.co/t/logstash-keeps-trying-configuration-thats-already-deleted/84470 "2017-05-17T05:16:28Z")
Hi, I'm getting into this issue where logstash keeps trying to send events to a syslog server even after I've deleted the output plugin for syslog server. I've set logstash to look for new config every 10 seconds. Bu… --- ## [Log file decode error](https://discuss.elastic.co/t/log-file-decode-error/85707)
**Author:** [@LmYjQ](https://discuss.elastic.co/u/LmYjQ)\ **Replies:** 2\ **Last updated:** [May 17, 2017, 1:17am UTC](https://discuss.elastic.co/t/log-file-decode-error/85707 "2017-05-17T01:17:40Z")
I'm using gz\_lines codec in logstash referring this post Read a gzip file with gzip\_lines codec - #12 by TC\_Huang But end up with 13:26:37.747 \[\[main\]\"/data1/l… --- ## [Logstash TCP output blocks entire pipeline](https://discuss.elastic.co/t/logstash-tcp-output-blocks-entire-pipeline/85818)
**Author:** [@danwhitacre](https://discuss.elastic.co/u/danwhitacre)\ **Replies:** 4\ **Last updated:** [May 16, 2017, 10:09pm UTC](https://discuss.elastic.co/t/logstash-tcp-output-blocks-entire-pipeline/85818 "2017-05-16T22:09:16Z")
So I am trying to use the Logstash-tcp-output plugin to send data from one logstash service to another using the logstash-tcp-input plugin. Everything works just fine when the receiving logstash tcp input is up and runni… --- ## [Tags vs Fields for conditional search](https://discuss.elastic.co/t/tags-vs-fields-for-conditional-search/85912)
**Author:** [@sahar\_q](https://discuss.elastic.co/u/sahar_q)\ **Replies:** 5\ **Last updated:** [May 16, 2017, 9:05pm UTC](https://discuss.elastic.co/t/tags-vs-fields-for-conditional-search/85912 "2017-05-16T21:05:49Z")
Which way is more efficient for conditional searching? i know little of the implementation but according to my way of thinking Tags should be better is that correct? If so i'd like to know the implementation differen… --- ## [Exporting aggregated data out of ES](https://discuss.elastic.co/t/exporting-aggregated-data-out-of-es/85856)
**Author:** [@pavant](https://discuss.elastic.co/u/pavant)\ **Replies:** 4\ **Last updated:** [May 16, 2017, 7:24pm UTC](https://discuss.elastic.co/t/exporting-aggregated-data-out-of-es/85856 "2017-05-16T19:24:43Z")
I am trying to export aggregared data out of Elasticsearch using log stash input. Below is my code. But I am getting a error. Can anyone help me? My code in Logstash.conf input { elasticsearch { hosts =\> "… --- ## [One log type doesn't parse Ver.2](https://discuss.elastic.co/t/one-log-type-doesnt-parse-ver-2/85959)
**Author:** [@Darya\_Semenova](https://discuss.elastic.co/u/Darya_Semenova)\ **Replies:** 1\ **Last updated:** [May 16, 2017, 2:55pm UTC](https://discuss.elastic.co/t/one-log-type-doesnt-parse-ver-2/85959 "2017-05-16T14:55:17Z")
Hello, Got new problem. My custom patterns are: SMDPROG (systemd|systemd\\-logint|Systemd\\-logint) SMDSESSIONNUM \\b(?:\[1-9\]\[0-9\]\*)\\b SMDUSER \[A-Za-z\]{1,20}(\[.\]\[A-Za-z\]{0,20})? SMDACTION1 (\[A-Za-z\]{1,9}) SMDACTION2 (… --- ## [One log type doesn't parse](https://discuss.elastic.co/t/one-log-type-doesnt-parse/85947)
**Author:** [@Darya\_Semenova](https://discuss.elastic.co/u/Darya_Semenova)\ **Replies:** 1\ **Last updated:** [May 16, 2017, 2:12pm UTC](https://discuss.elastic.co/t/one-log-type-doesnt-parse/85947 "2017-05-16T14:12:31Z")
Hello, I made custom patterns like that: SMDPROG (systemd|systemd\\-logint|Systemd\\-logint) SMDSESSIONNUM \\b(?:\[1-9\]\[0-9\]\*)\\b SMDUSER \[A-Za-z\]{1,20}(\[.\]\[A-Za-z\]{0,20})? SMDACTION1 (\[A-Za-z\]{1,9}(\[.\]?)) SMDACTION2 (\[… --- ## [Error parsing json using jdbc input](https://discuss.elastic.co/t/error-parsing-json-using-jdbc-input/48954)
**Author:** [@JvonRudno](https://discuss.elastic.co/u/JvonRudno)\ **Replies:** 1\ **Last updated:** [May 16, 2017, 1:29pm UTC](https://discuss.elastic.co/t/error-parsing-json-using-jdbc-input/48954 "2017-05-16T13:29:25Z")
Hi everybody, I am trying to parse a json in order to have a nested field in my docu but I get the following message: Error parsing json {:source=\>"catejories\_json", :raw=\>#, :exception=\>java.lang.ClassCastException: o… --- ## [Elasticsearch Logstash Error](https://discuss.elastic.co/t/elasticsearch-logstash-error/85795)
**Author:** [@sudhir\_mca](https://discuss.elastic.co/u/sudhir_mca)\ **Replies:** 1\ **Last updated:** [May 16, 2017, 12:50pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-error/85795 "2017-05-16T12:50:28Z")
Hi All, I am using the following logstash config file. We are seeing below provided special character in logstash log. then around 5-6 json messages are not indexed in elasticsearch. Any input will be greatly appreci… --- ## [MySQL \> JDBC \> Logstash \> Elastic](https://discuss.elastic.co/t/mysql-jdbc-logstash-elastic/85748)
**Author:** [@open7](https://discuss.elastic.co/u/open7)\ **Replies:** 2\ **Last updated:** [May 16, 2017, 12:47pm UTC](https://discuss.elastic.co/t/mysql-jdbc-logstash-elastic/85748 "2017-05-16T12:47:44Z")
I got some Fields in MySQL that holds JSON Encoded Strings. I want them in Elastic also to get stored as Object, but they are Stored as Stings. When i post them normally to the API everything looks good. But i have t… --- ## [Pattern does not contain any of the following field types: geo\_point](https://discuss.elastic.co/t/pattern-does-not-contain-any-of-the-following-field-types-geo-point/85834)
**Author:** [@Rahul\_Lao](https://discuss.elastic.co/u/Rahul_Lao)\ **Replies:** 5\ **Last updated:** [May 16, 2017, 11:31am UTC](https://discuss.elastic.co/t/pattern-does-not-contain-any-of-the-following-field-types-geo-point/85834 "2017-05-16T11:31:39Z")
Hi, I am trying to fetch data from a csv file and store it in elastic search. Finally i want to visualize it on Tile Map. logstash conf file input { file { path =\> "/airports\_test.csv" codec =\> "plain" start\_po… --- ## [Error while setting up connection for rabbitmq input](https://discuss.elastic.co/t/error-while-setting-up-connection-for-rabbitmq-input/85890)
**Author:** [@mathias](https://discuss.elastic.co/u/mathias)\ **Replies:** 0\ **Last updated:** [May 16, 2017, 9:25am UTC](https://discuss.elastic.co/t/error-while-setting-up-connection-for-rabbitmq-input/85890 "2017-05-16T09:25:10Z")
Hi, When I send many events to logstash I get heaps of warnings in logstash log: All the exchanges defined in logstash config exists in rabbitmq. It seems like a new channel is opened and closed for every event. Is th… --- ## [Parsing csv file with dynamic headers/columns](https://discuss.elastic.co/t/parsing-csv-file-with-dynamic-headers-columns/85354)
**Author:** [@sirsyedian](https://discuss.elastic.co/u/sirsyedian)\ **Replies:** 6\ **Last updated:** [May 16, 2017, 9:23am UTC](https://discuss.elastic.co/t/parsing-csv-file-with-dynamic-headers-columns/85354 "2017-05-16T09:23:09Z")
Hi, I need to parse a csv file which contains some dynamic headers/Columns. (lets say) Header format-1: Column-1, Column-2, Collumn-3, X Header format-2: Column-1, Column-2, Collumn-3, Y These files are generated eve… --- ## [Logstash service fails - OpenSSL X509 StoreError](https://discuss.elastic.co/t/logstash-service-fails-openssl-x509-storeerror/60282)
**Author:** [@Benom](https://discuss.elastic.co/u/Benom)\ **Replies:** 2\ **Last updated:** [May 16, 2017, 7:34am UTC](https://discuss.elastic.co/t/logstash-service-fails-openssl-x509-storeerror/60282 "2017-05-16T07:34:06Z")
Hello, I've noticed the logstash service fail on each start. logstash.err shows: \> OpenSSL::X509::StoreError: setting default path failed: IOError \> set\_default\_paths at org/jruby/ext/openssl/X509Store.java:185 \> … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=981) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=983)