# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=771 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 772 --- ## [\[solved\] How to work with sql\_last\_value on table that has multiple rows on same second](https://discuss.elastic.co/t/solved-how-to-work-with-sql-last-value-on-table-that-has-multiple-rows-on-same-second/129736)
**Author:** [@skatemple](https://discuss.elastic.co/u/skatemple)\ **Replies:** 4\ **Last updated:** [April 26, 2018, 11:27pm UTC](https://discuss.elastic.co/t/solved-how-to-work-with-sql-last-value-on-table-that-has-multiple-rows-on-same-second/129736 "2018-04-26T23:27:57Z")
Hello, I'm having troubles with a table, it has not a unique id column, so I have to pull data using a datetime column in my "where" statement. Problem is, that this tables saves the Datetime like this: "2018-04-26 02… --- ## [Import data from redis](https://discuss.elastic.co/t/import-data-from-redis/129489)
**Author:** [@brfreitas](https://discuss.elastic.co/u/brfreitas)\ **Replies:** 1\ **Last updated:** [April 26, 2018, 10:59pm UTC](https://discuss.elastic.co/t/import-data-from-redis/129489 "2018-04-26T22:59:30Z")
Hello folks, I'm new using ELK, I wanna know what is the best approach to import all redis keys and values to elastisearch with logstash. --- ## [Xml processing](https://discuss.elastic.co/t/xml-processing/129533)
**Author:** [@sandeepnarla22322](https://discuss.elastic.co/u/sandeepnarla22322)\ **Replies:** 8\ **Last updated:** [April 26, 2018, 8:00pm UTC](https://discuss.elastic.co/t/xml-processing/129533 "2018-04-26T20:00:34Z")
\ 1794480 20180425195021 SES 10.174.79.158 2018-04-25 07:20:24.741642168 Wed, 25 Apr 2018 14:20:24 GMT 10.174.79.158 Apache-HttpClient/4… --- ## [Import settings for a Input plugin from an external file](https://discuss.elastic.co/t/import-settings-for-a-input-plugin-from-an-external-file/129334)
**Author:** [@ljak](https://discuss.elastic.co/u/ljak)\ **Replies:** 4\ **Last updated:** [April 26, 2018, 7:59pm UTC](https://discuss.elastic.co/t/import-settings-for-a-input-plugin-from-an-external-file/129334 "2018-04-26T19:59:05Z")
Hi everyone, I'm asking if it is possible to specify a(n) (path to an) external file into the input section. The reference to these external file is not be used for defining an input events streams (like all other avail… --- ## [What version of JRuby should plugins be built with?](https://discuss.elastic.co/t/what-version-of-jruby-should-plugins-be-built-with/129731)
**Author:** [@wakinov](https://discuss.elastic.co/u/wakinov)\ **Replies:** 1\ **Last updated:** [April 26, 2018, 6:09pm UTC](https://discuss.elastic.co/t/what-version-of-jruby-should-plugins-be-built-with/129731 "2018-04-26T18:09:53Z")
I see a lot of mentions of "1.9" but that doesn't seem to be a real version. --- ## [JDBC input MS SQL](https://discuss.elastic.co/t/jdbc-input-ms-sql/129719)
**Author:** [@M\_B\_I](https://discuss.elastic.co/u/M_B_I)\ **Replies:** 1\ **Last updated:** [April 26, 2018, 6:05pm UTC](https://discuss.elastic.co/t/jdbc-input-ms-sql/129719 "2018-04-26T18:05:53Z")
Hi, everyone! I try to move my data from MS SQL DB to elastic. I found JDBC plugin and it's cool =) But I need to store in elastic only actual data that exist at this moment in SQL DB. How can I do this correct? I mea… --- ## [Remove duplicates in a logstash array](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705)
**Author:** [@SKing](https://discuss.elastic.co/u/SKing)\ **Replies:** 2\ **Last updated:** [April 26, 2018, 3:32pm UTC](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705 "2018-04-26T15:32:15Z")
I have a json String with name / value pairs { "A1":"123", "A2":"225", "A3":"668", "A4":"225" } Using logstash I can mutate the string and end up with an array by using mutate { add\_field =\> {A =\> "%{A1}") add\_… --- ## [Logstash date pattern creation](https://discuss.elastic.co/t/logstash-date-pattern-creation/129693)
**Author:** [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\ **Replies:** 1\ **Last updated:** [April 26, 2018, 2:08pm UTC](https://discuss.elastic.co/t/logstash-date-pattern-creation/129693 "2018-04-26T14:08:31Z")
How can we write a pattern for this date fromat "11 Apr 2018 10:40:11,301 " I am Facing difficulty to fetch the details. --- ## [\[Simple\] Logstash config file](https://discuss.elastic.co/t/simple-logstash-config-file/129613)
**Author:** [@Kevin\_Csuka](https://discuss.elastic.co/u/Kevin_Csuka)\ **Replies:** 11\ **Last updated:** [April 26, 2018, 1:51pm UTC](https://discuss.elastic.co/t/simple-logstash-config-file/129613 "2018-04-26T13:51:23Z")
I've got these lines in my log: 1524708231.794259376 8 c0:4a:00:40:e6:0e c0:4a:00:40:AA:AA -45 1524708231.829154447 6 3c:08:f6:e6:AA:AA 3c:08:f6:e6:AA:AA -71 1st item is Unix timestamp 2nd item is channel … --- ## [Getting Connection Error with Twitter Input in Logstash](https://discuss.elastic.co/t/getting-connection-error-with-twitter-input-in-logstash/129588)
**Author:** [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\ **Replies:** 3\ **Last updated:** [April 26, 2018, 10:04am UTC](https://discuss.elastic.co/t/getting-connection-error-with-twitter-input-in-logstash/129588 "2018-04-26T10:04:20Z")
Hi All, I am in China where Twitter is Blocked. I created my dev app in https://apps.twitter.com and collected all essential Key and tokens. My Conf file is as below :- input { twitter { consumer\_key =\> 'XXXXX' c… --- ## [Error geoip](https://discuss.elastic.co/t/error-geoip/129474)
**Author:** [@Charaf\_Ahmed](https://discuss.elastic.co/u/Charaf_Ahmed)\ **Replies:** 3\ **Last updated:** [April 26, 2018, 9:58am UTC](https://discuss.elastic.co/t/error-geoip/129474 "2018-04-26T09:58:54Z")
No Compatible Fields: The "apache2-\*" index pattern does not contain any of the following field types: geo\_point --- ## [If field contains question mark](https://discuss.elastic.co/t/if-field-contains-question-mark/129608)
**Author:** [@KevSex](https://discuss.elastic.co/u/KevSex)\ **Replies:** 3\ **Last updated:** [April 26, 2018, 9:00am UTC](https://discuss.elastic.co/t/if-field-contains-question-mark/129608 "2018-04-26T09:00:16Z")
Hi, I'm trying to filter out some URLs to split up the uri stem and query. I've got the following pattern match which works when there is a questionmark to separate them both. filter { grok { patterns\_dir =\> "/e… --- ## [Log stash Installation](https://discuss.elastic.co/t/log-stash-installation/128808)
**Author:** [@manojbhupatiraju](https://discuss.elastic.co/u/manojbhupatiraju)\ **Replies:** 13\ **Last updated:** [April 26, 2018, 8:56am UTC](https://discuss.elastic.co/t/log-stash-installation/128808 "2018-04-26T08:56:46Z")
Hi, I have installed logstash in my machine and I am getting below mentioned error while installing.Kindly help me. \[2018-04-19T23:49:11,825\]\[INFO \]\[logstash.config.source.local.configpathloader\] No config files found … --- ## [Parse error on Tshark generated JSON](https://discuss.elastic.co/t/parse-error-on-tshark-generated-json/129587)
**Author:** [@pohsun.teh](https://discuss.elastic.co/u/pohsun.teh)\ **Replies:** 4\ **Last updated:** [April 26, 2018, 8:36am UTC](https://discuss.elastic.co/t/parse-error-on-tshark-generated-json/129587 "2018-04-26T08:36:30Z")
Greetings. I have a JSON generated using Tshark -T ek command converting Wireshark PCAP file to JSON to be inserted to Elasticsearch. The thing is I am having parse error exception on production side. :exception=\>#\ **Author:** [@Steven\_Lin](https://discuss.elastic.co/u/Steven_Lin)\ **Replies:** 3\ **Last updated:** [April 26, 2018, 8:29am UTC](https://discuss.elastic.co/t/parse-cvs-file-with-a-key-and-multiple-values/129578 "2018-04-26T08:29:04Z") Hi! i have a cvs file with a key and multiple value by date to show the devices status like this: deviceid, 2018-04-01, 2018-04-02, 2018-04-03, ... d001, 0, 0, … --- ## [Error while consuming from topic](https://discuss.elastic.co/t/error-while-consuming-from-topic/129610)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [April 26, 2018, 8:26am UTC](https://discuss.elastic.co/t/error-while-consuming-from-topic/129610 "2018-04-26T08:26:55Z")
Hi, I am using ELK GA 5.0.0. I am using Logstash to consume from Kafka topic and my configuration is like below; input { kafka{ group\_id =\> "logstash-ga5-group" topics =\> \["testtopic"\] bootstrap\_servers =\> "192.… --- ## [A single event break into array based on patterns](https://discuss.elastic.co/t/a-single-event-break-into-array-based-on-patterns/129606)
**Author:** [@wolecharles\_job](https://discuss.elastic.co/u/wolecharles_job)\ **Replies:** 0\ **Last updated:** [April 26, 2018, 7:56am UTC](https://discuss.elastic.co/t/a-single-event-break-into-array-based-on-patterns/129606 "2018-04-26T07:56:05Z")
A single events is broken to an array based on the pattern matched..Something like this 4/26/2018 12:00:19 3467272 done $30 ;4/20/2018 12:03:19 3467273 done $40 ;4/27/2018 12:03:19 3467274 done $40 ; { \[ { date : ""… --- ## [Ruby filter: Ruby exception occurred:undefined local variable or method \`map'](https://discuss.elastic.co/t/ruby-filter-ruby-exception-occurred-undefined-local-variable-or-method-map/129596)
**Author:** [@vinayakfutak](https://discuss.elastic.co/u/vinayakfutak)\ **Replies:** 4\ **Last updated:** [April 26, 2018, 7:19am UTC](https://discuss.elastic.co/t/ruby-filter-ruby-exception-occurred-undefined-local-variable-or-method-map/129596 "2018-04-26T07:19:34Z")
I wanna run ruby script for calculating the diference between documents.here is my script input { elasticsearch { hosts =\> "http://192.168.55.213:9200" index =\> "test\_index" query =\> '{"sort": \[{"logTimestamp": {"or… --- ## [\[Solved\] Logstash sends logs to the same elasticsearch index even after I configured not to do so](https://discuss.elastic.co/t/solved-logstash-sends-logs-to-the-same-elasticsearch-index-even-after-i-configured-not-to-do-so/129574)
**Author:** [@JohnHuElastic](https://discuss.elastic.co/u/JohnHuElastic)\ **Replies:** 2\ **Last updated:** [April 26, 2018, 6:17am UTC](https://discuss.elastic.co/t/solved-logstash-sends-logs-to-the-same-elasticsearch-index-even-after-i-configured-not-to-do-so/129574 "2018-04-26T06:17:49Z")
I got two logs, one is nginx access log, another is gunicorn access log. The two logs has similar contents, but I want them to be two different elasticsearch indices. Here are my conf files: # /etc/logstash/conf.d/ngi… --- ## [How to parse json in grok](https://discuss.elastic.co/t/how-to-parse-json-in-grok/129465)
**Author:** [@jimmy.zhao](https://discuss.elastic.co/u/jimmy.zhao)\ **Replies:** 2\ **Last updated:** [April 26, 2018, 1:54am UTC](https://discuss.elastic.co/t/how-to-parse-json-in-grok/129465 "2018-04-26T01:54:50Z")
I was trying to parse my Neutron log through grok pattern, the last two fileds are two json string, here is the line. 2018-04-19T05:08:42.833+02:00 localhost neutron-api INFO \[type:operate\] \[pid:2157\] \[MainThread\] \[acce… --- ## [Why doesn't add\_tag =\> \["${HOSTNAME}"\] work?](https://discuss.elastic.co/t/why-doesnt-add-tag-hostname-work/129366)
**Author:** [@JimP](https://discuss.elastic.co/u/JimP)\ **Replies:** 18\ **Last updated:** [April 26, 2018, 12:44am UTC](https://discuss.elastic.co/t/why-doesnt-add-tag-hostname-work/129366 "2018-04-26T00:44:09Z")
This is an environment variable, but I can't get it to work? The tag save is, ${HOSTNAME}, not the environment variable value. https://www.elastic.co/guide/en/logstash/current/environment-variables.html#\_setting\_the\_va… --- ## [Netflow index issues](https://discuss.elastic.co/t/netflow-index-issues/129522)
**Author:** [@Andreas\_Kaiser](https://discuss.elastic.co/u/Andreas_Kaiser)\ **Replies:** 1\ **Last updated:** [April 25, 2018, 9:13pm UTC](https://discuss.elastic.co/t/netflow-index-issues/129522 "2018-04-25T21:13:46Z")
Hello everybody, i tried to setup netflow logstash based on this tutorial : https://www.elastic.co/guide/en/logstash/current/netflow-module.html Everything works fine except the kibana dashboards I get a lot of error… --- ## [Loading many (big) json files into elasticsearch](https://discuss.elastic.co/t/loading-many-big-json-files-into-elasticsearch/128078)
**Author:** [@nkaenzig](https://discuss.elastic.co/u/nkaenzig)\ **Replies:** 13\ **Last updated:** [April 25, 2018, 8:20pm UTC](https://discuss.elastic.co/t/loading-many-big-json-files-into-elasticsearch/128078 "2018-04-25T20:20:00Z")
Dear community, I have about 1TB of data splitted into many smaller .json files in newline delimited JSON (NDJSON) format. The sizes of the .json files vary between 500MB and 20GB. The files are too big to load using t… --- ## [Adding 1 day to the date](https://discuss.elastic.co/t/adding-1-day-to-the-date/129168)
**Author:** [@HRG](https://discuss.elastic.co/u/HRG)\ **Replies:** 42\ **Last updated:** [April 25, 2018, 7:06pm UTC](https://discuss.elastic.co/t/adding-1-day-to-the-date/129168 "2018-04-25T19:06:52Z")
I am new to Logstash and have a requirement to add 1 day to the date(Businessdate) and compare log event date with Businessdate. Can you please suggest how to achieve this. Ex: Parsed string to extract YEAR, MONTH AND … --- ## [3 Node Setup](https://discuss.elastic.co/t/3-node-setup/129480)
**Author:** [@homood](https://discuss.elastic.co/u/homood)\ **Replies:** 1\ **Last updated:** [April 25, 2018, 6:24pm UTC](https://discuss.elastic.co/t/3-node-setup/129480 "2018-04-25T18:24:28Z")
Hello! so I'm trying to do the following: Retrieve Linux servers logs and filter them for specific events and then pass them to a SIEM solution. Can I achieve that on the Log Stash level? And do I need all three layer… --- ## [Netflow.conf.erb need help updating](https://discuss.elastic.co/t/netflow-conf-erb-need-help-updating/129542)
**Author:** [@dalamar666](https://discuss.elastic.co/u/dalamar666)\ **Replies:** 0\ **Last updated:** [April 25, 2018, 5:21pm UTC](https://discuss.elastic.co/t/netflow-conf-erb-need-help-updating/129542 "2018-04-25T17:21:12Z")
Hi, There is some code I need help updating with this file. According to a log from cisco they have changed the format they are sending the bytes in. instead of in\_permanent like in the file, they are using another fo… --- ## [Grok pattern file does not exist](https://discuss.elastic.co/t/grok-pattern-file-does-not-exist/129442)
**Author:** [@goofansu](https://discuss.elastic.co/u/goofansu)\ **Replies:** 6\ **Last updated:** [April 25, 2018, 4:24pm UTC](https://discuss.elastic.co/t/grok-pattern-file-does-not-exist/129442 "2018-04-25T16:24:42Z")
Hi, I have configured a custom pattern and using patterns\_dir to locate it. But it reports "Grok pattern file does not exist". root@openapply-prod-app:/etc/logstash/conf.d# ls -la total 36 drwxrwxr-x 3 root root 4096 … --- ## [Mutate function not working (Logstash 6.2.4)](https://discuss.elastic.co/t/mutate-function-not-working-logstash-6-2-4/129067)
**Author:** [@alicia1](https://discuss.elastic.co/u/alicia1)\ **Replies:** 22\ **Last updated:** [April 25, 2018, 4:06pm UTC](https://discuss.elastic.co/t/mutate-function-not-working-logstash-6-2-4/129067 "2018-04-25T16:06:36Z")
Hi, Any mutate function is working on my PC. In fact, i want use mutate lowercase function in my logstash config, but the function is not working, any data was sent to ES. I tried to test other mutate function like ad… --- ## [Agregation in logstash](https://discuss.elastic.co/t/agregation-in-logstash/129518)
**Author:** [@ca082dcaae565fd0dcae](https://discuss.elastic.co/u/ca082dcaae565fd0dcae)\ **Replies:** 0\ **Last updated:** [April 25, 2018, 3:10pm UTC](https://discuss.elastic.co/t/agregation-in-logstash/129518 "2018-04-25T15:10:02Z")
Hi. I use agregation in logstash to concatenate string logs into one. and my agregation field looks in kibana like: { "message": "2018-04-23 10:43:14.439 \[I/O 23\] http--30150 \>\> "Accept: application/json\[\\r\]\[\\n\]"" }, … --- ## [Getting Problem with Logstash to Index Mysql Data](https://discuss.elastic.co/t/getting-problem-with-logstash-to-index-mysql-data/129409)
**Author:** [@nandan\_bigdata](https://discuss.elastic.co/u/nandan_bigdata)\ **Replies:** 5\ **Last updated:** [April 25, 2018, 3:33pm UTC](https://discuss.elastic.co/t/getting-problem-with-logstash-to-index-mysql-data/129409 "2018-04-25T15:33:49Z")
Hi All, Details are as below. I am using Ubuntu 17.10 and installed Elastic Seach , Kibana,Logstash \[Latest version 6+\]. All 3 services are running perfectly. In my project , For Testing I am trying to Input Mysql tabl… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=770) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=772)