# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1145 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1146 --- ## [General Question regarding specific use-case](https://discuss.elastic.co/t/general-question-regarding-specific-use-case/52038)
**Author:** [@ckayay](https://discuss.elastic.co/u/ckayay)\ **Replies:** 3\ **Last updated:** [June 7, 2016, 8:55pm UTC](https://discuss.elastic.co/t/general-question-regarding-specific-use-case/52038 "2016-06-07T20:55:27Z")
Hi All, I am newbie for ES and Shield. I am searching the possibility of doing the following use-case: - I have XML logs that will be ingested by Logstash - Logstash will store the data in ES. - I need to secure specif… --- ## [Logstash-input-journald utf8 issues](https://discuss.elastic.co/t/logstash-input-journald-utf8-issues/52104)
**Author:** [@Peter\_Lang](https://discuss.elastic.co/u/Peter_Lang)\ **Replies:** 0\ **Last updated:** [June 7, 2016, 3:50pm UTC](https://discuss.elastic.co/t/logstash-input-journald-utf8-issues/52104 "2016-06-07T15:50:57Z")
Hi there! The logstash-input-journald plugin does not seem to support "special" characters (like 'ö', 'ő', 'ű' etc.). If I log a string containing such characters, they will end up messed up in the output (ES or file -… --- ## [How to replace a String with another string in a field while Re-indexing an old Index](https://discuss.elastic.co/t/how-to-replace-a-string-with-another-string-in-a-field-while-re-indexing-an-old-index/49905)
**Author:** [@sanju1323](https://discuss.elastic.co/u/sanju1323)\ **Replies:** 16\ **Last updated:** [June 7, 2016, 10:39am UTC](https://discuss.elastic.co/t/how-to-replace-a-string-with-another-string-in-a-field-while-re-indexing-an-old-index/49905 "2016-06-07T10:39:52Z")
Hi, I'm trying to Re-Index an old index in which I have to replace a String with another in a field. Here is the old index data. From this I want to replace "Mr.Rakesh Kumar" with "RK" in the new Index. I have tried … --- ## [Private ip to gioip points](https://discuss.elastic.co/t/private-ip-to-gioip-points/52019)
**Author:** [@venkat-accel](https://discuss.elastic.co/u/venkat-accel)\ **Replies:** 2\ **Last updated:** [June 7, 2016, 8:48am UTC](https://discuss.elastic.co/t/private-ip-to-gioip-points/52019 "2016-06-07T08:48:36Z")
how to get geo-ip from private IP address in ElasticSearch. --- ## [Parsing json with delimiters using logstash](https://discuss.elastic.co/t/parsing-json-with-delimiters-using-logstash/49281)
**Author:** [@LoriG](https://discuss.elastic.co/u/LoriG)\ **Replies:** 23\ **Last updated:** [June 6, 2016, 9:37pm UTC](https://discuss.elastic.co/t/parsing-json-with-delimiters-using-logstash/49281 "2016-06-06T21:37:57Z")
I am trying to parse json which is divided by delimiter($). Does someone have idea how I can configure my logstash? Example: My log includes: ReportBody:{"prop1": val1, "prop2":val2}${"prop1": val3, "prop2":val4}${"prop1… --- ## [Grok help needed](https://discuss.elastic.co/t/grok-help-needed/51838)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 1\ **Last updated:** [June 6, 2016, 9:12pm UTC](https://discuss.elastic.co/t/grok-help-needed/51838 "2016-06-06T21:12:48Z")
Hello Everyone, I have the following record sets int the following format Bytes pps bps flow 45678 12 1.2 M 1 2.2 M 12 231 1 I am try… --- ## [Logstash sync from Oracle to MongoDB and Elasticsearch insert duplicate records when doing partial update](https://discuss.elastic.co/t/logstash-sync-from-oracle-to-mongodb-and-elasticsearch-insert-duplicate-records-when-doing-partial-update/31374)
**Author:** [@rajivmca2004](https://discuss.elastic.co/u/rajivmca2004)\ **Replies:** 1\ **Last updated:** [June 6, 2016, 8:26pm UTC](https://discuss.elastic.co/t/logstash-sync-from-oracle-to-mongodb-and-elasticsearch-insert-duplicate-records-when-doing-partial-update/31374 "2016-06-06T20:26:53Z")
We are facing issue of getting duplicate records with Logstash JDBC plugin when syncing Oracle to MongoDB and Elasticsearch after every 1 minute. How we can do partial update ,so that delta/modified records should only … --- ## [Ganglia into logstash](https://discuss.elastic.co/t/ganglia-into-logstash/47000)
**Author:** [@jfisher](https://discuss.elastic.co/u/jfisher)\ **Replies:** 2\ **Last updated:** [June 6, 2016, 7:03pm UTC](https://discuss.elastic.co/t/ganglia-into-logstash/47000 "2016-06-06T19:03:34Z")
I know this isn't the first issue with ganglia, but I've dug through all the other forum posts on it and it's not getting anywhere. I've established through the command "tcpdump -i any udp" that I'm getting the UDP ganli… --- ## [How can I add a list of fields?](https://discuss.elastic.co/t/how-can-i-add-a-list-of-fields/51991)
**Author:** [@iamthealex](https://discuss.elastic.co/u/iamthealex)\ **Replies:** 0\ **Last updated:** [June 6, 2016, 6:46pm UTC](https://discuss.elastic.co/t/how-can-i-add-a-list-of-fields/51991 "2016-06-06T18:46:19Z")
I know how to use mutate add\_field to add a field. I know how to use mutate add\_tag to add to \[tags\]. Is there any way to add a list of fields with structure? I'd like to add something like this to the top-level json … --- ## [Logstash Ruby code](https://discuss.elastic.co/t/logstash-ruby-code/51712)
**Author:** [@sharma.shilpy](https://discuss.elastic.co/u/sharma.shilpy)\ **Replies:** 4\ **Last updated:** [June 6, 2016, 4:35pm UTC](https://discuss.elastic.co/t/logstash-ruby-code/51712 "2016-06-06T16:35:30Z")
Hi guys, I am trying to convert time: HH:mm:ss into seconds using ruby code inside the logstash configuration file and getting error message: \_ Ruby exception occurred: uninitialized constant LogStash::Time {:level=\>:… --- ## [Logstash SQLite input without write permission to database](https://discuss.elastic.co/t/logstash-sqlite-input-without-write-permission-to-database/51969)
**Author:** [@yannakopoulos](https://discuss.elastic.co/u/yannakopoulos)\ **Replies:** 0\ **Last updated:** [June 6, 2016, 3:08pm UTC](https://discuss.elastic.co/t/logstash-sqlite-input-without-write-permission-to-database/51969 "2016-06-06T15:08:17Z")
I have an SQLite database I'd like to read in real time, but I won't have write permissions to it. From what I understand, the SQLite input plugin needs to create the equivalent of other input method's .sincedb files … --- ## [Dynamic Nested fields](https://discuss.elastic.co/t/dynamic-nested-fields/51949)
**Author:** [@tharris](https://discuss.elastic.co/u/tharris)\ **Replies:** 0\ **Last updated:** [June 6, 2016, 2:08pm UTC](https://discuss.elastic.co/t/dynamic-nested-fields/51949 "2016-06-06T14:08:24Z")
Hi guys, I’m looking to make some Elasticsearch documents in a very similar format to what is shown here: Nested Objects | Elasticsearch: The Definitive Guide \[2.x\] | Elastic I started off inputting a log into Logstash… --- ## [Scaling up logstash to process 100 requests per second](https://discuss.elastic.co/t/scaling-up-logstash-to-process-100-requests-per-second/51896)
**Author:** [@elssar](https://discuss.elastic.co/u/elssar)\ **Replies:** 0\ **Last updated:** [June 6, 2016, 5:25am UTC](https://discuss.elastic.co/t/scaling-up-logstash-to-process-100-requests-per-second/51896 "2016-06-06T05:25:51Z")
I have a simple logstash that has a single http input, running behind a nginx proxy on a m4.2xlarge machine on AWS. I have allocated 2 gb of heap to logstash. I noticed that there are a lot of connections in TIME-WAIT … --- ## [How to convert a timestamp field to add a new field like "YYYY-MM-dd"](https://discuss.elastic.co/t/how-to-convert-a-timestamp-field-to-add-a-new-field-like-yyyy-mm-dd/51893)
**Author:** [@changxy](https://discuss.elastic.co/u/changxy)\ **Replies:** 1\ **Last updated:** [June 6, 2016, 4:40am UTC](https://discuss.elastic.co/t/how-to-convert-a-timestamp-field-to-add-a-new-field-like-yyyy-mm-dd/51893 "2016-06-06T04:40:13Z")
i need add a new field from a date filed this date filed value is : 1465071797 $ date -d @1465071797 Sun Jun 5 04:23:17 CST 2016 the new field hope looks like "2016-06-05" --- ## [Config '\_id' generation](https://discuss.elastic.co/t/config--id-generation/51860)
**Author:** [@evgeniy](https://discuss.elastic.co/u/evgeniy)\ **Replies:** 2\ **Last updated:** [June 5, 2016, 8:56pm UTC](https://discuss.elastic.co/t/config--id-generation/51860 "2016-06-05T20:56:31Z")
Hi all, How I can set what fields to use in '\_id' generation? I use logstash to fetch log files from different places. Sometimes those files are identical. So I want to prevent elasticsearch from adding dublicated lo… --- ## [Nothing is getting indexed](https://discuss.elastic.co/t/nothing-is-getting-indexed/51616)
**Author:** [@Deepak\_Patnaik](https://discuss.elastic.co/u/Deepak_Patnaik)\ **Replies:** 4\ **Last updated:** [June 5, 2016, 6:31pm UTC](https://discuss.elastic.co/t/nothing-is-getting-indexed/51616 "2016-06-05T18:31:14Z")
Hi, I am new to ELK. I had some log files indexed. Then I deleted the indexes using XDELETE and \_all. Now when i run logstash, and try to see if the files get indexed again. it does not show any index. Please help. … --- ## [Logstash - newbie question - how to serach single event for a pattern, string](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742)
**Author:** [@mke](https://discuss.elastic.co/u/mke)\ **Replies:** 18\ **Last updated:** [June 5, 2016, 3:33pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742 "2016-06-05T15:33:27Z")
Hi, I am just beginning my story with ELK, trying to understand grok and how to filter events Here is what I try to accomplish, I try to gather syslogs from routers and switches, here is example I 05/23/16 11:50:14 … --- ## [Problems with lumberjack input and encrypted private key](https://discuss.elastic.co/t/problems-with-lumberjack-input-and-encrypted-private-key/51877)
**Author:** [@jammann](https://discuss.elastic.co/u/jammann)\ **Replies:** 0\ **Last updated:** [June 5, 2016, 1:24pm UTC](https://discuss.elastic.co/t/problems-with-lumberjack-input-and-encrypted-private-key/51877 "2016-06-05T13:24:25Z")
I'm trying to get logstash 2.3.1 working with a lumberjack input with TLS. As long as I'm using an unencrypted private key, everything is fine. When I'm using a encrypted private key and an passphrase, all I'm getting is … --- ## [Logstash error when starting "logstash: no such user"](https://discuss.elastic.co/t/logstash-error-when-starting-logstash-no-such-user/51200)
**Author:** [@yeaiforgot](https://discuss.elastic.co/u/yeaiforgot)\ **Replies:** 2\ **Last updated:** [June 4, 2016, 7:09pm UTC](https://discuss.elastic.co/t/logstash-error-when-starting-logstash-no-such-user/51200 "2016-06-04T19:09:06Z")
Hi all! I'm running ELK on a CentOS 7 server and running into an issue when trying to start Logstash. Belos is some output from Logstash status. May 27 14:06:29 x.x.x.com logstash\[3896\]: id: logstash: no such user T… --- ## [Config file compatibility](https://discuss.elastic.co/t/config-file-compatibility/51743)
**Author:** [@zozo6015](https://discuss.elastic.co/u/zozo6015)\ **Replies:** 9\ **Last updated:** [June 4, 2016, 3:02pm UTC](https://discuss.elastic.co/t/config-file-compatibility/51743 "2016-06-04T15:02:08Z")
Hello, I would like to ask if Logstash 5.0.0 alpha3 configuration is compatible with the version 2 config file? I am asking cause I have copied my old version 2.x config file into /etc/logstash/conf.d/ but in elastic… --- ## [Shipping Apache log files to ELK](https://discuss.elastic.co/t/shipping-apache-log-files-to-elk/51833)
**Author:** [@Abe\_Bazouie](https://discuss.elastic.co/u/Abe_Bazouie)\ **Replies:** 1\ **Last updated:** [June 4, 2016, 8:26am UTC](https://discuss.elastic.co/t/shipping-apache-log-files-to-elk/51833 "2016-06-04T08:26:55Z")
Hi guys, I have a qustion that might be a little silly. I did lots of Google but I think my issue is completely fundamental because I am ELK newbie \[:wink:\] I wounder to know how can I ship the Apache log files to the… --- ## [Grok Filter + Expect Blank value into Variable/Parameter](https://discuss.elastic.co/t/grok-filter-expect-blank-value-into-variable-parameter/51621)
**Author:** [@sarthak2016](https://discuss.elastic.co/u/sarthak2016)\ **Replies:** 6\ **Last updated:** [June 3, 2016, 4:56pm UTC](https://discuss.elastic.co/t/grok-filter-expect-blank-value-into-variable-parameter/51621 "2016-06-03T16:56:24Z")
Hello All, Here is grok Filter grok { match =\> \["message", "%{DATESTAMP:timestamp1},%{GREEDYDATA:logmsg1},%{GREEDYDATA:logmsg2},%{INT:auth\_method1},%{INT:auth\_method2},%{GREEDYDATA:logmsg}"\] add\_field =\> \[ … --- ## [Grok filter problem](https://discuss.elastic.co/t/grok-filter-problem/51748)
**Author:** [@etfeet](https://discuss.elastic.co/u/etfeet)\ **Replies:** 2\ **Last updated:** [June 3, 2016, 3:19pm UTC](https://discuss.elastic.co/t/grok-filter-problem/51748 "2016-06-03T15:19:49Z")
I'm having problems matching against a string in the middle of a field field data contains: "change-data" =\> "{ gtm\_pool\_member { gtm\_pool\_member\_pool\_type 1 gtm\_pool\_member\_pool\_name /Common/panel\_preprod… --- ## [JSON data validation using scheme](https://discuss.elastic.co/t/json-data-validation-using-scheme/51559)
**Author:** [@michaltptk](https://discuss.elastic.co/u/michaltptk)\ **Replies:** 3\ **Last updated:** [June 3, 2016, 1:52pm UTC](https://discuss.elastic.co/t/json-data-validation-using-scheme/51559 "2016-06-03T13:52:08Z")
Is there a possibility to validate JSON data using a proper JSON scheme? I put the data into Logstash, parsed it using JSON plugin but I want to make sure that the data match the given pattern. There are few fields where… --- ## [Is there a way to check if logstash environment variables got set](https://discuss.elastic.co/t/is-there-a-way-to-check-if-logstash-environment-variables-got-set/51801)
**Author:** [@dkota](https://discuss.elastic.co/u/dkota)\ **Replies:** 0\ **Last updated:** [June 3, 2016, 1:47pm UTC](https://discuss.elastic.co/t/is-there-a-way-to-check-if-logstash-environment-variables-got-set/51801 "2016-06-03T13:47:39Z")
is there a way to check if the LS\_OPEN\_FILES set in logstash configuration is actually getting set? --- ## [Logstash field matching](https://discuss.elastic.co/t/logstash-field-matching/51791)
**Author:** [@Kishore](https://discuss.elastic.co/u/Kishore)\ **Replies:** 2\ **Last updated:** [June 3, 2016, 11:24am UTC](https://discuss.elastic.co/t/logstash-field-matching/51791 "2016-06-03T11:24:02Z")
I want to use one of the field from the following output in translate filter. Below is the output looks like. { "message" =\> "May 19 08:27:11 ip-21-54-0-85 openvpn MULTI\_sva: pool returned IPv4=12.24.254.14,… --- ## [Config File Cannot Work on Logstash 2.3](https://discuss.elastic.co/t/config-file-cannot-work-on-logstash-2-3/51783)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 1\ **Last updated:** [June 3, 2016, 11:23am UTC](https://discuss.elastic.co/t/config-file-cannot-work-on-logstash-2-3/51783 "2016-06-03T11:23:54Z")
Is there any information on how config file written for logstash 2.2 should be modified in order to be used on logstash 2.3.x? All my config fails to work. Nothing runs. All I get is this JAVA\_OPTS was set to \[ -… --- ## [Grokking log file](https://discuss.elastic.co/t/grokking-log-file/51751)
**Author:** [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\ **Replies:** 4\ **Last updated:** [June 3, 2016, 6:15am UTC](https://discuss.elastic.co/t/grokking-log-file/51751 "2016-06-03T06:15:41Z")
I have been using logstash to read a log file in which I have come across with some difficulty in grokking the log statement. The log pattern is like this "20160401-00:00:00","abc3ap5a","AB2","EAP-Production(Productio… --- ## [Compare IP's in netflow to Reputational List](https://discuss.elastic.co/t/compare-ips-in-netflow-to-reputational-list/51708)
**Author:** [@Bradly](https://discuss.elastic.co/u/Bradly)\ **Replies:** 1\ **Last updated:** [June 3, 2016, 3:09am UTC](https://discuss.elastic.co/t/compare-ips-in-netflow-to-reputational-list/51708 "2016-06-03T03:09:32Z")
I am trying to figure out how to write a filter that will add a tag to any ip's that come out of netflow that are listed on specific blacklist (zeuss, edrop, ect). I have the list and I have the netflow data coming into… --- ## [How to extract fields from the hadoop logs using logstash](https://discuss.elastic.co/t/how-to-extract-fields-from-the-hadoop-logs-using-logstash/51649)
**Author:** [@rohit\_prusty](https://discuss.elastic.co/u/rohit_prusty)\ **Replies:** 1\ **Last updated:** [June 3, 2016, 1:13am UTC](https://discuss.elastic.co/t/how-to-extract-fields-from-the-hadoop-logs-using-logstash/51649 "2016-06-03T01:13:33Z")
My requirement it to parse the hadoop hdfs-audit log file using logstash and extract the operation type and add this as an additional field along with the complete message before storing the event to the elasticsearch. … [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1144) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1146)