# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=724 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 725 --- ## [Customize document fields in elasticsearch](https://discuss.elastic.co/t/customize-document-fields-in-elasticsearch/139522)
**Author:** [@waitangi](https://discuss.elastic.co/u/waitangi)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 11:25am UTC](https://discuss.elastic.co/t/customize-document-fields-in-elasticsearch/139522 "2018-07-11T11:25:02Z")
I need to customize document fields in elasticsearch. My logstash config is: input { file { type =\> "IISLog" path =\> "C:/inetpub/logs/LogFiles/W3SVC\*/\*.log" start\_position =\> "beginning" } } filter { #ignore co… --- ## [Problems doing an \_update\_by\_query with the logstash http output plugin](https://discuss.elastic.co/t/problems-doing-an-update-by-query-with-the-logstash-http-output-plugin/139057)
**Author:** [@Efrain\_Gonzalez](https://discuss.elastic.co/u/Efrain_Gonzalez)\ **Replies:** 3\ **Last updated:** [July 11, 2018, 10:58am UTC](https://discuss.elastic.co/t/problems-doing-an-update-by-query-with-the-logstash-http-output-plugin/139057 "2018-07-11T10:58:48Z")
Using kibana's Dev tools I have a working REST api call POST /unicast-prefixes-x.x.x.x/\_update\_by\_query { "script" : { "source" : "if (ctx.\_source.containsKey('tags')) { ctx.\_source.tags.ad… --- ## [\[ANNOUNCEMENT\] logstash-filter-test](https://discuss.elastic.co/t/announcement-logstash-filter-test/139423)
**Author:** [@noamraph](https://discuss.elastic.co/u/noamraph)\ **Replies:** 2\ **Last updated:** [July 11, 2018, 9:42am UTC](https://discuss.elastic.co/t/announcement-logstash-filter-test/139423 "2018-07-11T09:42:30Z")
Hi, I tried logstash-filter-verifier for a few days and grew frustrated with it. So I wrote my own tester: logstash-filter-test. I find it to be much simpler and easier. Here is an example test suite file: \[ // Test a… --- ## [Logs are missing in elasticsearch](https://discuss.elastic.co/t/logs-are-missing-in-elasticsearch/139300)
**Author:** [@clash\_of\_clanss](https://discuss.elastic.co/u/clash_of_clanss)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 9:27am UTC](https://discuss.elastic.co/t/logs-are-missing-in-elasticsearch/139300 "2018-07-11T09:27:27Z")
Hi, I am sending log files to Elasticsearch cluster through logstash, but log lines are missing while searching in kibana and I haven't seen any error in logstash and elasticsearch side. Thanks, --- ## [Logstash: Adding custom field to documennt](https://discuss.elastic.co/t/logstash-adding-custom-field-to-documennt/139361)
**Author:** [@delly.fofie](https://discuss.elastic.co/u/delly.fofie)\ **Replies:** 5\ **Last updated:** [July 11, 2018, 9:21am UTC](https://discuss.elastic.co/t/logstash-adding-custom-field-to-documennt/139361 "2018-07-11T09:21:06Z")
Hi all. I have the following XML file format: \ \ \ \ **Author:** [@bab](https://discuss.elastic.co/u/bab)\ **Replies:** 6\ **Last updated:** [July 11, 2018, 7:54am UTC](https://discuss.elastic.co/t/parsing-different-dates-from-xml-logs-in-one-date-filed/139205 "2018-07-11T07:54:00Z") Hi all, I have different types of timestamps coming into my logstash input from different logfiles xml and log type: My Configuration for filebeat looks like below: filebeat.inputs: - type: log paths: - /home/De… --- ## [Required Logstash grok pattern to read tableau server log](https://discuss.elastic.co/t/required-logstash-grok-pattern-to-read-tableau-server-log/139162)
**Author:** [@swatititame](https://discuss.elastic.co/u/swatititame)\ **Replies:** 0\ **Last updated:** [July 9, 2018, 12:05pm UTC](https://discuss.elastic.co/t/required-logstash-grok-pattern-to-read-tableau-server-log/139162 "2018-07-09T12:05:43Z")
Hi, Sample Log Data com.tableausoftware.model.workgroup.workers.EnqueueActiveDirectoryGroupSyncWorker - Sync group background job for group corp.intusurg.com\\TBX-Dept-Quality-SU has been scheduled. Site id=\[3\], schedu… --- ## [Copy \`@timestamp\` in Logstash](https://discuss.elastic.co/t/copy-timestamp-in-logstash/139430)
**Author:** [@amomchilov](https://discuss.elastic.co/u/amomchilov)\ **Replies:** 2\ **Last updated:** [July 11, 2018, 6:25am UTC](https://discuss.elastic.co/t/copy-timestamp-in-logstash/139430 "2018-07-11T06:25:20Z")
I would like to preserve the @timestamp field prior to overwriting it from a date filter, so that I can know when a document was processed by logstash. I also tried using the copy filter, but that fails silently. mut… --- ## [JDBC Input set jdbc\_default\_timezone loses precision](https://discuss.elastic.co/t/jdbc-input-set-jdbc-default-timezone-loses-precision/139408)
**Author:** [@skatemple](https://discuss.elastic.co/u/skatemple)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 6:14am UTC](https://discuss.elastic.co/t/jdbc-input-set-jdbc-default-timezone-loses-precision/139408 "2018-07-11T06:14:01Z")
So I'm facing a duplicate info issue because of a malformed SQL query on JDBC input type. I saw this issue (https://github.com/logstash-plugins/logstash-input-jdbc/issues/140) but I'm still getting that bug. The last s… --- ## [Update in mongodb](https://discuss.elastic.co/t/update-in-mongodb/139466)
**Author:** [@Chiranjeevi\_Medichar](https://discuss.elastic.co/u/Chiranjeevi_Medichar)\ **Replies:** 0\ **Last updated:** [July 11, 2018, 4:36am UTC](https://discuss.elastic.co/t/update-in-mongodb/139466 "2018-07-11T04:36:48Z")
how can i update record in mongodb using logstash output --- ## [Access custom fields on nginx from logstash](https://discuss.elastic.co/t/access-custom-fields-on-nginx-from-logstash/139119)
**Author:** [@Jonghun\_Park1](https://discuss.elastic.co/u/Jonghun_Park1)\ **Replies:** 4\ **Last updated:** [July 11, 2018, 2:30am UTC](https://discuss.elastic.co/t/access-custom-fields-on-nginx-from-logstash/139119 "2018-07-11T02:30:44Z")
I've wrote a nginx log\_format as below. log\_format main '$http\_host $remote\_addr - $remote\_user \[$time\_local\] "$request" $status $body\_bytes\_sent "$http\_referer" "$http\_user\_agent" "$http\_x\_forwarded\_for" "$http\_user\_br… --- ## [How to interpret Logstash Pipeline Stats](https://discuss.elastic.co/t/how-to-interpret-logstash-pipeline-stats/139440)
**Author:** [@anhlqn](https://discuss.elastic.co/u/anhlqn)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 11:28pm UTC](https://discuss.elastic.co/t/how-to-interpret-logstash-pipeline-stats/139440 "2018-07-10T23:28:59Z")
I've just upgraded from Logstash 2.4.0 to 5.6.8 with the same configuration file. LS 5.6.8 runs for a while and the process takes over 100% CPU. If I restarted LS, it will then consumes 100% CPU again. I'm using quite a … --- ## [Logstash Filter development: rspec test for filter success/failure](https://discuss.elastic.co/t/logstash-filter-development-rspec-test-for-filter-success-failure/139222)
**Author:** [@fholzer](https://discuss.elastic.co/u/fholzer)\ **Replies:** 1\ **Last updated:** [July 10, 2018, 9:56pm UTC](https://discuss.elastic.co/t/logstash-filter-development-rspec-test-for-filter-success-failure/139222 "2018-07-10T21:56:28Z")
Hi, i couldn't find a way to test for filter success, i.e. whether the filter\_matched method was called on an event. I tried to find this by looking at the tests of some of the official filter plugins but couldn't find … --- ## [Elasticsearch output connection pool error, with Linux VIP, but still indexing](https://discuss.elastic.co/t/elasticsearch-output-connection-pool-error-with-linux-vip-but-still-indexing/139427)
**Author:** [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 8:40pm UTC](https://discuss.elastic.co/t/elasticsearch-output-connection-pool-error-with-linux-vip-but-still-indexing/139427 "2018-07-10T20:40:47Z")
Hi running 6.2.4 I have the following config... elasticsearch { hosts =\> \["data6dev.marathon.l4lb.thisdcos.directory:9200"\] user =\> "elastic" password =\> "xxxxxx" index =\> "dcos-%{+YYYY.MM.dd}" } data6dev.mara… --- ## [Cannot run logstash using cmd please help!](https://discuss.elastic.co/t/cannot-run-logstash-using-cmd-please-help/139426)
**Author:** [@Shraddha\_Srivastav](https://discuss.elastic.co/u/Shraddha_Srivastav)\ **Replies:** 3\ **Last updated:** [July 10, 2018, 9:25pm UTC](https://discuss.elastic.co/t/cannot-run-logstash-using-cmd-please-help/139426 "2018-07-10T21:25:07Z")
Hi, I installed logstash ZIP folder using https://www.elastic.co/downloads/logstash. Also prepared a logstash.conf config file, followed the steps as explained on this: https://www.elastic.co/guide/en/logstash/current/c… --- ## [Unable to resolve error-Logstash .conf file(CentOS 7 Linux)](https://discuss.elastic.co/t/unable-to-resolve-error-logstash-conf-file-centos-7-linux/139390)
**Author:** [@sdubbudu](https://discuss.elastic.co/u/sdubbudu)\ **Replies:** 4\ **Last updated:** [July 10, 2018, 8:39pm UTC](https://discuss.elastic.co/t/unable-to-resolve-error-logstash-conf-file-centos-7-linux/139390 "2018-07-10T20:39:30Z")
Hi all I have recently started using the Elastic Stack. I have everything up and running. But when I try to create pipeline I get a very weird error which I dont find any where in my logstash .conf file I am posting bel… --- ## [Pipeline yaml file cannot run multiple conf](https://discuss.elastic.co/t/pipeline-yaml-file-cannot-run-multiple-conf/139419)
**Author:** [@sukesh\_nagaraja](https://discuss.elastic.co/u/sukesh_nagaraja)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 7:02pm UTC](https://discuss.elastic.co/t/pipeline-yaml-file-cannot-run-multiple-conf/139419 "2018-07-10T19:02:28Z")
Error : \[2018-07-10T14:50:49,640\]\[FATAL\]\[logstash.runner \] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple i… --- ## [Logstash Variables accros files](https://discuss.elastic.co/t/logstash-variables-accros-files/139418)
**Author:** [@Sebastian\_Herrera](https://discuss.elastic.co/u/Sebastian_Herrera)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 6:52pm UTC](https://discuss.elastic.co/t/logstash-variables-accros-files/139418 "2018-07-10T18:52:54Z")
Hi all, i there a way to have a global variable that i can use from different files? I am wondering to have a variable managed via an automated system where i can have some static values that i will need in different pla… --- ## [Ruby Filter - Event.Remove Inconsistent Results](https://discuss.elastic.co/t/ruby-filter-event-remove-inconsistent-results/138268)
**Author:** [@kazul33](https://discuss.elastic.co/u/kazul33)\ **Replies:** 8\ **Last updated:** [July 10, 2018, 5:35pm UTC](https://discuss.elastic.co/t/ruby-filter-event-remove-inconsistent-results/138268 "2018-07-10T17:35:29Z")
Problem Statement I believe there is a bug in the ruby filter plugin when using event.remove Expected Output: ... event\_data.sql\_text =\> "some sql text" event\_data.logType =\> "Oracle" ... With no root fields (e.g., sql\_… --- ## [Logstash error while receiving filebeat messages](https://discuss.elastic.co/t/logstash-error-while-receiving-filebeat-messages/139380)
**Author:** [@brunoof1](https://discuss.elastic.co/u/brunoof1)\ **Replies:** 1\ **Last updated:** [July 10, 2018, 5:20pm UTC](https://discuss.elastic.co/t/logstash-error-while-receiving-filebeat-messages/139380 "2018-07-10T17:20:47Z")
I have been stashing files from filebeat, Here the file structure: Jul 10 11:50:32 prod-es-ingest1e logstash\[1061\]: "statusResponse" =\> "PROCESSED", Jul 10 11:50:32 prod-es-ingest1e logstash\[1061\]: "of… --- ## [Create a new field using Regular Expressions](https://discuss.elastic.co/t/create-a-new-field-using-regular-expressions/139397)
**Author:** [@martb](https://discuss.elastic.co/u/martb)\ **Replies:** 2\ **Last updated:** [July 10, 2018, 4:37pm UTC](https://discuss.elastic.co/t/create-a-new-field-using-regular-expressions/139397 "2018-07-10T16:37:26Z")
Hi, I'm trying to extract part of a field using RegEx and create a new field with the extracted text. i.e string = alert message|User: johnc |Host: test |Event type: test capture = johnc RegEx example : User: (… --- ## [Logstash and Ruby](https://discuss.elastic.co/t/logstash-and-ruby/138555)
**Author:** [@dersoi](https://discuss.elastic.co/u/dersoi)\ **Replies:** 10\ **Last updated:** [July 10, 2018, 3:30pm UTC](https://discuss.elastic.co/t/logstash-and-ruby/138555 "2018-07-10T15:30:56Z")
Hello, I am trying to integrate a Ruby script into Logstash, but I am having a hard time... Basically, I have a field which I want to modify in Ruby and then give back to Logstash. What I'm having a hard time with is th… --- ## [No configuration sources](https://discuss.elastic.co/t/no-configuration-sources/139371)
**Author:** [@oumeyma](https://discuss.elastic.co/u/oumeyma)\ **Replies:** 1\ **Last updated:** [July 10, 2018, 2:36pm UTC](https://discuss.elastic.co/t/no-configuration-sources/139371 "2018-07-10T14:36:15Z")
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22\] sourceloader - No configuration found in the configured sources. --- ## [Warn on testing the conf of logstash](https://discuss.elastic.co/t/warn-on-testing-the-conf-of-logstash/139346)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 4\ **Last updated:** [July 10, 2018, 2:27pm UTC](https://discuss.elastic.co/t/warn-on-testing-the-conf-of-logstash/139346 "2018-07-10T14:27:32Z")
Hi, Please why do I have this warn ? \[root@frghcslnetv10 conf.d\]# /usr/share/logstash/bin/logstash --config.test\_and\_exit --path.settings /etc/logstash -f /etc/logstash/conf.d/parsing-log.conf Sending Logstash's logs t… --- ## [\[FATAL\]\[logstash.runner \] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting](https://discuss.elastic.co/t/fatal-logstash-runner-logstash-could-not-be-started-because-there-is-already-another-instance-using-the-configured-data-directory-if-you-wish-to-run-multiple-instances-you-must-change-the-path-data-setting/139362)
**Author:** [@dyl](https://discuss.elastic.co/u/dyl)\ **Replies:** 1\ **Last updated:** [July 10, 2018, 2:21pm UTC](https://discuss.elastic.co/t/fatal-logstash-runner-logstash-could-not-be-started-because-there-is-already-another-instance-using-the-configured-data-directory-if-you-wish-to-run-multiple-instances-you-must-change-the-path-data-setting/139362 "2018-07-10T14:21:29Z")
Hello; I try to run two pipelines together but When I launch my pipelines with the command : bin/logstash --path.settings /etc/logstash I've got this error : \[FATAL\]\[logstash.runner \] Logstash could not be started be… --- ## [Error LogStash::Inputs::Elasticsearch](https://discuss.elastic.co/t/error-logstash-elasticsearch/139363)
**Author:** [@npontes](https://discuss.elastic.co/u/npontes)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 2:03pm UTC](https://discuss.elastic.co/t/error-logstash-elasticsearch/139363 "2018-07-10T14:03:28Z")
Hello everyone, I have a .conf file which I will execute (logstash -f file.conf -w 1) to extract data from ES to a csv file. I can do it perfectly on my dev machine. The problem is when we try to execute it on machine… --- ## [How to filter nested JSON Data in Logstash](https://discuss.elastic.co/t/how-to-filter-nested-json-data-in-logstash/139124)
**Author:** [@Sami\_Hilaneh](https://discuss.elastic.co/u/Sami_Hilaneh)\ **Replies:** 8\ **Last updated:** [July 10, 2018, 1:49pm UTC](https://discuss.elastic.co/t/how-to-filter-nested-json-data-in-logstash/139124 "2018-07-10T13:49:18Z")
hey all, i am trying to get some specific fileds(filter them) from a big nested JSON Data, i have like example this JSON Data: { "agent": "Mozilla/5.0 (compatible; MSIE 9.0)", "ip": "192.168.24.44", "request": "/in… --- ## [I can't parse my logs anymore](https://discuss.elastic.co/t/i-cant-parse-my-logs-anymore/139117)
**Author:** [@asalma](https://discuss.elastic.co/u/asalma)\ **Replies:** 6\ **Last updated:** [July 10, 2018, 1:29pm UTC](https://discuss.elastic.co/t/i-cant-parse-my-logs-anymore/139117 "2018-07-10T13:29:12Z")
\[2018-07-09T10:56:45,779\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>400, :action=\>\["index", {:\_id=\>"c5653e88eeea542ff562333cfea5db5e6bba4f3e", :\_index=\>"logstash-2018.07.09"… --- ## [Logstash config JDBC Password](https://discuss.elastic.co/t/logstash-config-jdbc-password/139255)
**Author:** [@Preethi](https://discuss.elastic.co/u/Preethi)\ **Replies:** 3\ **Last updated:** [July 10, 2018, 1:17pm UTC](https://discuss.elastic.co/t/logstash-config-jdbc-password/139255 "2018-07-10T13:17:54Z")
Hi, Would like to know whether can use environmental variables to get the username and password for JDBC plugin in log stash config file? Or what is the safest way to pass the password to JDBC connection? Thanks --- ## [How to associate exactly one partition with one logstash consumer](https://discuss.elastic.co/t/how-to-associate-exactly-one-partition-with-one-logstash-consumer/139343)
**Author:** [@Krishna\_Oza](https://discuss.elastic.co/u/Krishna_Oza)\ **Replies:** 0\ **Last updated:** [July 10, 2018, 12:23pm UTC](https://discuss.elastic.co/t/how-to-associate-exactly-one-partition-with-one-logstash-consumer/139343 "2018-07-10T12:23:26Z")
I have a kafka producer which is producing messages for two topics and for each topic I have allocated 4 partitions. I have allocated 8 consumer threads for the above two topics and hence assume that each consumer threa… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=723) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=725)