# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=662 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 663 --- ## [Ship /var/log/secure onto Elasticsearch](https://discuss.elastic.co/t/ship-var-log-secure-onto-elasticsearch/153659)
**Author:** [@rrs](https://discuss.elastic.co/u/rrs)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 5:23pm UTC](https://discuss.elastic.co/t/ship-var-log-secure-onto-elasticsearch/153659 "2018-10-23T17:23:16Z")
Hi Team, We have setup ELK with 5.6.3 to forward all RHEL/Centos system logs. Initially, we are in process of shipping /var/log/secure logs which file has different events such as ssh, su, and sudo events each with diff… --- ## [Update by Query in Logstash](https://discuss.elastic.co/t/update-by-query-in-logstash/153653)
**Author:** [@Andrew\_Stroz](https://discuss.elastic.co/u/Andrew_Stroz)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 4:56pm UTC](https://discuss.elastic.co/t/update-by-query-in-logstash/153653 "2018-10-23T16:56:13Z")
I am ingesting documents continually into elastic and one of my main search queries is getting the most recent document in a series. I am currently doing this using a top\_hits aggregation and then sorting the results wit… --- ## [Field reference for Numbers](https://discuss.elastic.co/t/field-reference-for-numbers/153647)
**Author:** [@redX](https://discuss.elastic.co/u/redX)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 4:03pm UTC](https://discuss.elastic.co/t/field-reference-for-numbers/153647 "2018-10-23T16:03:18Z")
Can I use field references for number fields? For example I have an aggregate filter with a timeout. The timeout configuration expects a number. I have: if \[type\] == "start" { aggregate { task\_id =\> "%{filename… --- ## [Input Exclude File Option with dot in filename behaviour](https://discuss.elastic.co/t/input-exclude-file-option-with-dot-in-filename-behaviour/153645)
**Author:** [@londonx](https://discuss.elastic.co/u/londonx)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 3:57pm UTC](https://discuss.elastic.co/t/input-exclude-file-option-with-dot-in-filename-behaviour/153645 "2018-10-23T15:57:18Z")
I have noticed (Logstash 6.4.2) that the file exclude option in the file input plugin does not handle files with dots in the filename..... e.g. aaa.bbbb-ERRORS-yyyy-mm-dd.log these work: path =\> \[ "C:/Elastic/TestAppL… --- ## [Grok - extract text from in between double quotes](https://discuss.elastic.co/t/grok-extract-text-from-in-between-double-quotes/153644)
**Author:** [@jimmy54321](https://discuss.elastic.co/u/jimmy54321)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 3:45pm UTC](https://discuss.elastic.co/t/grok-extract-text-from-in-between-double-quotes/153644 "2018-10-23T15:45:42Z")
Hi, Very new to logstash and grok filtering and I expect this is a really easy thing to do but I've been stuck on it all day after googling lots and not figuring it out. Any help would be very appreciated! I'm looking … --- ## [Clone multiple JSON events from input source count](https://discuss.elastic.co/t/clone-multiple-json-events-from-input-source-count/153512)
**Author:** [@rocket](https://discuss.elastic.co/u/rocket)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 3:55am UTC](https://discuss.elastic.co/t/clone-multiple-json-events-from-input-source-count/153512 "2018-10-23T03:55:29Z")
Receiving JSON input lines with count value as one of the fields. Count value should be used to clone elasticsearch events before ingesting in elasticsearch. Tried ruby code below to loop the count and could not get the … --- ## [Grok Filter Parsing](https://discuss.elastic.co/t/grok-filter-parsing/153641)
**Author:** [@rahprasingh](https://discuss.elastic.co/u/rahprasingh)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 3:19pm UTC](https://discuss.elastic.co/t/grok-filter-parsing/153641 "2018-10-23T15:19:09Z")
Hi Everyone , Can any one help me on some issue , below is my application logs 10.92.9.247 bdrvl-pd-fif03 - - \[23/Oct/2018:19:02:04 +0530\] "GET /inventory/api/gis/fiberRoute?lat=12.74266438&lon=77.83363018&distance=60… --- ## [Grok filter on dynamic source field name](https://discuss.elastic.co/t/grok-filter-on-dynamic-source-field-name/153640)
**Author:** [@rodLN](https://discuss.elastic.co/u/rodLN)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 3:13pm UTC](https://discuss.elastic.co/t/grok-filter-on-dynamic-source-field-name/153640 "2018-10-23T15:13:59Z")
I'm applying a grok filter followed by a kv successfully on a log entry that looks like this: "Mylog\[something=1;somethingelse=NA;another=19163;exception@3257ms=3;total=3258ms;\]" Everything works well for the correctly… --- ## [Reference field dynamically from another event](https://discuss.elastic.co/t/reference-field-dynamically-from-another-event/153639)
**Author:** [@IC91](https://discuss.elastic.co/u/IC91)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 3:13pm UTC](https://discuss.elastic.co/t/reference-field-dynamically-from-another-event/153639 "2018-10-23T15:13:59Z")
So I am using Http poller input plugin to poll two different URLs, So I am trying to add a new field in the second URL response(JSON) as from first URL event response: \` add\_field =\> {"\[CountyOutageInfo\]\[0\]\[StateName… --- ## [Need suggestions/possibilities for a problem](https://discuss.elastic.co/t/need-suggestions-possibilities-for-a-problem/153453)
**Author:** [@IC91](https://discuss.elastic.co/u/IC91)\ **Replies:** 0\ **Last updated:** [October 22, 2018, 5:53pm UTC](https://discuss.elastic.co/t/need-suggestions-possibilities-for-a-problem/153453 "2018-10-22T17:53:39Z")
Hey guys, I am using http input poller to poll 2 urls, my requirement is to aggregate the JSON replies based on a common field which is "StateFIPS" into a single structure, Here is my data: Data 1: Give results for all… --- ## [How to retrieve JSON object from log in logstash confuguration?](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375)
**Author:** [@Vyankatesh\_S\_Repal](https://discuss.elastic.co/u/Vyankatesh_S_Repal)\ **Replies:** 4\ **Last updated:** [October 23, 2018, 11:50am UTC](https://discuss.elastic.co/t/how-to-retrieve-json-object-from-log-in-logstash-confuguration/153375 "2018-10-23T11:50:59Z")
I am creating an application where I need to retrieve key value pairs from json object in logs with the help of logstash configuraition. Here is my configuration: input{ file{ path =\> "D:\\ELK\_Info\\TestLogs\_Upd… --- ## [Can dissect filter be used, when message contains unknown number of names separated by comma](https://discuss.elastic.co/t/can-dissect-filter-be-used-when-message-contains-unknown-number-of-names-separated-by-comma/153581)
**Author:** [@Preben\_Nilsson](https://discuss.elastic.co/u/Preben_Nilsson)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 10:47am UTC](https://discuss.elastic.co/t/can-dissect-filter-be-used-when-message-contains-unknown-number-of-names-separated-by-comma/153581 "2018-10-23T10:47:04Z")
Hi experts. I am trying to use the dissect filter to parse the below line: 2018-09-24 13:22:38,562 | tshadm | UserAdmin,Admin,CryptoAdmin | 10.128.13.207 | USER | LOGIN\_ATTEMPT | | Login was successful So far my filt… --- ## [Load sql server data into elasticsearch without query using logstash](https://discuss.elastic.co/t/load-sql-server-data-into-elasticsearch-without-query-using-logstash/153557)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 4\ **Last updated:** [October 23, 2018, 10:23am UTC](https://discuss.elastic.co/t/load-sql-server-data-into-elasticsearch-without-query-using-logstash/153557 "2018-10-23T10:23:19Z")
Hello, I want to load all the tables along with the data present within the database schema(i.e, QWEQWE), without using query. With query we can load data from sql server from multiple tables into elasticsearch as show… --- ## [How to how can I create a logstash filter to generate log with filename](https://discuss.elastic.co/t/how-to-how-can-i-create-a-logstash-filter-to-generate-log-with-filename/153574)
**Author:** [@naga\_kunchala](https://discuss.elastic.co/u/naga_kunchala)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 10:17am UTC](https://discuss.elastic.co/t/how-to-how-can-i-create-a-logstash-filter-to-generate-log-with-filename/153574 "2018-10-23T10:17:17Z")
i have wrote the logstash configuration file to get the all server application logs using udp input filter. i can able to created individual server log by host filed but i cant able get the logs with file name. could you… --- ## [Load sql server data into elasticsearch without query](https://discuss.elastic.co/t/load-sql-server-data-into-elasticsearch-without-query/153558)
**Author:** [@balumurari1](https://discuss.elastic.co/u/balumurari1)\ **Replies:** 2\ **Last updated:** [October 23, 2018, 10:06am UTC](https://discuss.elastic.co/t/load-sql-server-data-into-elasticsearch-without-query/153558 "2018-10-23T10:06:14Z")
Hello, I want to load all the tables along with the data present within the database schema, without using query. With query we can load data from sql server from multiple tables into elasticsearch as shown in below ex… --- ## [S3 plugin missing data](https://discuss.elastic.co/t/s3-plugin-missing-data/153571)
**Author:** [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 10:00am UTC](https://discuss.elastic.co/t/s3-plugin-missing-data/153571 "2018-10-23T10:00:25Z")
Logstash Version: 6.2.2 S3 Plugin: 3.4.1 Conf: input{ s3 { bucket =\> "xxxxxxx-xxxxxxxl-elb-logs" region =\> "ap-south-1" type =\> "elb" interval =\> 5 } I am facing an issue where the data populated is … --- ## [Adding new fields from grok filter in logstash](https://discuss.elastic.co/t/adding-new-fields-from-grok-filter-in-logstash/153570)
**Author:** [@userASR](https://discuss.elastic.co/u/userASR)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 9:59am UTC](https://discuss.elastic.co/t/adding-new-fields-from-grok-filter-in-logstash/153570 "2018-10-23T09:59:34Z")
I have a grok filter in my logstash conf. Where I match a message and I want to add a new field "newfield" and populate value from one specific field "data1" from the message. Here is how my configuration looks like: i… --- ## [Xml parser with xpath not giving result](https://discuss.elastic.co/t/xml-parser-with-xpath-not-giving-result/153566)
**Author:** [@clementb](https://discuss.elastic.co/u/clementb)\ **Replies:** 1\ **Last updated:** [October 23, 2018, 9:55am UTC](https://discuss.elastic.co/t/xml-parser-with-xpath-not-giving-result/153566 "2018-10-23T09:55:32Z")
Hi, I'm trying to parse xml files with the xpath plugin but it doesn't give me any result, Here are my xml data and logstash conf. \ \ \ \ **Author:** [@kpover](https://discuss.elastic.co/u/kpover)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 9:34am UTC](https://discuss.elastic.co/t/best-way-to-parse-multiline-csv/153559 "2018-10-23T09:34:40Z") Hello, I'm having an issue parsing a csv file that contains multi line data in one of the cells. An example of the data: (sorry an image seemed the easiest way to illustrate) What would be the correct way to go throu… --- ## [Index Json Data in Search using Logstash where data is a value of some key in original Json](https://discuss.elastic.co/t/index-json-data-in-search-using-logstash-where-data-is-a-value-of-some-key-in-original-json/153532)
**Author:** [@Panda8](https://discuss.elastic.co/u/Panda8)\ **Replies:** 4\ **Last updated:** [October 23, 2018, 9:19am UTC](https://discuss.elastic.co/t/index-json-data-in-search-using-logstash-where-data-is-a-value-of-some-key-in-original-json/153532 "2018-10-23T09:19:32Z")
I have a json structure as below { "Name" : "", "Type" : "", "Document" : { "id" : 1, "category": "books" } } I want to index only the "Document" part of the Json into Elasticsearch. Is there any way of doin… --- ## [Is it possible to use if else conditional within elastic serach output plugin?](https://discuss.elastic.co/t/is-it-possible-to-use-if-else-conditional-within-elastic-serach-output-plugin/153541)
**Author:** [@Panda8](https://discuss.elastic.co/u/Panda8)\ **Replies:** 3\ **Last updated:** [October 23, 2018, 9:19am UTC](https://discuss.elastic.co/t/is-it-possible-to-use-if-else-conditional-within-elastic-serach-output-plugin/153541 "2018-10-23T09:19:22Z")
Hi Can following configuration : output { if(\[type\] == "A"){ elasticsearch{ hosts =\> \["localhost:9200"\] index =\> "aaa" document\_type =\> "\_doc" document\_id =\> "11" http\_compression =\> true manage\_template =\> tru… --- ## [Multiple matches in grok won't work](https://discuss.elastic.co/t/multiple-matches-in-grok-wont-work/153551)
**Author:** [@bab](https://discuss.elastic.co/u/bab)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 9:06am UTC](https://discuss.elastic.co/t/multiple-matches-in-grok-wont-work/153551 "2018-10-23T09:06:19Z")
Hi, my logs : \[9/16/18 19:00:22:154 CEST\] 00001609 SystemOut O 2018-09-16 19:00:22,154 ERROR \[ContentQueueWorker\_ADR\_Worker background task processing6064\] - Failed to delete Centera clip id 840VGECLTS8FAe7GT8CKT6L… --- ## [Logstash should point to Master or Slave?](https://discuss.elastic.co/t/logstash-should-point-to-master-or-slave/153522)
**Author:** [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Replies:** 3\ **Last updated:** [October 23, 2018, 8:32am UTC](https://discuss.elastic.co/t/logstash-should-point-to-master-or-slave/153522 "2018-10-23T08:32:27Z")
ES:6.3.2 Below is my Master Conf bootstrap.memory\_lock: true cluster.name: pm\_production\_elasticsearch discovery.zen.minimum\_master\_nodes: 2 http.port: 9200 network.host: 0.0.0.0 node.data: false node.master: tru… --- ## [Logstash multiple pipelines and event processing](https://discuss.elastic.co/t/logstash-multiple-pipelines-and-event-processing/153537)
**Author:** [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 7:46am UTC](https://discuss.elastic.co/t/logstash-multiple-pipelines-and-event-processing/153537 "2018-10-23T07:46:53Z")
Logstash version: 6.3.2 Hello, in order to process Netflow logs I installed Elastiflow which requires use of a different pipeline; the only changes I made was to comment out path.config in the logstash.yaml file and add… --- ## [Logstash.outputs.elasticsearch : Failed to install template](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-failed-to-install-template/153406)
**Author:** [@Gobinda\_Nandi1](https://discuss.elastic.co/u/Gobinda_Nandi1)\ **Replies:** 2\ **Last updated:** [October 23, 2018, 7:05am UTC](https://discuss.elastic.co/t/logstash-outputs-elasticsearch-failed-to-install-template/153406 "2018-10-23T07:05:37Z")
this is logstash conf file input { file { path =\> "/home/skills180/Es/templetes/test2.txt" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" codec =\> "json" } } filter… --- ## [Is it possible to shorten this Logstash filter cascade?](https://discuss.elastic.co/t/is-it-possible-to-shorten-this-logstash-filter-cascade/153525)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [October 23, 2018, 6:29am UTC](https://discuss.elastic.co/t/is-it-possible-to-shorten-this-logstash-filter-cascade/153525 "2018-10-23T06:29:55Z")
Hi, i am using ELK GA 6.3.0. I read from Kafka and I have the below filter cascade; filter { dissect { mapping =\> { "message" =\> "\<%{timestamp}\> \<%{f2}\> \<%{f3}\> \<%{f4}\> \<%{f5}\> \<%{f6}\> %{rest}" } } grok { … --- ## [How to set sequence to run multiple pipeline](https://discuss.elastic.co/t/how-to-set-sequence-to-run-multiple-pipeline/152048)
**Author:** [@chitreshg](https://discuss.elastic.co/u/chitreshg)\ **Replies:** 2\ **Last updated:** [October 23, 2018, 6:08am UTC](https://discuss.elastic.co/t/how-to-set-sequence-to-run-multiple-pipeline/152048 "2018-10-23T06:08:53Z")
Hi all, I'm working on multiple pipeline, I've created 3 different pipeline on single instance. but on running logstash i'm not getting the output in the sequece. is there any way to get output according mentioned seq… --- ## [Filter or remove some logs in logstash](https://discuss.elastic.co/t/filter-or-remove-some-logs-in-logstash/153519)
**Author:** [@prem\_kumar](https://discuss.elastic.co/u/prem_kumar)\ **Replies:** 4\ **Last updated:** [October 23, 2018, 5:59am UTC](https://discuss.elastic.co/t/filter-or-remove-some-logs-in-logstash/153519 "2018-10-23T05:59:34Z")
I have Filebeat -\> logstash -\> Elasticsearch setup and everything is working fine. I have filter condition in logstash so that I can parse. In my log file it will have some additional logs lines like +++++++++++++++++… --- ## [Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction](https://discuss.elastic.co/t/failed-to-execute-action-id-main-action-type-logstash-failedaction/153521)
**Author:** [@groverjatin17](https://discuss.elastic.co/u/groverjatin17)\ **Replies:** 1\ **Last updated:** [October 23, 2018, 5:41am UTC](https://discuss.elastic.co/t/failed-to-execute-action-id-main-action-type-logstash-failedaction/153521 "2018-10-23T05:41:03Z")
\[2018-10-23T05:27:51,602\]\[ERROR\]\[logstash.pipeline \] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"\\\"jdbc:oracle:thin:@host:port/servicename\\", jdbc\_user… --- ## [Logstash whitelist nestet json objects, filter only specific fields](https://discuss.elastic.co/t/logstash-whitelist-nestet-json-objects-filter-only-specific-fields/151216)
**Author:** [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\ **Replies:** 1\ **Last updated:** [October 23, 2018, 4:37am UTC](https://discuss.elastic.co/t/logstash-whitelist-nestet-json-objects-filter-only-specific-fields/151216 "2018-10-23T04:37:42Z")
I cant filter out nestet json objects I tried to implement network packets analysis. The result has too-many fields and I want to filter out only messages having content "http\_http\_file\_data" and import only specific f… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=661) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=663)