# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1126 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1127 --- ## [Force logstash to reindex csv files](https://discuss.elastic.co/t/force-logstash-to-reindex-csv-files/55702)
**Author:** [@pittoch](https://discuss.elastic.co/u/pittoch)\ **Replies:** 3\ **Last updated:** [July 19, 2016, 9:40am UTC](https://discuss.elastic.co/t/force-logstash-to-reindex-csv-files/55702 "2016-07-19T09:40:23Z")
Hello, Sometimes I need to force the re-indexing of files csv from the beginning. I try to delete the sincedb file and restart logstash, but re-indexing never restart from the beginning. What is the solution to reindex … --- ## [Event Correlation \\ populate one field with data from other field in elasticsearch](https://discuss.elastic.co/t/event-correlation-populate-one-field-with-data-from-other-field-in-elasticsearch/55372)
**Author:** [@shaigb](https://discuss.elastic.co/u/shaigb)\ **Replies:** 17\ **Last updated:** [July 19, 2016, 8:11am UTC](https://discuss.elastic.co/t/event-correlation-populate-one-field-with-data-from-other-field-in-elasticsearch/55372 "2016-07-19T08:11:02Z")
Hi, i'm looking to do correlation searches in elastic (kibana 4.4 actually) is there a way to have a search like the following - 1 - find all ip addresses from a certain type (lets say from critical stack intel, via file… --- ## [Shield does not work - no credentials checked](https://discuss.elastic.co/t/shield-does-not-work-no-credentials-checked/36959)
**Author:** [@achechen](https://discuss.elastic.co/u/achechen)\ **Replies:** 18\ **Last updated:** [July 19, 2016, 5:23am UTC](https://discuss.elastic.co/t/shield-does-not-work-no-credentials-checked/36959 "2016-07-19T05:23:36Z")
I've just installed shield plugin (2.0) according to the documentation: -stopped all nodes -installed shield on all nodes -started all nodes -added an admin user using esusers I also verified it as per documentation: … --- ## ["\_grokparsefailure" field in Kibana](https://discuss.elastic.co/t/-grokparsefailure-field-in-kibana/55813)
**Author:** [@SamT](https://discuss.elastic.co/u/SamT)\ **Replies:** 1\ **Last updated:** [July 19, 2016, 12:26am UTC](https://discuss.elastic.co/t/-grokparsefailure-field-in-kibana/55813 "2016-07-19T00:26:50Z")
Hi, We see "\_grokparsefailure" field in Kibana for one of the field types, but there are no error messages in the "logstash". This is for each event. Any idea how to debug and resolve the issue? Thanks --- ## [Logstash - Elastisearch template](https://discuss.elastic.co/t/logstash-elastisearch-template/55774)
**Author:** [@Diego\_Gomez](https://discuss.elastic.co/u/Diego_Gomez)\ **Replies:** 4\ **Last updated:** [July 18, 2016, 4:07pm UTC](https://discuss.elastic.co/t/logstash-elastisearch-template/55774 "2016-07-18T16:07:04Z")
Hi, I'm working with twitter and want to map coordinates for the reconnoitre Kibana . This is my template.json { "template": "twitter", "order": 1, "settings": { "number\_of\_shards": 1 }, "mappings": { "tweet": { … --- ## [Logstash download individual plugins (solved)](https://discuss.elastic.co/t/logstash-download-individual-plugins-solved/55787)
**Author:** [@eperry](https://discuss.elastic.co/u/eperry)\ **Replies:** 1\ **Last updated:** [July 18, 2016, 6:09pm UTC](https://discuss.elastic.co/t/logstash-download-individual-plugins-solved/55787 "2016-07-18T18:09:28Z")
Hey guys, Quick question, I was hoping there was (even if it hacky ) way to download individual logstash plugins. I know I can do a logstash install logstash-codex-blah, then do a logstash-plugin pack. Then Distr… --- ## [Parse Amazon S3 access log with multiple files](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 7\ **Last updated:** [July 18, 2016, 5:29pm UTC](https://discuss.elastic.co/t/parse-amazon-s3-access-log-with-multiple-files/55700 "2016-07-18T17:29:35Z")
Hello All, I am trying to parse Amazon S3 access log that has multiple chunks of the file. When i say multiple chunks i mean there are thousand of separate files with each files containing data in it. The data in these… --- ## [Parse custom Apache2 log](https://discuss.elastic.co/t/parse-custom-apache2-log/55736)
**Author:** [@massinissa](https://discuss.elastic.co/u/massinissa)\ **Replies:** 4\ **Last updated:** [July 18, 2016, 4:20pm UTC](https://discuss.elastic.co/t/parse-custom-apache2-log/55736 "2016-07-18T16:20:20Z")
Hello, i have an Apache log like this: 443 84.14.49.234 - - \[04/Jul/2016:10:11:32 +0200\] "GET /ws/1/kpi HTTP/1.1" 200 63 "-" "Mozilla/5.0 (iPhone; CPU \*\*iPhone OS 8\_1\_3\*\* like Mac OS X) AppleWebKit/600.1.4 (KHTML, like… --- ## [Custom Log Level Log4J](https://discuss.elastic.co/t/custom-log-level-log4j/55733)
**Author:** [@pat123](https://discuss.elastic.co/u/pat123)\ **Replies:** 3\ **Last updated:** [July 18, 2016, 3:45pm UTC](https://discuss.elastic.co/t/custom-log-level-log4j/55733 "2016-07-18T15:45:12Z")
Hello, I would like to have a custom log level for some tests in my code. I using log4j in a java-project. The needed level is created and works in my local console output, but I can not show the log in Kibana. I think … --- ## [Loop through fields using ruby resource?](https://discuss.elastic.co/t/loop-through-fields-using-ruby-resource/55557)
**Author:** [@Maxwell\_Flanders](https://discuss.elastic.co/u/Maxwell_Flanders)\ **Replies:** 3\ **Last updated:** [July 18, 2016, 9:51am UTC](https://discuss.elastic.co/t/loop-through-fields-using-ruby-resource/55557 "2016-07-18T09:51:02Z")
Hi there! I want to apply a gsub to ALL fields in my logstash object arbitrarily. I know logstash has no tool for loops like that, but I thought I could perhaps do it in a ruby resource?? Something like: code =\> " ev… --- ## [Loop in Logstash with ruby code](https://discuss.elastic.co/t/loop-in-logstash-with-ruby-code/55404)
**Author:** [@Jeff\_Contour](https://discuss.elastic.co/u/Jeff_Contour)\ **Replies:** 4\ **Last updated:** [July 18, 2016, 9:42am UTC](https://discuss.elastic.co/t/loop-in-logstash-with-ruby-code/55404 "2016-07-18T09:42:35Z")
Hi there I hope someone can give me some advice for this. I have a portion of ruby code that makes Logstash looping (CPU = 100 %, no more indexing in Elasticsearch). I try to index about 400 000 events and I see the in… --- ## [Follows twitter input](https://discuss.elastic.co/t/follows-twitter-input/54837)
**Author:** [@josefboukersi](https://discuss.elastic.co/u/josefboukersi)\ **Replies:** 9\ **Last updated:** [July 18, 2016, 7:48am UTC](https://discuss.elastic.co/t/follows-twitter-input/54837 "2016-07-18T07:48:23Z")
hey, i try to give a follows with logstash but i have this error Pipeline aborted due to error {:exception=\>#\ **Author:** [@gfafei](https://discuss.elastic.co/u/gfafei)\ **Replies:** 1\ **Last updated:** [July 18, 2016, 7:23am UTC](https://discuss.elastic.co/t/how-can-i-extract-a-value-from-a-field-as-a-new-field/55712 "2016-07-18T07:23:29Z") As the title.For example, extract a property from a url. --- ## [Failed to query elasticsearch for previous event {:query=\>"\_index:s1index AND type:a AND A:1 ",....:error=\>#\, :level=\>:warn}](https://discuss.elastic.co/t/failed-to-query-elasticsearch-for-previous-event-query--index-s1index-and-type-a-and-a-1-error-nomethoderror-undefined-method-start-with-for-nil-nilclass-level-warn/47254)
**Author:** [@rupal](https://discuss.elastic.co/u/rupal)\ **Replies:** 8\ **Last updated:** [July 18, 2016, 6:21am UTC](https://discuss.elastic.co/t/failed-to-query-elasticsearch-for-previous-event-query--index-s1index-and-type-a-and-a-1-error-nomethoderror-undefined-method-start-with-for-nil-nilclass-level-warn/47254 "2016-07-18T06:21:47Z")
Hi I have two csv files , s1.csv with following data, 1,2,3 1,2,3 2,3,4 And s2.csv with following data, 1,4 2,5 Consider column name as specified in configuration files like for s1 - A,B,C and… --- ## [How to troubleshoot logstash and es issues?](https://discuss.elastic.co/t/how-to-troubleshoot-logstash-and-es-issues/55626)
**Author:** [@JoeLi](https://discuss.elastic.co/u/JoeLi)\ **Replies:** 3\ **Last updated:** [July 17, 2016, 11:51pm UTC](https://discuss.elastic.co/t/how-to-troubleshoot-logstash-and-es-issues/55626 "2016-07-17T23:51:00Z")
My issue is that I cannot see the index log message from kibana. Here is version I am using: All of them are installed on the same virtual machine. CentOS 7 Oracle JDK 8 Kibana 4.5.2 Elaticsearch 2.3.4 logstash 2.3.… --- ## [S3 input, process files once](https://discuss.elastic.co/t/s3-input-process-files-once/55695)
**Author:** [@Alexander\_Popov](https://discuss.elastic.co/u/Alexander_Popov)\ **Replies:** 0\ **Last updated:** [July 17, 2016, 2:13pm UTC](https://discuss.elastic.co/t/s3-input-process-files-once/55695 "2016-07-17T14:13:19Z")
Does it possible to to exit from pipeline after processed initial files? tried to set interval =0 no luck --- ## [Any trick to avoid "Beats input: the pipeline is blocked, temporary refusing new connection."?](https://discuss.elastic.co/t/any-trick-to-avoid-beats-input-the-pipeline-is-blocked-temporary-refusing-new-connection/48283)
**Author:** [@mathias](https://discuss.elastic.co/u/mathias)\ **Replies:** 8\ **Last updated:** [July 17, 2016, 1:39pm UTC](https://discuss.elastic.co/t/any-trick-to-avoid-beats-input-the-pipeline-is-blocked-temporary-refusing-new-connection/48283 "2016-07-17T13:39:22Z")
Hi, I see a few traces (in logstash.log) like below and wonder if there is a way to configure them away. The input to logstash from filebeat (not on same machines) should not cause congestion 383KBits/s (pure text stri… --- ## [Jdbc input to close db connection](https://discuss.elastic.co/t/jdbc-input-to-close-db-connection/55689)
**Author:** [@Govind\_Raj](https://discuss.elastic.co/u/Govind_Raj)\ **Replies:** 0\ **Last updated:** [July 17, 2016, 11:32am UTC](https://discuss.elastic.co/t/jdbc-input-to-close-db-connection/55689 "2016-07-17T11:32:38Z")
Hi Team I am using Jdbc as my input in logstash configuration . our configs has a schedule of once in every hour. we want to ensure that that the dbconnections are closed after the read operation is completed. Is ther… --- ## [Logstash-ES communication issue and Kibana not coming up](https://discuss.elastic.co/t/logstash-es-communication-issue-and-kibana-not-coming-up/55613)
**Author:** [@vienodp](https://discuss.elastic.co/u/vienodp)\ **Replies:** 2\ **Last updated:** [July 17, 2016, 7:44am UTC](https://discuss.elastic.co/t/logstash-es-communication-issue-and-kibana-not-coming-up/55613 "2016-07-17T07:44:47Z")
I have 2 nodes in a cluster , 35 and 72 with ELK/Shield installed on it. After enabling the Shield parameters in elasticsearch.yml, Logstash to ES communications not working and Kibana is not starting up. I have enabled… --- ## [Architecture advice(for my needs: several applications, multiple files...)](https://discuss.elastic.co/t/architecture-advice-for-my-needs-several-applications-multiple-files/55606)
**Author:** [@mostolog](https://discuss.elastic.co/u/mostolog)\ **Replies:** 1\ **Last updated:** [July 17, 2016, 7:41am UTC](https://discuss.elastic.co/t/architecture-advice-for-my-needs-several-applications-multiple-files/55606 "2016-07-17T07:41:50Z")
What would you think it will be a good ELK architecture? (needs described below) Hi We have several applications hosted on different URLs (eg: app1.foo,com, app2.foo.com) even some of them sharing domains but on … --- ## [Url for 2.3.2 logstash codec for spool goes to 404](https://discuss.elastic.co/t/url-for-2-3-2-logstash-codec-for-spool-goes-to-404/55678)
**Author:** [@baggy23](https://discuss.elastic.co/u/baggy23)\ **Replies:** 1\ **Last updated:** [July 17, 2016, 6:13am UTC](https://discuss.elastic.co/t/url-for-2-3-2-logstash-codec-for-spool-goes-to-404/55678 "2016-07-17T06:13:14Z")
Does anyone know the git repo for spool ? On the logstash 2.3.2 reference page it links to https://github.com/logstash-plugins/logstash-codec-spool. However, when I click on the link, I get the 404 page and when I back … --- ## [Decorate events with logstash-input-kafka 3.0.x plugin](https://discuss.elastic.co/t/decorate-events-with-logstash-input-kafka-3-0-x-plugin/55567)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 1\ **Last updated:** [July 16, 2016, 7:55am UTC](https://discuss.elastic.co/t/decorate-events-with-logstash-input-kafka-3-0-x-plugin/55567 "2016-07-16T07:55:53Z")
There is this https://github.com/logstash-plugins/logstash-input-kafka/pull/103 But I don't know if this will be available soon. I am using logstash 2.2.2 --- ## [Pulling topic metadata from kafka 0.9](https://discuss.elastic.co/t/pulling-topic-metadata-from-kafka-0-9/55440)
**Author:** [@stecino](https://discuss.elastic.co/u/stecino)\ **Replies:** 2\ **Last updated:** [July 16, 2016, 7:53am UTC](https://discuss.elastic.co/t/pulling-topic-metadata-from-kafka-0-9/55440 "2016-07-16T07:53:49Z")
I am using logstash-input-kafka 3.0.x and logstash-output-kafka 3.0.x with logstash 2.2.2 How can I pull topic metadata from it. I could with zk\_connect with plugin 2.x, but not with the plugin upgrade. --- ## [How do I use an array field as a variable in logstash?](https://discuss.elastic.co/t/how-do-i-use-an-array-field-as-a-variable-in-logstash/55545)
**Author:** [@dmurawsky](https://discuss.elastic.co/u/dmurawsky)\ **Replies:** 7\ **Last updated:** [July 15, 2016, 8:59pm UTC](https://discuss.elastic.co/t/how-do-i-use-an-array-field-as-a-variable-in-logstash/55545 "2016-07-15T20:59:26Z")
I'm building out logstash and would like to build functionality to anonymize fields as specified in the message itself. Given the message below, the field fta is an array of fields to anonymize. I would like to just u… --- ## [Postgresql error](https://discuss.elastic.co/t/postgresql-error/55652)
**Author:** [@ggp](https://discuss.elastic.co/u/ggp)\ **Replies:** 0\ **Last updated:** [July 15, 2016, 8:13pm UTC](https://discuss.elastic.co/t/postgresql-error/55652 "2016-07-15T20:13:02Z")
hello i am trying to load data from postgresql db to ES , below is my configuration on the INPUT input { jdbc { # Postgres jdbc connection string to our database, mydb jdbc\_connection\_string =\> "jdbc:postgresql://xxx… --- ## [Adding GeoIP using logstash](https://discuss.elastic.co/t/adding-geoip-using-logstash/55621)
**Author:** [@vishnubraj](https://discuss.elastic.co/u/vishnubraj)\ **Replies:** 2\ **Last updated:** [July 15, 2016, 8:07pm UTC](https://discuss.elastic.co/t/adding-geoip-using-logstash/55621 "2016-07-15T20:07:52Z")
Hi, I am trying to add geoip detail for my data. I only want the longitude and latitude value. I have the below filter config. filter { if \[label\] == "edge" { geoip { … --- ## [Can we create two GeoIP Filters in one logstash config file?](https://discuss.elastic.co/t/can-we-create-two-geoip-filters-in-one-logstash-config-file/42849)
**Author:** [@vikas\_gopal](https://discuss.elastic.co/u/vikas_gopal)\ **Replies:** 16\ **Last updated:** [July 15, 2016, 4:18pm UTC](https://discuss.elastic.co/t/can-we-create-two-geoip-filters-in-one-logstash-config-file/42849 "2016-07-15T16:18:57Z")
Hi Experts, My requirement is to create 2 maps , one is for Source IP and other is for Destination IP. For Source IP what I have done is I used GeoIP filter as below geoip { source =\> "src"} Now I am trying the sam… --- ## [Elasticsearch filter no longer working in 2.3.1](https://discuss.elastic.co/t/elasticsearch-filter-no-longer-working-in-2-3-1/47908)
**Author:** [@xdzhou](https://discuss.elastic.co/u/xdzhou)\ **Replies:** 5\ **Last updated:** [July 15, 2016, 3:14pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-no-longer-working-in-2-3-1/47908 "2016-07-15T15:14:40Z")
I was using logstash-all-plugins-2.2.0 package before and now I am switching to logstash-all-plugins-2.3.1, the same configuration goes against a ES 2.3.1 works using 2.2.0 version, but not 2.3.1 version. The elasticse… --- ## [How to index .rvt file ( BIM )](https://discuss.elastic.co/t/how-to-index-rvt-file-bim/55623)
**Author:** [@josefboukersi](https://discuss.elastic.co/u/josefboukersi)\ **Replies:** 0\ **Last updated:** [July 15, 2016, 1:52pm UTC](https://discuss.elastic.co/t/how-to-index-rvt-file-bim/55623 "2016-07-15T13:52:18Z")
hello words, how to index a file .rvt ( it's the Autodesk Revit for modelesing a builing information it's a new technologie similary to AutoCAD) thank you for any help --- ## [Ruby code in filter does not executed in sequence](https://discuss.elastic.co/t/ruby-code-in-filter-does-not-executed-in-sequence/55622)
**Author:** [@OlivierD](https://discuss.elastic.co/u/OlivierD)\ **Replies:** 0\ **Last updated:** [July 15, 2016, 1:23pm UTC](https://discuss.elastic.co/t/ruby-code-in-filter-does-not-executed-in-sequence/55622 "2016-07-15T13:23:45Z")
I have an input CSV file that I want to parse with logstash. I want to maintain a variable across events loaded from the file, using a variable in the Ruby code. I have an issue so I write a simple CSV input file and a… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1125) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1127)