# Creating Multiple Pipelines Arcsight, Netflow, filebeat, packet beat etc **URL:** \ **Category:** Logstash\ **Created:** [April 3, 2019, 5:49am UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100 "2019-04-03T05:49:36Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![vidurmittal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vidurmittal/32/42843_2.png) [@vidurmittal](https://discuss.elastic.co/u/vidurmittal)\ **Post date:** [April 3, 2019, 5:49am UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/1 "2019-04-03T05:49:36Z")
Hi, I am trying to install module Arcsight, netflow etc with currently running services like filebeat, packetbeat, suricata, wazuh but when I am installing Arcsight or netflow module, I am not to insert indexes, dashboard etc in elasticsearch and kibana getting below error. Any help or guidance will be much appreciated. ELK Stack Version is **"6.6.2"** [2019-04-02T20:28:17,170][ERROR][logstash.modules.kibanaclient] Error when execu ting Kibana client request {:error=\>#\} [2019-04-02T20:28:20,300][ERROR][logstash.modules.kibanaclient] Error when execu ting Kibana client request {:error=\>#\} [2019-04-02T20:28:20,731][ERROR][logstash.config.sourceloader] Could not fetch a ll the sources {:exception=\>LogStash::ConfigLoadingError, :message=\>"Failed to i mport module configurations to Elasticsearch and/or Kibana. Module: arcsight has Elasticsearch hosts: ["localhost:9200"] and Kibana hosts: ["localhost:5601"]", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/config/modules\_ common.rb:108:in `block in pipeline_configs'", "org/jruby/RubyArray.java:1734:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/config/modules\_common. rb:54:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/co nfig/source/modules.rb:14:in`pipeline\_configs'", "/usr/share/logstash/logstash- core/lib/logstash/config/source\_loader.rb:61:in `block in fetch'", "org/jruby/Ru byArray.java:2481:in`collect'", "/usr/share/logstash/logstash-core/lib/logstash /config/source\_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/ logstash/agent.rb:150:in`converge\_state\_and\_update'", "/usr/share/logstash/logs tash-core/lib/logstash/agent.rb:101:in `execute'", "/usr/share/logstash/logstash -core/lib/logstash/runner.rb:362:in`block in execute'", "/usr/share/logstash/ve ndor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initia lize'"]} [2019-04-02T20:28:20,753][ERROR][logstash.agent] An exception happene d when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Failed to import module configurations to Elas ticsearch and/or Kibana. Module: arcsight has Elasticsearch hosts: [\"localhost: 9200\"] and Kibana hosts: [\"localhost:5601\"]", :backtrace=>["/usr/share/logsta sh/logstash-core/lib/logstash/agent.rb:157:in`converge\_state\_and\_update'", "/us r/share/logstash/logstash-core/lib/logstash/agent.rb:101:in `execute'", "/usr/sh are/logstash/logstash-core/lib/logstash/runner.rb:362:in`block in execute'", "/ usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:2 4:in `block in initialize'"]} ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 3, 2019, 4:00pm UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/2 "2019-04-03T16:00:01Z")
Those are SSL errors, are you using SSL at all? What are your inputs and outputs that you are using? ---
**Author:** ![vidurmittal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vidurmittal/32/42843_2.png) [@vidurmittal](https://discuss.elastic.co/u/vidurmittal)\ **Post date:** [April 3, 2019, 4:44pm UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/3 "2019-04-03T16:44:53Z")
Above 2 error are of ssl then error for import of module configuration. Well I am not using ssl, I will add parameters for disabling ssl. I want to run netflow, arcsight and other beats in single instance of logstash. Please help in suggesting the way out and sample logstash configuration file ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 3, 2019, 5:00pm UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/4 "2019-04-03T17:00:14Z")
My configuration for multiple pipelines follows the pipeline to pipeline configuration: [https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html) In my filebeat.yml I add a field to the log itself ```auto - type: log enabled: true paths: - /root/ForumHelp.log fields: forum: true ``` My logstash pipelines.yml calls a routing pipeline that I have setup: ```auto - pipeline.id: route5044 path.config: "/etc/logstash/conf.d/route5044/*.conf" - pipeline.id: route5045 path.config: "/etc/logstash/conf.d/route5045/*.conf" - pipeline.id: dmesg path.config: "/etc/logstash/conf.d/dmesg/*.conf" - pipeline.id: postgres path.config: "/etc/logstash/conf.d/postgres/*.conf" - pipeline.id: forum_help path.config: "/etc/logstash/conf.d/ForumHelp/*.conf" - pipeline.id: meraki path.config: "/etc/logstash/conf.d/meraki/*.conf" - pipeline.id: catchall path.config: "/etc/logstash/conf.d/catchall/*.conf" ``` My input for my route5044 just opens up ports and determines types: ```auto input { tcp { port => 5146 codec => "json_lines" ssl_enable => true ssl_verify => false ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_cert => "/etc/logstash/ssl/logstash-proxy.crt" ssl_extra_chain_certs => "/etc/logstash/ssl/ca.me.com.crt" } } ``` I will be adding logic to this section as well as I am doing some consolidations right now. Then the output of my route5044 handles the routing to the correct pipeline. ```auto output { if "postgres" in [fields] { pipeline { send_to => postgres_log } } else if "meraki" in [fields] { pipeline { send_to => meraki_syslog } } else if "forum" in [fields] { pipeline { send_to => forum_help } } else { pipeline { send_to => catch_all } } } ``` Then finally the input for my forum\_help looks like this: ```auto input { pipeline { address => forum_help } } ``` Quick run down: On beats I create a virtual pipeline address by creating a field Route pipeline accepts input Route pipeline output checks what the virtual pipeline address is and routes it to the correct pipeline. ---
**Author:** ![vidurmittal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vidurmittal/32/42843_2.png) [@vidurmittal](https://discuss.elastic.co/u/vidurmittal)\ **Post date:** [April 4, 2019, 5:59am UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/5 "2019-04-04T05:59:18Z")
> [@vidurmittal](#): > > help in suggesting the w Hi Ken, Thanks for kind and prompt response. Going through the configuration helped in getting good understanding over the configuration. It will be great if i can get the sample conf file for arcsight and netflow too. Thanks ---
**Author:** ![kharvey](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@kharvey](https://discuss.elastic.co/u/kharvey)\ **Post date:** [April 4, 2019, 3:09pm UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/6 "2019-04-04T15:09:56Z")
I'm afraid that I do not have a setup that I can test Netflow or Arcsight with. The only thing you would have to do is assign a field to your Netflow or Arcsight and it will work. If you can't add the field through those, then you can move those services to a different port. For example, I am pulling syslog info from Meraki, but Meraki doesn't support me adding custom fields. So I moved Meraki over to a different port. Here is my config for that: ```auto input { beats { id => "proxy-5044-in" port => 5044 ssl => true ssl_certificate_authorities => ["/etc/logstash/ssl/ca.me.com.crt"] ssl_certificate => "/etc/logstash/ssl/logstash-proxy.crt" ssl_key => "/etc/logstash/ssl/logstash-proxy-pkcs8.key" ssl_verify_mode => "force_peer" add_field => { "_forwarder" => "fwd-5044" "origin_host" => "%{host}" } } } input { udp { id => "proxy-5045-in" port => 5045 type => "syslog" add_field => { "[fields][meraki]" => true "_forwarder" => "fwd-5045" } } } ``` This is my logstash-proxy config, but you should be able to change it to add it to your regular pipeline. I have all of my Meraki stuff connect to port 5045, and then on the input I add the [fields][meraki] which then my regular pipeline logic takes over. So moving services that you are unable to add a field, to a different port will allow you to uniquely identify the logs and add in your own fields. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2019, 3:10pm UTC](https://discuss.elastic.co/t/creating-multiple-pipelines-arcsight-netflow-filebeat-packet-beat-etc/175100/7 "2019-05-02T15:10:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.