2 conf sending data to the same index

jefin_dark · May 19, 2023, 8:40pm

Hello,

I have 2 conf files and they are sending data at the same time to the 2 index (when I would like each conf to send the information to the specific index)

If you can help me, I can provide more information if needed.

Below is my configuration.

Firts.

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => {
      "[event_data][Data]" => [
        "Subject:\s*Security ID:\s*%{DATA:security_id}",
        "Subject:\s*Account Name:\s*%{DATA:account_name}",
        "Subject:\s*Account Domain:\s*%{DATA:account_domain}",
        "Subject:\s*Logon ID:\s*%{DATA:logon_id}",
        "Object:\s*Object Server:\s*%{DATA:object_server}",
        "Object:\s*Object Type:\s*%{DATA:object_type}",
        "Object:\s*Object Name:\s*%{DATA:object_name}",
        "Object:\s*Handle ID:\s*%{DATA:handle_id}",
        "Access:\s*Accesses:\s*%{DATA:accesses}",
        "Access:\s*Access Mask:\s*%{DATA:access_mask}",
        "Access:\s*Privileges:\s*%{DATA:privileges}"
      ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://192.168.12.109:9200"]
    index => "srvvmfs01_log-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "XXXXX"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }
}

Second

input {
  udp {
    port => 5514
    codec => plain
  }
}

filter {
  grok {
    match => {
      "message" => "<%{POSINT:priority}>%{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{DATA:hostname} %{WORD:event_type} %{GREEDYDATA:message_data}"
    }
  }

  mutate {
    convert => { "priority" => "integer" }
  }

  if [event_type] == "user" {
    grok {
      match => {
        "message_data" => "admin logged in from %{IP:source_ip} via %{WORD:login_method}"
      }
    }
  }

  if [event_type] == "filter" {
    grok {
      match => {
        "message_data" => "rule %{WORD:rule_action} by %{DATA:rule_modifier}"
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://192.168.12.109:9200"]
    index => "mikrotik_log-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "XXXXXXXX"
    ssl => true
    cacert => "/etc/logstash/http_ca.crt"
  }
}

pipeline.yml (locate on /etc/logstash/)

- pipeline.id: mikrotik
  path.config: "/etc/logstash/conf.d/mikrotik-log.conf"

- pipeline.id: srvvmfs01
  path.config: "/etc/logstash/conf.d/srvvmfs01-log.cof"

logstash conf service

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
#EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.config /etc/logstash/conf.d/*.conf
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

when some information is sent , by any of the conf, it duplicates in the 2 index

stephenb · May 19, 2023, 10:55pm

@jefin_dark Thanks for all the detail

Your pipelines.yml looks correct but

Curious did you create / edit the service file yourself or is that the default?

Seems to me the line above is the problem... that will concatonate the 2 files together which would explain what you are seeing ... Or your pipelines.yml not being read and therefore the files and the directory are being concatenated... Either way, pretty sure that's the issue

Seems like that should be

--path.settings /etc/logstash

I will need to triple check, but I would test with that first.

Yup I checked mine

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

If you intentionally want to concatenate them together which I don't think you want to, but if you do we can just put in logic so that the input will get a tag and then the output will be conditional on that tag

jefin_dark · May 20, 2023, 12:55am

@stephenb , thanks again for your attention

yes, i made intentionaly change, because this way, charge every file on conf.d, but i dont know that is correct.

when i put the service conf like you say

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

only one conf is up, but works correctly, just only one index has created and process

How can i made these TAGS for work correct.
1 Index for 1 Conf.

Thank you in advance
Regards.

stephenb · May 20, 2023, 1:00am

Hi @jefin_dark

Let's take a moment and debug why you don't now have 2 independent pipelines... because Your instincts are correct ...

Ahhh found it... typo... in pipelines.yml

- pipeline.id: srvvmfs01
  path.config: "/etc/logstash/conf.d/srvvmfs01-log.cof"

Missing the n should be

- pipeline.id: srvvmfs01
  path.config: "/etc/logstash/conf.d/srvvmfs01-log.conf"
.....................................................^

This is why "Pair Programming" is good!!

jefin_dark · May 20, 2023, 1:14am

it was a typo on my part. I'm sorry

On the pipeline.yml is correct

stephenb · May 20, 2023, 1:29am

Hi @jefin_dark

Apologies I am not clear what you are saying...

a) Your pipeline.yml has the correct path and it is still not working?

b) Or now it is fixed and it is working?

if a) That would indicate that If the path is correct and it is still not working there is a different issue... like logstash can not find / access the .conf file or there is an error in the conf file. You need to show the logstash startup logs...

I would comment out the mikrotik pipeline in the pipeline.yml so we can focus on the other one...

Things to check check list the actual file ... is is there... are the permissions correct?

ls -l /etc/logstash/conf.d/srvvmfs01-log.conf

You can also just start logstash in the foreground with the following command and watch the logs....

/usr/share/logstash/bin/logstash --path.settings /etc/logstash --path.config /etc/logstash

There is something simple at this point... most likely has to do with the pipelines.yml or the path... because when you used *.conf 2 pipelines ran which indicates there is not an error in the actual .conf file.

Minor I don't think you need this on the beats

    codec => plain

Do you have a line feed? You know blank line after the last line in the pipeline.yml

jefin_dark · May 20, 2023, 10:30pm

Hi @stephenb , sorry for my english.
I'm a Brazilian who speaks very bad English, so sometimes I express myself wrong

A) My pipeline.yml file was not ok.

B) I made the correction, as you pointed out and now everything is working perfectly.

The first was that I changed the logstash service file incorrectly.

The second was from so much editing the config pipeline.yml I ended up forgetting the N

Now everything is working as it should.

Follow the prints.

Thank you immensely for your help.

I'm sorry for the beginner's mistakes, but it's really a new environment for me.

I hope I can evolve here in my analysis.

A good rest.

Regards

stephenb · May 20, 2023, 10:41pm

@jefin_dark

You speak/ write great English...

My Portuguese is "inexistente"

Glad you got is solved

system · June 17, 2023, 10:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Multiple Logstash conf files with the same Index Logstash	1	1015	July 25, 2017
Keep one logstash conf file rather having multiple logstash conf file! Logstash	5	329	March 7, 2019
Separate configs, different indexes yet both logs go to both indexes Logstash	2	536	October 30, 2017
Logstash/ElasticSearch mixing data across indexes Logstash	3	449	May 27, 2018
ELK Multi index in signle logstash.conf file Logstash	5	560	August 24, 2020

2 conf sending data to the same index

Related topics