Filebeat modules

Elastic Stack Beats
,
1

I have, hopefully, a very simple question:

We have filebeat (zeek module) running on SERVER A. Filebeat takes it to kafka where it is then pulled down by logstash. The zeek.conf file in logstash is completely barren. It only has the input (kafka) and the output (elasticsearch) set- there are NO filters set.
We are told that filebeat automatically populates the field names of the network traffic so it is readable by elastic. We are wondering; however, WHEN and WHERE does filebeat create those fields? Does it make fields such as "source.address" on SERVER A before it ships data off to kafka OR are those fields not created until it hits the ES nodes?

2

That happens when Filebeat reads the data from the input (ie file).

3

OK so that means filebeat is creating the fields on SERVER A before it ships data out to kafka. We are running the following translate filter and the pipeline is still passing data but we are not receiving the new field (we created a new index with an updated mapping).

Filter {
  Translate {
      Field => “source.address”
      Destination => “custom”
      Dictionary => {
          “1.2.3.4” => “unique-hostname”
      }
      Fallback => “unknown_mapping”
  }
}
4

We even tried multiple mappings and none of them worked
Attempt#1}

“Custom” : {
    “dynamic”: “true”,
    “properties”: {
         “Hostname”: {
              “type” : “keyword”
      }
    }
}

Attempt#2)

“custom”: {
    “type”: “keyword”
}

Attempet#3)

“custom”: {
    “properties”: {
         “hostname”: {
              “type”: “keyword”
      }
   }
}
5

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Where is that filter?
What does the entire config look like?

6

Sorry about the improper format

The translate filter is located on the zeek.conf file on logstash- in between the input and output stanzas.
The "translate" filter is the only filter in the entire zeek.conf file.

CONTEXT: the zeek.conf file previously had ZERO filters until we added this "translate" filter. Previously it was strictly "input" and then "output." This works and doesn't produce any problems but now we are looking to manipulate some of the fields.

7

The reason I am attempting to emphasize WHERE/WHEN the fields are being created is because I can create new labels and fields via the filebeat processors easily. Those new labels/fields populate into Kibana with no problems. However, when I attempt to rename those fields via logstash it doesnt work at all. The syntax for the mutate filter is fairly straight forward so I cant seem to wrap my head around why logstash isnt altering the fields

8

Fixed the format for you. Sorry about that again

1 Like
9

Unless you can share your configs we're just playing guess games.

10

So specifically for the zeek module a lot of field translation is done in Filebeat and then the remainder is done in the elasticsearch ingest pipeline. source.address should exist coming into logstash. Can u post ur entire logstash config including the input?

1 Like
11

I dont want (or expect) you guys to solve these issues over this medium. We are more so looking for conceptual knowledge that only Elastic personnel would know so we will be able to improve our own trouble shooting methods. With the general knowledge you can provide we can take that and run with it.

The zeek.conf file we are speaking of has been in place for over 3 years and has worked without issue (it has also NEVER had a filter plug-in added to it).
We have used Filebeat processors to successfully add fields (We have Filebeat-zeek module- running on SERVER A- which is at the very "top" of the data stream) These new fields will successfully populate in Kibana. Giving credence that the filebeat processors are working properly.
The only singular change we have now made is attempting to add Logstash filters to further adjust those fields (the ones that were created by the Filebeat processors)
The attempted manipulation from the Logstash filters are not working.

This would leave two possible reasons why (unless you can give insight for additional reasons)
#1) The syntax is incorrect
1b) We tried a VARIETY of syntax variations and none worked. (it still very well could be a syntax error, but we wanted to see if you guys could give insight on #2

#2) The filebeat processor does not create the fields until further "down" the data pipeline
2b) this is where your Elastic expertise could come into play giving some insight on when the fields are created.

I know you are asking for config files but if the zeek.conf file has worked for over 3 years and the only addition we are making is a logstash filter, why would the inputs and the outputs matter? (again this is conceptual knowledge that you may have that we dont so any information could assist us and help us where we may be ignorant)

Information like what legoguy1000 provided (if true) is exactly what were looking for. Certain fields are created in Filebeat then the remainder in the ingest pipeline. We could then determine which fields fall into either of the categories and go from there.

12

Thanks a lot man I really appreciate that. Do you know any documentation that clarifies what fields are created in Filebeat vs which ones are created later in the ingest pipeline?

13

Change the branch to which ever version of Filebeat you're using, beats/x-pack/filebeat/module/zeek at master · elastic/beats · GitHub. In each fileset there is a config and ingest folder. The config folder will have the Filebeat processors that are run and the ingest folder contains the ingest pipelines that are run after being sent to ES.

1 Like
14

Dude thank you so much. Super appreciate it

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.