I am using the ArcSight module to receive events from my SmartConnectors. However I have an ArcSight Logger filled with events and I would like to use the Forwarding capabilities in ArcSight Logger to forward events to Elasticsearch as well. Has anyone done this? I think it makes perfect sense to do this.
I tested to create a fordwarding rule in ArcSight Logger to send to the same port in Elasticsearch where I receive events from the SmartConnectors, however that did not work.
I wrote a big part of the ArcSight module.
In reading up about ArcSight Logger, I see that its a platform similar to the Elastic Stack solution.
Question: Do you mean how to you move the CEF events you already have into the new solution?
Yes. How to move events I already have in ArcSight Logger into Elasticsearch.
We did not build in support for this.
If you are keen, we can work together to develop a sister module for this.
From my brief reading of the ArcSight Logger docs, you will need to export the CEF data as CSV files or maybe LS can access the Logger API over http?
/cc @alvin
Instead of sending from the ArcSight Logger directly to Elasticsearch; send it to your logstash instance where the ArcSight module is running.
So simply put Logger Forwarder --> Logstash instance with ArcSight module configured for SmartConnector (port 5001 default)
That way Logstash should take care of events delivered to Elasticsearch.
Hi Nic,
I may not have been clear, but that is what I already have tried. Have you tested this successfully? I am not sure if Arcsight Logger is sending CEF-format logs when forwarding with TCP Forwarder.
Hey Victor,
The logger forwarder should forward CEF events in order to work with the logstash module. Therefore, on your Logger Forwarder; configure your Filter to only forward CEF events (i.e. deviceVendor IS NOT NULL or cefVersion IS NOT NULL).
If you'd like to test it; you can theoretically set up a logstash TCP listener and see if indeed the events forwarded from your logger is CEF or otherwise.
My guess is you have a few receivers in your logger which might co-mingle some CEF messages with RAW syslog messages but do test the above out.
Hope that helps, ~Nic
Hi Nic,
All our events are stored in CEF format in ArcSight Logger. Perhaps it would be best to setup a tcp listener and see the event format that is used in the Logger Forwarder. Can you help me with that?
Regards Victor
@undelete
You can try this for starters. Edit the port setting. I don't know what the format of the data coming into the tcp port will be. Maybe you can post one event from the console output here?
input {
tcp {
port => 5005
# codec => "json"
type => "cef_forwarded"
}
}
filter {}
output {
stdout { codec => rubydebug }
}
Hi, here is some sample events from that ouput (I noticed that some tokens are removed when pasted here):
{
"@timestamp" => 2017-11-17T09:44:54.851Z,
"port" => 45199,
"@version" => "1",
"host" => "lxserver.mynd.se",
"@metdata" => {
"ip_address" => "172.10.10.1"
},
"message" => "CEF:1|Mynd|SYSTEM1|1|Search|Search|Unknown| eventId=114711 msg=<operation type\\=\"client-request\"> <description><IdentifierID>PI14-000013</IdentifierID><Sender>mynd</Sender><Receiver>BE/PRUEM</Receiver><Date>20171117104451</Date><Message>Prüm-sökning skickad BE</Message></description> </operation> categorySignificance=/Normal categoryBehavior=/Communicate/Query categoryDeviceGroup=/Application categoryOutcome=/Success categoryObject=/Host/Application/Service art=1510911892607 rt=1510911891000 suid=bis cs2=PR-000404-17 cs3=client-request flexNumber2=59967674 cs1Label=Objekttyp cs2Label=ObjektID cs3Label=Objektidtyp cs4Label=Kontext cs5Label=Logmappsid cs6Label=Funktion ahost=lxserver.mynd.se agt=172.10.10.1 agentZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 172.10.0.0-172.31.255.255 amac=l00-22-22-D1-03-3B av=7.6.0.8009.0 atz=Europe/Stockholm at=flexmulti_db dtz=Europe/Stockholm deviceProcessName=Databasspecifikation _cefVer=1.0 aid=3BR-bgV8BABCA57tDn6di1g\\=\\=\r",
"type" => "cef_forwarded"
}
{
"@timestamp" => 2017-11-17T09:45:03.317Z,
"port" => 45180,
"@version" => "1",
"host" => "lxserver.mynd.se",
"@metdata" => {
"ip_address" => "172.10.10.1"
},
"message" => "CEF:1|Mynd|SYSTEM2|2.2.1-SNAPSHOT|Heartbeat1|Heartbeat|Unknown| eventId=98273 msg=timeOfCall\\=2017-11-17T10:45:00.014+0100;schedulerName\\=se.mynd.mynd.heartbeat.HeartBeatBean;fileMetadata\\=rw-r--r--;fileNoLinks\\=;fileOwner\\=l;fileGroup\\=l;fileSizeInBytes\\=0 bytes;fileLastModified\\=2017-11-17T08:55:00Z;fileAbsolutePath\\=/tmp/SYSTEM2_2.2.1-SNAPSHOT.cache;partitionFilesystem\\=;partitionSizeInBytes\\=1056858112;partitionUsedInBytes\\=35098624;partitionAvailableInBytes\\=968073216;partitionUsedInPercentage\\=3;partitionMountedOn\\=;processPID\\=16519;processUser\\=;processGroup\\=;processCPUInPercent\\=0.62;processMemInPercent\\=30.0;processNoKernelThreads\\=116;processCumulativeCPUTime\\=[00-]00:50:04;noCachedMsg\\=0; art=1510911900413 act=FV2 rt=1510911900017 suid=SYSTEM spriv=1 dhost=lxserver2.mynd.se dst=172.10.11.188 destinationZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 172.10.0.0-172.31.255.255 dmac=00-50-56-B3-2B-0C destinationServiceName=SYSTEM2 filePermission=1 cs1=HeartbeatMessage cs3=N/A cs5=bbdda3b7-35d7-49e9-b78e-737171b872a2 flexNumber2=55 deviceCustomDate1=1510911900026 cs1Label=Objekttyp cs2Label=ObjektID cs3Label=Objektidtyp cs4Label=Info cs5Label=LogMapID cs6Label=Reserv5 c6a2Label=SourceIPv6Adress c6a3Label=DestinationIPv6Adress flexNumber2Label=LopNummer ahost=lxserver.mynd.se agt=172.10.10.1 agentZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 172.10.0.0-172.31.255.255 amac=l00-22-22-D1-03-3B av=7.6.0.8009.0 atz=Europe/Stockholm at=sdkmultifolderreader dvchost=localhost deviceNtDomain=F dtz=Europe/Stockholm deviceProcessName=SLK _cefVer=1.0 aid=3ymLTgV8BABDOiu5p6qO+yQ\\=\\=\r",
"type" => "cef_forwarded"
}
{
"@timestamp" => 2017-11-17T09:44:06.983Z,
"port" => 45199,
"@version" => "1",
"host" => "lxserver.mynd.se",
"@metdata" => {
"ip_address" => "172.10.10.1"
},
"message" => "CEF:1|ArcSight|ArcSight|7.6.0.8009.0|agent:050|Connector Raw Event Statistics|Low| eventId=114639 mrt=1510911764927 categorySignificance=/Informational categoryBehavior=/Execute/Response categoryDeviceGroup=/Application catdt=Security Management categoryOutcome=/Success categoryObject=/Host/Application art=1510911764932 cat=/Agent/RawEvent/Statistics deviceSeverity=Warning rt=1510911764927 fileType=Agent cs1=0.0 cs2=0.0 cs3=0.0 cs4=0 cs5=0.0 cs6=3gePWgV8BABDnRWY2rLFGDA\\=\\= cn1=0 cn2=0 cn3=0 deviceCustomDate1=1510911464927 cs1Label=Event throughput cs2Label=Raw event character throughput cs3Label=Event throughput (SLC) cs4Label=Raw event length (SLC) cs5Label=Raw event character throughput (SLC) cs6Label=Destination ID cn1Label=Total event count cn2Label=Total raw event length cn3Label=Event count (SLC) deviceCustomDate1Label=Last time ahost=lxserver.mynd.se agt=172.10.10.1 agentZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 172.10.0.0-172.31.255.255 amac=l00-22-22-D1-03-3B av=7.6.0.8009.0 atz=Europe/Stockholm at=sdkiddatabase dvchost=lxserver.mynd.se dvc=172.10.10.1 deviceZoneURI=/All Zones/ArcSight System/Private Address Space Zones/RFC1918: 172.10.0.0-172.31.255.255 dvcmac=l00-22-22-D1-03-3B dtz=Europe/Stockholm _cefVer=1.0 aid=3gePWgV8BABDnRWY2rLFGDA\\=\\=\r",
"type" => "cef_forwarded"
}
Ok so the message field looks like a standard CEF text string.
Now try adding the cef codec to the tcp input
input {
tcp {
port => 5005
codec => cef { delimiter => "\r" }
type => "cef_forwarded"
}
}
filter {}
output {
stdout { codec => rubydebug }
}
Again paste at least one event from console output.
When you paste the event in here use two triple backticks to "fence" in the format.
e.g.
```
pasted
...
event
...
text
```
I have tried this multiple times now but there was no console output at all.
Try turning debug logging on and report paste some of the log here - look for tcp lines that show receiving a packet and any cef codec lines that appear in the logs.
@undelete - would you mind sharing your logstash config with us? The messages that you've attached in your earlier posts were indeed CEF and should parsed.
Hi, I don't know how to turn debug logging on and I have not had the time to investigate how to do that 
.
The only thing I have configured in logstash.yml is this, otherwise default:
xpack.monitoring.elasticsearch.url: ["http://localhost:9200"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xxx
path.logs: /xxx/logs/logstash
path.data: /xxx/data/logstash
@undelete; I was referring to your config file which contains your input, filter and output blob.
As to running it on debug; here's a link to it - Running Logstash from the command line
I'm sorry, I'm very new to Elastic and I am not sure what you mean. I just start logstash with the ArcSight Module (https://www.elastic.co/guide/en/logstash/current/arcsight-module.html) and collect my events that way. I don't think I had to configure any input output filters etc. so I'm not sure what you are asking.
Hey @undelete - yea you're right in a sense that you won't need to configure anything but seeing the contents from your logstash.yml it doesn't seem like you have the modules configured to run; unless you're keeping logstash running from the command line?
Can you share how you're running the logstash module? Is it via a command line?
Yes I run it via command line. Like this:
bin/logstash --modules arcsight --path.settings /etc/logstash/ -M "arcsight.var.inputs=smartconnector" -M "arcsight.var.input.smartconnector.port=5000" -M "arcsight.var.elasticsearch.username=elastic" -M "arcsight.var.elasticsearch.password=ff -M "arcsight.var.kibana.host:5601=elastic01.server.com" -M "arcsight.var.kibana.username=elastic" -M "arcsight.var.kibana.password=ff &
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.