Hey @guahos, welcome to the discussion boards!
There are a couple of ways to do this:
I took the second route (ingest pipeline), as it doesn't require you to download or install any additional software.
Step 1: Define an ingest pipeline:
This pipeline has a number of processors. Some of these could technically be combined, but I find it easier to reason about when they're split apart. The processors run in the order shown.
- The first (grok) processor parses out the
backup_name.
- The second (grok) processor parses out the duration string, which we process further later on.
- The third (grok) processor parses out the bytes sent/received
- The fourth (script) processor calculates the duration in seconds, by converting the
backup_duration_minutes and backup_duration_seconds fields into integers and doing the simple math to convert the two into seconds. The value is stored in backup_duration
- The fifth/sixth (convert) processors convert the bytes sent/received into integers (All grok processors return string values, but you likely want this data stored as an integer so you can query, aggregate, and visualize correctly)
- The seventh/eighth (remove) processors delete the
backup_duration_minutes and backup_duration_seconds fields, as they were only used to perform the final backup_duration calculation above, and likely don't need to be stored.
PUT _ingest/pipeline/guahos
{
"description" : "my pipeline",
"processors" : [
{
"grok": {
"field": "message",
"patterns": [
"%{GREEDYMULTILINE}%{ROW_TITLE}%{WORD:backup_name}%{SRC_START}%{IP:backup_src}%{DST_START}%{WORD:backup_dst}"
],
"pattern_definitions": {
"GREEDYMULTILINE" : "(.|\n)*",
"ROW_TITLE": "Backup name: ",
"SRC_START": " \\(from \\[",
"DST_START": "\\] to \\["
}
}
},
{
"grok": {
"field": "message",
"patterns": [
"%{GREEDYMULTILINE}%{TIME_TAKEN_ROW_TITLE}%{NUMBER:backup_duration_minutes}%{GREEDYDATA}%{NUMBER:backup_duration_seconds}"
],
"pattern_definitions": {
"GREEDYMULTILINE" : "(.|\n)*",
"TIME_TAKEN_ROW_TITLE": "Time taken: "
}
}
},
{
"grok": {
"field": "message",
"patterns": [
"%{GREEDYMULTILINE}%{BYTES_SENT_ROW_TITLE}%{NUMBER:bytes_sent}%{GREEDYMULTILINE}%{BYTES_RECEIVED_ROW_TITLE}%{NUMBER:bytes_received}"
],
"pattern_definitions": {
"NEWLINE": "\n",
"GREEDYMULTILINE" : "(.|\n)*",
"BYTES_SENT_ROW_TITLE": "Total bytes sent: ",
"BYTES_RECEIVED_ROW_TITLE": "Total bytes received: "
}
}
},
{
"script": {
"source": "ctx.backup_duration = (Integer.parseInt(ctx.backup_duration_minutes) * 60) + Integer.parseInt(ctx.backup_duration_seconds)"
}
},
{
"convert": {
"field": "bytes_sent",
"type": "integer"
}
},
{
"convert": {
"field": "bytes_received",
"type": "integer"
}
},
{
"remove": {
"field": "backup_duration_minutes"
}
},
{
"remove": {
"field": "backup_duration_seconds"
}
}
]
}
Step 2: Index your data
When you index these documents, tell Elasticsearch to use the ingest pipeline you defined above (I used a pipeline id of guahos:
POST guahos/_doc?pipeline=guahos
{
"message": """
---------------------------------------------------------------
Backup name: Testbackup_new (from [192.168.1.1] to [linuxbox01])
Result: SUCCESS
Start time: Wed Oct 23 02:00:01 UTC 2019
End time: Wed Oct 23 02:00:02 UTC 2019
Time taken: 0 min 1 sec
Statistics:
Number of files: 7 (reg: 6, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 262.15M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 141
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 27
Total bytes received: 152
sent 27 bytes received 152 bytes 119.33 bytes/sec
total size is 262.15M speedup is 1,464,504.72
"""
}
Step 3: Query your index to verify
GET guahos/_search
{
}
Returns:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "guahos",
"_type" : "_doc",
"_id" : "mPfl-G0BGHnlifmLZXzK",
"_score" : 1.0,
"_source" : {
"bytes_received" : 152,
"message" : """
---------------------------------------------------------------
Backup name: Testbackup_new (from [192.168.1.1] to [linuxbox01])
Result: SUCCESS
Start time: Wed Oct 23 02:00:01 UTC 2019
End time: Wed Oct 23 02:00:02 UTC 2019
Time taken: 0 min 1 sec
Statistics:
Number of files: 7 (reg: 6, dir: 1)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 262.15M bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 141
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 27
Total bytes received: 152
sent 27 bytes received 152 bytes 119.33 bytes/sec
total size is 262.15M speedup is 1,464,504.72
""",
"bytes_sent" : 27,
"backup_dst" : "linuxbox01",
"backup_name" : "Testbackup_new",
"backup_duration" : 1,
"backup_src" : "192.168.1.1"
}
}
]
}
}