# Dec 23rd, 2024: \[EN\] Building a Gift Production Monitoring System with Elastic Stack

**URL:** <https://discuss.elastic.co/t/dec-23rd-2024-en-building-a-gift-production-monitoring-system-with-elastic-stack/371773>\
**Category:** Advent Calendar\
**Created:** [December 23, 2024, 8:00am UTC](https://discuss.elastic.co/t/dec-23rd-2024-en-building-a-gift-production-monitoring-system-with-elastic-stack/371773 "2024-12-23T08:00:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex\_Salgado-Elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_salgado-elastic/32/103081_2.png) [@Alex\_Salgado-Elastic](https://discuss.elastic.co/u/Alex_Salgado-Elastic)\
**Post date:** [December 23, 2024, 8:00am UTC](https://discuss.elastic.co/t/dec-23rd-2024-en-building-a-gift-production-monitoring-system-with-elastic-stack/371773/1 "2024-12-23T08:00:20Z")

</div>

This post is also available [in portuguese](https://discuss.elastic.co/t/372025).

 ![North Pole Monitoring System](https://us1.discourse-cdn.com/elastic/original/3X/3/1/3163bbaab5e866dcde0c53b5bdf3841f54348e2a.jpeg)

## The Challenge at the North Pole

"Ho ho... oh no!" exclaimed Santa Claus on a frosty morning. "There are only two days until Christmas! How can I keep track of the wishes of billions of children without proper monitoring?"

For centuries, Santa's workshop relied on magic dust and elf intuition to track toy production. But in 2024, even the North Pole needs a tech upgrade! Today, we’ll help Santa modernize his factory by implementing a real-time monitoring solution with the Elastic Stack.

## Requirements

Mrs. Claus was clear about what we need:

- Track toy production across all factory lines
- Monitor elf performance (keeping the Christmas spirit high!)
- Ensure quality control (no child should receive broken toys!)
- Monitor factory conditions (elves need ideal temperatures for their hot cocoa!)

## Magical Prerequisites

Before we start enchanting the factory, make sure you have:

- Python 3.8+ (tested by elves, approved by Santa)
- Access to Elastic Cloud (or local Elastic Stack 8.16)

## Factory Structure

First, let’s organize our factory like Santa organizes his gift list. Clone the repository from GitHub:

```bash
git clone https://github.com/salgado/santa_advent_monitoring.git 
cd santa_advent_monitoring

```

By the end, your folder structure will look like this:

```bash
santa_advent_monitoring/
├── manage_simulation.sh
├── toy_workshop_simulator.py
│   
├── var/
│ └── log/                      
│ └── workshop/
│ └── production/
└── README.md

```

## The Factory Example

We created a Python script that simulates Santa’s workshop, generating JSON logs with toy production metrics. The simulation is managed by a Bash script, responsible for starting, stopping, and monitoring production.

The Python code generates realistic data on toy types, production rates, quality scores, and environmental conditions. The Bash script manages the simulation lifecycle, rotates logs, and provides status updates. Think of it as a miniature version of a real production line—but instead of physical toys, it produces structured data to feed the Elastic Stack in real time.

### Factory JSON Example

```json
{
  "@timestamp": "2024-12-11T16:54:36.980882",
  "toy_type": "robot",
  "production_line": "line1",
  "production_rate": 101,
  "quality_score": 95,
  "elf_id": "elf_20",
  "errors_detected": 0,
  "temperature": 22.8,
  "humidity": 47.3,
  "shift": "evening",
  "machine_status": "normal",
  "toys_completed": 25,
  "version": "8.16.1"
}

```

This log contains production information: toy type, production line, rate, quality, environmental conditions, and more.

### Field Descriptions

| **Field** | **Description** | **Examples** | **Importance** |
| --- | --- | --- | --- |
| **toy\_type** | Type of toy produced | robot, doll, car, puzzle, board\_game | Defines complexity and production rate |
| **production\_line** | Production line | line1, line2, line3 | Locates production origin |
| **quality\_score** | Quality score | 0-100 | Critical: must be ≥ 90 |
| **elf\_id** | Elf identifier | elf\_1 to elf\_20 | Tracks performance |
| **errors\_detected** | Errors detected | 0-5 | Critical if \> 3 |
| **machine\_status** | Machine status | normal, error | Indicates production health |
| **toys\_completed** | Completed units | 15-50 | Measures productivity |

### Initial Factory Configuration

`configs/simulator/production_config.yml`:

```yaml
toy_types:
  robot:
    base_rate: 100
    complexity: 0.8
    min_quality: 85
    components: ["circuit_board", "motors", "sensors"]
  doll:
    base_rate: 150
    complexity: 0.6
    min_quality: 88
    components: ["fabric", "stuffing", "clothes"]
  # ... more toy types ...

production_lines:
  line1:
    efficiency: 0.95
    error_rate: 0.02
    maintenance_schedule: "0 */4 * * *"
  # ... more production lines ...

```

## Implementing the Factory Simulator

Let’s get Santa’s factory up and running! 🎅

### The Production Simulator

Ensure you’re in the correct directory:

```bash
cd santa_advent_monitoring

```

### Management Script

`manage_simulation.sh`

#### Starting the Simulation

To start toy production:

```bash
./manage_simulation.sh start

```

This starts the simulator in the background, creates the log directory structure, and runs `toy_workshop_simulator.py`, which generates JSON events simulating production. The process PID is saved in `simulator.pid` for future management, and all logs go to `toys.log`. We’re powering up our virtual factory! 🎅🏭

## Connecting Our Factory to the Cloud ☁

With the simulator running, let’s connect it to Elastic Cloud using the Elastic Agent.

### Elastic Agent Installation

Follow the instructions in Kibana under _Fleet → Agents_ to install the Elastic Agent for your environment.

 ![Creating Agents](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a376604e1dde15c3d57bcffb891392a36296a581.jpeg)

Configure the factory log path:

```bash
/your-full-path/santa_advent_monitoring/var/log/workshop/production/toys.log

```

Note: At the end of the installation, selecting the "Enroll in Fleet" option will also set up the Fleet Server.

## Configuring Factory Log Collection 🎄

With logs being generated, we need to configure the Elastic Agent to collect them. Let’s adjust the "Custom Logs" integration.

### Navigating to the Configuration

1. In Kibana, go to **Fleet**
2. Under **Agent policies** , select **Agent policy 1**
3. Click **Add integration**
4. Search for "Custom logs" and select it

### Adjusting the Integration

#### 1. Basic Settings

- Integration name: `santa_workshop_logs`
- Description: `Santa's Workshop Pro`

#### 2. Log Path

```bash
/Users/username/your_path/santa_advent_monitoring/var/log/workshop/production/toys.log

```

#### 3. Advanced Configurations

In "Custom configurations":

```yaml
json:
  keys_under_root: true
  add_error_key: true
  overwrite_keys: true
  decode_json_fields:
    fields: ["message"]
    target: ""
    process_array: false

```

**Key points:**

- **json.keys\_under\_root:** Allows fields like `@timestamp`, `toy_type`, `production_rate` to appear at the root level for easier searches.
- **json.add\_error\_key:** Adds an `error` field if parsing issues occur, helping identify malformed entries.
- **json.overwrite\_keys:** In case of duplicate keys, the latest one prevails. Maintains data consistency.

#### 4. Dataset

- Dataset name: "generic" (or "santa\_workshop" if preferred)
- Namespace: "default"

#### 5. Finalizing

- Under "Where to add this integration?", confirm that **Agent policy 1** is selected
- Click **Save integration**

### Verifying the Configuration

After saving, wait a few moments. In Kibana, go to **Discover** and look for the `logs-*` index. You should see events arriving, showing toy types, production rates, quality, and more.

## Building Santa’s Command Center 🎄

Let’s create a dashboard that will impress even the most technical elf.

### Initial Data View Setup

1. In **Stack Management → Data Views → Create data view** :
  - Name: `santa-workshop`
  - Index pattern: `logs-*`
  - Timestamp field: `@timestamp`

### Creating the Dashboard

1. In Kibana, go to **Dashboards**
2. Click **Create dashboard**

### Creating Visualizations

Use **Create visualization** for each one:

#### 1. Total Toys Produced (Metric)

- Metric: Sum of `toys_completed`
- Title: "Total Toys Produced"

#### 2. Quality Score (Metric)

- Metric: Average of `quality_score`
- Title: "Quality Score"
- Format: Percentage

#### 3. Average Production Rate (Metric)

- Metric: Average of `production_rate`
- Title: "Average Production Rate (per hour)"

#### 4. Distribution by Toy Type (Pie)

- Metric: Count
- Split slices: `toy_type`
- Title: "Production by Toy Type"

#### 5. Line Performance (Vertical Bar)

- Y-axis: `production_rate`
- X-axis: `production_line`
- Title: "Production Line Performance"

#### 6. Environmental Conditions (Line)

- Y-axis: `temperature` and `humidity`
- X-axis: `@timestamp`
- Title: "Workshop Environmental Conditions"

### Dashboard Layout

 ![blog-image3](https://us1.discourse-cdn.com/elastic/original/3X/7/9/79c18d15c921673f0ce5e6d0c97601e3580590a3.png)

Organize visualizations:

- First row: Key metrics (Total, Quality, Rate)
- Second row: Toy type distribution and Line performance
- Third row: Environmental conditions

Resize as needed. The goal is to make the dashboard easy to interpret at a glance.

## Proactive Monitoring with SLOs and Alerts

Ho ho ho! Let’s ensure quality remains high! Let’s configure a Service Level Objective (SLO).

 ![Create new SLO](https://us1.discourse-cdn.com/elastic/original/3X/5/d/5de48839a186619f5418496e2168683995f7780f.jpeg)

### Creating a Quality SLO

1. In **Observability → SLOs** :

2. **SLI Definition:**

3. **Objectives:**

4. **Description:**

### Setting Up Alerts

 ![Create RULE](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0eb799bfc1ec11c5ed2868a6a50b3a1379035b78.jpeg)

We’ll set clear goals:

- **Gift Quality SLO:**

- **Production Efficiency SLO:**

#### Creating an Alert Rule

1. **Observability → Alerts → Create rule**

2. Type: "SLO burn rate"

3. **Alert Configuration:**

4. **Alert Message:**

### Example Alert in Action

```json
{
  "@timestamp": "2024-12-11T16:54:40.999778",
  "toy_type": "puzzle",
  "production_line": "line1",
  "production_rate": 103,
  "quality_score": 69,
  "elf_id": "elf_4",
  "errors_detected": 5,
  "temperature": 20.8,
  "humidity": 50.9,
  "shift": "evening",
  "machine_status": "error",
  "toys_completed": 25,
  "version": "8.16.1"
}

```

In this case, quality dropped. The alert will help quickly fix the issue.

### Configuring Alerts via Elasticsearch Query

1. **Stack Management → Rules → Create rule**
2. Type: "Elasticsearch query"
3. Query:

```json
{
  "bool": {
    "must": [
      {
        "range": {
          "quality_score": {
            "lt": 90
          }
        }
      }
    ]
  }
}

```

4. Parameters:
  - Name: "Gift Quality Alert"
  - Indices: `logs-*`
  - Schedule: Every 5 minutes
  - Actions: Notify supervisors (email/Slack)

### SLO Panel

 ![SLOs Panel](https://us1.discourse-cdn.com/elastic/original/3X/5/0/50a4cdcf25fb60f01f0ae0e5b0295fbff439fc08.png)

Our SLO panel will show quality trends, maintaining the Christmas spirit and children’s happiness.

## Conclusion: A Modern and Magical Workshop! 🎄✨

Our complete solution provides:

1. Data collection for production ✅
2. Elastic Agent setup ✅
3. Visualization dashboard ✅
4. Proactive SLOs and alerts ✅

Santa’s workshop now combines magic and technology! “This system has transformed gift production,” celebrates Bernard, Chief Operations Elf. “Now we ensure every child gets their perfect gift on time!”

## Additional Resources

- [Elastic Agent Documentation](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html)
- [Kibana Dashboard Guide](https://www.elastic.co/guide/en/kibana/current/dashboard.html)
- [Elastic Alerting Documentation](https://www.elastic.co/guide/en/kibana/current/alerting-getting-started.html)

_Remember: The best gifts arrive at the right time, and the best monitoring system makes that possible! 🎁✨_

#ElasticAdvent #ModernSanta #ObservabilityMagic #ElasticsearchForElves #Elastic