Background

  • To secure additional clients and enhance product competitiveness, the business model aimed to expand the scope of performance monitoring. This led to the necessity of collecting system performance data from Kubernetes.
  • The goal was to collect real-time performance data from Kubernetes resources, as well as from individual Pods and Containers on each node, input it into the company’s database, and display it in the existing server monitoring solution, onTune Viewer.

Preparation

Individual Contributions

  • Kubernetes Data Review: Assessed the information available for collection from Kubernetes.
  • Performance Data Collection Review: Examined methods for collecting performance data from Kubernetes.

  • Resource and Performance Metric Analysis: Analyzed available key resource information and performance metrics.
  • Process and Architecture Development: Created the development process and architecture.

Alpha Development

(2022.07~12)

Individual Contributions

  • Planning, Process, and Architecture Design: Carried out the planning and design of the process and architecture independently.
  • DB Schema and Environment Setup: Configured the database schema and environment.
  • Resource and Performance Data Collection Program Development: Developed the program to collect performance data.

Development Environment

  • Language: Python
  • Kubernetes: v1.24
  • Platform: Windows, Linux (Independent)

Development Work

  • Collected resource data using the Kubernetes API.
  • Collected metric data using the Kubelet Summary API.
  • Input and updated the collected resource and metric data into a PostgreSQL database.
  • Performed real-time monitoring of Kubernetes and database status.
  • Developed a watchdog to ensure continuous program operation.

Development Results

  • Successfully initiated the beta version development
  • Achieving the required features and functionality.

Beta Development

(2022.12~2023.07)

Team Project Contributions (4-member team)

  • System Planning, Process, and Architecture Design: Led the planning and design of system architecture and process.
  • DB Schema Expansion and Improvement: Expanded and improved the database schema.
  • Development of Collection and Input Modules: Focused on developing the collection and input modules, while the other team members assisted with development. I completed the final product.

Development Environment

  • Language: Go
  • Kubernetes: v1.26
  • Platform: Windows, Linux (Independent)

Key Changes

  • Migration from Python to Go.
  • Integration with the company’s main product, onTune Viewer, for real-time monitoring.
  • Incorporated additional requirements based on feedback from the QA team.

Development Work

  • Gathered basic program information through configuration files (initially using env files, later switched to toml files).
  • Developed a logging module for performance data collection and database input.
  • Collected performance data from multiple clusters in a Kubernetes Multi-Cluster environment.
  • Resource Data Collection: Used the Kubernetes API (same as the Alpha version).
  • Metric Data Collection: Improved upon the Alpha version by using the Kubelet-embedded cAdvisor for metric data collection.
  • Input and updated the collected resource and metric data in the PostgreSQL database.
  • Monitored Kubernetes and database status in real-time.
  • Managed the collection intervals of resources and metrics, and handled cluster state changes.

Development Results

  • Completed the beta version development and QA.
  • Integration of this module into the web version of the company’s main product is planned once that product is complete.

Architecture

 

Kubernetes Operation Experience?

I had no prior experience operating a Kubernetes cluster. Kubernetes was not utilized in the company’s development environment, and I took it upon myself to build a Kubernetes cluster and create proposals and documentation to integrate it with a DevOps environment, but it was not adopted. At the time, a new development director had been hired, and a large-scale project was underway to convert the company’s existing CS program into a web-based version. The development team had no experience with CI/CD or working in a DevOps environment, so the team members were anticipated to struggle with completely restructuring the development process. As a result, the plan to introduce a DevOps environment using Kubernetes was postponed, and we prioritized web version development and GitHub-based code management for integration.

Thus, the Kubernetes cluster served primarily as a monitoring test target, and while I personally experimented with various ways to operate it for resource and performance data collection, it was limited. It would have been ideal to utilize Kubernetes more broadly for internal DevOps and other tasks, but practical constraints prevented that.

Migration from Python (Alpha) to Go (Beta) Version

The primary reason for migrating from Python to Go was simple: Kubernetes is an open-source platform developed in Go, so Go provided better compatibility and superior performance for our program. There were no significant difficulties in migrating programming languages, but three key issues required resolution:

  • How to develop independent modules using Go’s lightweight threading (Go routines).
  • How to effectively transfer data using Go channels.
  • How to mitigate memory leaks caused by program execution.

This was my first project based on Go, but collaboration among team members allowed us to reference many valuable resources, which greatly contributed to our learning. Although the team members were not deeply knowledgeable about Kubernetes, they provided many best practices for Go, while I proposed solutions for memory leak issues using sync modules, benefiting the development process.

In contrast to a previous dashboard project where I served as project manager and team lead, this project was purely collaborative. While I led much of the planning, each member developed their respective modules, sharing and collaborating on key information and issues, resulting in a successful development without major problems.