야, 너두 GPU에 LLM 배포할 수 있어: 온프레미스 GPU 기반 k8s 환경의 LLMOps 최적화 여정 | Model & Agent

Summary

The presentation describes Kakao Pay’s effort to serve open-weight LLMs on an on-premises GPU Kubernetes environment, focusing on infrastructure and operations rather than model training or algorithm design.

Why Use On-Premises GPUs?

• The speakers argue that on-premises GPUs can become more economical than cloud GPU rentals as usage grows. Their comparison reports API costs of roughly 2.8× for GLM, 4–6× for Sonnet, and 10× for Opus relative to their on-premises baseline.
• Open-weight models make this approach practical: organizations can run models using publicly available weights without training them themselves.
• An internal user survey found GLM’s quality sufficient for many tasks. The team reserved a higher-end API for tasks requiring stronger performance.

LLM Serving Fundamentals and GPU Configuration

The talk explains GPU VRAM, the KV cache (stored intermediate results that avoid recomputing prior conversation context), and key serving metrics:

• TTFT: Time until the first response token.
• KV-cache hit rate: How often prior computation can be reused.
• End-to-end latency: The response time experienced by users.

For a 122-billion-parameter model, FP16 weights require about 244 GB, exceeding the capacity of one 141-GB H200. Splitting the model across two GPUs leaves memory available for the KV cache.

Quantization reduces weight memory requirements, potentially at a cost to accuracy. The presentation gives approximate requirements of 244 GB for FP16, 122 GB for FP8, and 61 GB for FP4.

The speakers distinguish between using a full GPU, partitioning it with MIG, and connecting GPUs with tensor parallelism (TP). MIG can help host multiple small models, but its partitions cannot communicate directly. For large models, the team recommends disabling MIG and using GPU-to-GPU links for TP; their measurements showed higher throughput and lower TTFT.

Making GPU Capacity Go Further

The initial design dedicated GPU capacity to individual services. As demand grew, this left resources idle at different times—for example, when batch workloads ran overnight.

Kakao Pay moved compatible services into shared GPU pools while keeping dedicated capacity where isolation was needed. The reported result was recovery of roughly 25% of previously wasted GPU capacity.

The pool uses priorities to protect user-facing services: customer-service workloads take precedence, while lower-priority work can wait. Autoscaling combines concurrency monitoring with scheduled scaling for predictable demand.

Because GPU pods take a long time to start, the team reduced startup delays by:

1. Copying model files to node-local disks instead of having pods fetch them simultaneously over the network.
2. Measuring and pre-setting memory allocation for the KV cache, avoiding repeated profiling.
3. Caching and reusing GPU kernel optimizations and related CUDA-graph work.

Together, these measures reduced a startup process that had taken around ten minutes to just over three minutes. The talk reports a reduction of up to about 70% from the combined optimizations.

Improving Serving Speed and Cache Reuse

Speculative decoding uses a smaller draft model to propose tokens that the main model verifies. The talk also discusses MTP, which uses model-internal layers to generate candidate tokens. The team reports an approximately 18% throughput increase after applying MTP to GLM.

When GPU memory is insufficient for the KV cache, KV-cache offloading can extend available cache capacity beyond GPU memory. Although offloading introduces I/O costs, it can be beneficial when it prevents expensive recomputation. The speakers also mention remote offload and GPU Direct Storage.

Default Kubernetes round-robin routing can send turns from the same conversation to different pods, preventing reuse of that conversation’s cache. Kakao Pay added routing based on request headers, then refined it to use session identifiers to reduce uneven load when users run multiple sessions.

After the improvements, the team reports better cache-hit rate, TTFT, and latency even as traffic more than doubled.

LMCache and features in newer KServe versions—such as cache management, prefill/decode separation, and failure scaling—are discussed as potential ways to reduce custom operational work.

Guides, Evaluations, and Automation

1. GPU sizing guide: Estimate model-weight memory from parameter count and data precision, then account for memory needed by the KV cache.
2. Serving configuration guide: Use the vLLM recipe as a starting point to select a model and configure launch parameters, quantization, and speculative decoding.
3. Model evaluation process: Benchmark throughput with representative traffic and separately assess production quality using service-specific evaluation datasets. Official benchmarks alone may not predict results for a particular service.
4. Automated model onboarding: A Kubeflow-pipeline-based workflow downloads and benchmarks models during idle GPU periods, generates reports and alerts, and runs evaluations against maintained datasets. Teams can decide whether to deploy based on the results. A separate endpoint allows a model to change without requiring routing changes.

Main Speakers

• Lee Yong-hee (Kakao Pay AI Platform Team): Serving fundamentals, GPU utilization, and startup optimization.
• Choi Il-ji (Kakao Pay MLOps): Serving performance, model evaluation, and onboarding automation.