LINEのLLMOps — 大規模言語モデル運用の取り組み

LINE (now LINE Yahoo) has been advancing research and operation of its proprietary "HyperCLOVA X" series of large language models while running Japan's most widely used messaging service. This article gives an overview of the efforts — so-called "LLMOps" — for stably running LLMs in production. Details vary by published material and period, so please read this as general trends.

HyperCLOVA X is positioned as a multilingual model series originally developed at NAVER centered on Korean, then heavily augmented and fine-tuned by LINE with Japanese data. It is characterized as strengthening areas where global models tend to be weak, such as Japanese honorifics (keigo), business documents, and proper nouns.

Because LLM inference is computationally very costly, mechanisms for efficiently sharing GPUs are essential. LINE is reportedly building shared GPU pools in-house and adopting a configuration where inference requests from each service are received on Kubernetes. There is a trend of widely using OSS inference servers such as vLLM, which is characterized by efficient KV-cache management via PagedAttention and continuous batching.

A single LLM inference request consumes substantial computation, yet fully utilizing GPU compute requires batching multiple requests. Continuous batching is a technique that raises throughput by inserting new requests into the gaps of in-flight ones. Furthermore, parallelization techniques — tensor, pipeline, and expert parallelism (MoE) — are combined to place giant models across multiple GPUs.

To control cost in production without dramatically lowering accuracy, INT8/FP8 weight quantization and knowledge distillation from large to small models are performed in parallel. This improves response speed and GPU memory usage, with the ultimate goal of lowering cost per token.

In LLM applications, prompts become assets equivalent in importance to code. Many companies, including LINE, tend to set up mechanisms that retain versions, approval flows, and experiment history by managing prompts in Git or building in-house prompt management services. This makes it possible to trace regressions like "quality dropped after updating a certain prompt."

LLM quality cannot be measured by traditional accuracy and F-score alone — human evaluation, LLM-as-a-Judge, benchmarks (JGLUE, Japanese MT-Bench, etc.), and safety evaluation must be combined. LINE is thought to be building a setup where evaluation jobs are embedded in CI, with a battery of evaluations running automatically each time prompts or models are updated.

RAG (Retrieval-Augmented Generation) is a technique that retrieves documents relevant to a user's question via vector search, embeds them in the prompt, and passes them to the LLM. For a company like LINE with vast internal content — user history, FAQs, official account information — the choice of an appropriate vector embedding model and vector database (OpenSearch, Milvus, pgvector, etc.) becomes critical.

There are areas LLMs must not output, such as personal information, discriminatory expressions, and false medical advice. LINE is seen to insert safety filters on both input and output, concurrently using guardrail models and rule-based mitigation. Furthermore, a layer that applies anonymization and pseudonymization is needed to avoid feeding data corresponding to "special-care-required personal information" under the Act on the Protection of Personal Information into prompts.

LLM operating cost does not grow linearly with usage — it grows non-linearly with GPU occupancy time, model size, and context length. LINE is thought to commonly route requests through a layer that uses different-size models (giant, medium, lightweight) per use case — "simple summaries to a small model, hard reasoning to a large model."

LLMOps is not a world where "building a model and deploying it is done" — it is a continuous operational practice including evaluation, safety mitigation, cost management, and the buildout of RAG and data pipelines. A company like LINE, with a giant messaging platform, can be said to be in a position to create unique value by leveraging accumulated data and context.