Recruit データ基盤刷新事例 — 数十サービスの統合

Recruit Group operates dozens of services — SUUMO, Hot Pepper, Townwork, Zexy, Rikunabi, and more. Each had built its own data platform over many years, and as a result, data duplication and consistency problems became apparent. This article introduces the company-wide data platform renewal that Recruit implemented over several years.

Previously, each service individually adopted Hadoop, Redshift, Snowflake, and so on, and there were numerous cases where the same "member ID" was managed under different column names. When the marketing department tried to analyze across multiple services, the time required for data normalization each time was a major problem.

Recruit adopted Google BigQuery as the central data warehouse for the entire company. Reasons for selection include separated storage and compute making cost management easier, columnar storage enabling fast large-scale aggregation, and easy integration with other Google Cloud services.

For data transformations, dbt (data build tool) is adopted as a company-wide standard. dbt models are managed in Git and changes go through pull requests. As a result, SQL is no longer siloed in individuals, and it is now possible to track who changed what and when. Furthermore, dbt's features automatically visualize dependencies between tables, and data lineage is shared across the company.

As organizational scale grows, it becomes difficult for a central data platform team to manage data for all services. Recruit places a "Data Product Owner" in each service domain, who is responsible for the quality and publication of that domain's data. This is the so-called data mesh way of thinking. The central platform team provides common infrastructure and standards, while each domain operates autonomously.

Japan's revised Personal Information Protection Act strictly requires specification of usage purposes and acquisition of consent for third-party provision. Recruit built a common consent management platform and technically controls so that data can only be used within the scope users have permitted. On the BigQuery side as well, row-level security and column-level security are utilized to realize least-privilege access control.

In an environment with tens of thousands of tables company-wide, it is extremely important to be able to trace "from which source data is the number on this dashboard ultimately calculated." By combining dbt's lineage information with metadata management services like Dataplex, impact analysis and incident response can be performed quickly.

Recruit has put forward a policy to invest going forward in natural-language data search using generative AI, automated data quality diagnosis, and multi-region support envisioning international expansion.