Case study · Microsoft XiaoIce · 2017–2020

XiaoIce music: from a KDD paper to a $1M+ business

I led AI music creation at XiaoIce: co-designing the model, building the data pipeline under it, and turning it into a product and a business API.

Project lead, AI music creation·2017–2020

LayersResearchModelSystemProductGrowthRevenue

At a glance

$1M+annual revenue from AI music

KDD'18Best Student Paper (Research Track), co-authorSource: Synced, 2018-08-21 ↗

CCTVXiaoIce's machine composition, built on the model, shown on 机智过人Source: 中国科大新闻网, 2018-08-28 ↗

01context

Context

The question in 2017: could a chatbot write a whole song from a line of text, with melody, arrangement, and lyrics that fit together, well enough to ship and to sell?

02system

How it works

Two tracks, one system. The research track became XiaoIce Band: CRMCG generates melody from chord progression and rhythm, and MICA arranges multiple instruments with multi-task learning. The product track became text-to-song: extract topic and emotion, choose a melody, write lyrics to it, produce the song.

Research
- XiaoIce Band (KDD 2018)
- CRMCG: melody from chords and rhythm
- MICA: multi-instrument arrangement
Data
- Data format definition
- Data gathering and mining
- Training data pipeline
Product
- Text → topic and emotion
- Melody → lyrics → song
- Business API for customers

Song generation from a text inputExtracts topic and emotion from a text input, determines a melody conditioned on them, writes lyrics from the melody and the text, and produces a song.

03role

What I owned

Co-designed the algorithm architecture.
Defined the data format, and owned data gathering, data mining, and the data pipeline for the model.
Led the product build.
Delivered the business API, pitched customers, and did business development.

04more

Coverage & related

Coverage

→ The KDD paper → Patent: song generation from a text input