Improve Diverse Text Generation by Self Labeling Conditional
Improve Diverse Text Generation by Self Labeling Conditional Variational Auto Encoder Yuchi Zhang*, Yongliang Wang, Liping Zhang, Zhiqiang Zhang, Kun Gai Alibaba Group May 14 th, 2019
Contents • Background • Related Work • Method • Experiment • EGOODS Dataset • Conclusion
Background What should generated sentences of high quality satisfy? • Semantically Meaningful • Grammatically Correct • Diverse
Background Example 1 Open Domain Dialogue Generation
Background Example 2 Ecommerce Applications @ Tabao one-to-many / one source, multi target
Related Work • Seq 2 Seq: No Diversity • MMI-Anti. LM, Diverse BS: Cannot handle one-to-many problems well • VAE/CVAE: KL Vanishing • KL Anneal/Free Bits/Word Dropout/BOW Loss: Better but still limited
Related Work Maximize the ELBO VAE CVAE
Related Work • Fatal problem of vanilla VAE/CVAE: KL-Vanishing • Our analysis of KL-Vanishing taking VAE as an example • The second term of (1) • According to Jensen’s Inequality: sign holds when is independent of reaches its global minimum of 0 when equal
Related Work
Related Work • Advantages • • KL Annealing Mitigate the KL-Vanishing Problem Disadvantages • Heuristic: To what extent? • Harm the accuracy of generation Bag of word (BOW) L Free bits, Word Dropout and so on…
Method
Method Objective Function Labeling Loss: Recovery Loss:
Method Alternative Training Process CVAE Phase Labeling Phase
Experiments • • Dataset • Daily Dialogue: 13118 Multi-turn dialogue sessions, 11118/1000 for training/evaluation/tesing • EGOODS Dataset Evaluation Metrics • BLEU-prec/rec • Distinct-1/2: # of distinct words (uni/bi grams) divided by # of total words in generated sentences • Human Evaluation: 7 experts are hired to judge Fluency, Coherence and Diversity
EGOODS Dataset • Native one-to-many dataset • Item description corpus collected from tabao • 3001140 source-target pairs from 789582 items • One Source: The item description by its seller • Multi Targets：Several recommendation sentences of this item written by third-party who is payed to make these sentences more attractive to customers • 2961319/19536/20287 pairs in Training/Validation/Testing Sets
Experiments
Experiments
Experiments
Conclusion • Diversity plays an important role • SLCVAE achieves higher diversity while the accuracy is as same as that of baselines • Future work: Gaussian Mixture Modeling instead of single Gaussian latent, Compared to more related works by more experiments
- Slides: 19