Improve Diverse Text Generation by Self Labeling Conditional

Improve Diverse Text Generation by Self Labeling Conditional Variational Auto Encoder Yuchi Zhang*, Yongliang Wang, Liping Zhang, Zhiqiang Zhang, Kun Gai Alibaba Group May 14 th, 2019

Contents • Background • Related Work • Method • Experiment • EGOODS Dataset • Conclusion

Background What should generated sentences of high quality satisfy? • Semantically Meaningful • Grammatically Correct • Diverse

Background Example 1 Open Domain Dialogue Generation

Background Example 2 Ecommerce Applications @ Tabao one-to-many / one source, multi target

Related Work • Seq 2 Seq: No Diversity • MMI-Anti. LM, Diverse BS: Cannot handle one-to-many problems well • VAE/CVAE: KL Vanishing • KL Anneal/Free Bits/Word Dropout/BOW Loss: Better but still limited

Related Work Maximize the ELBO VAE CVAE

Related Work • Fatal problem of vanilla VAE/CVAE: KL-Vanishing • Our analysis of KL-Vanishing taking VAE as an example • The second term of (1) • According to Jensen’s Inequality: sign holds when is independent of reaches its global minimum of 0 when equal

Related Work

Related Work • Advantages • • KL Annealing Mitigate the KL-Vanishing Problem Disadvantages • Heuristic: To what extent? • Harm the accuracy of generation Bag of word (BOW) L Free bits, Word Dropout and so on…

Method

Method Objective Function Labeling Loss: Recovery Loss:

Method Alternative Training Process CVAE Phase Labeling Phase

Experiments • • Dataset • Daily Dialogue: 13118 Multi-turn dialogue sessions, 11118/1000 for training/evaluation/tesing • EGOODS Dataset Evaluation Metrics • BLEU-prec/rec • Distinct-1/2: # of distinct words (uni/bi grams) divided by # of total words in generated sentences • Human Evaluation: 7 experts are hired to judge Fluency, Coherence and Diversity

EGOODS Dataset • Native one-to-many dataset • Item description corpus collected from tabao • 3001140 source-target pairs from 789582 items • One Source: The item description by its seller • Multi Targets：Several recommendation sentences of this item written by third-party who is payed to make these sentences more attractive to customers • 2961319/19536/20287 pairs in Training/Validation/Testing Sets

Experiments

Experiments

Experiments

Conclusion • Diversity plays an important role • SLCVAE achieves higher diversity while the accuracy is as same as that of baselines • Future work: Gaussian Mixture Modeling instead of single Gaussian latent, Compared to more related works by more experiments