IDEAS home Printed from https://ideas.repec.org/p/bfi/wpaper/2020-154.html
   My bibliography  Save this paper

How Well Generative Adversarial Networks Learn Distributions

Author

Listed:
  • Tengyuan Liang

    (University of Chicago - Booth School of Business)

Abstract

This paper studies the rates of convergence for learning distributions implicitly with the adversarial framework and Generative Adversarial Networks (GAN), which subsume Wasserstein, Sobolev, MMD GAN, and Generalized/Simulated Method of Moments (GMM/SMM) as special cases. We study a wide range of parametric and nonparametric target distributions, under a host of objective evaluation metrics. We investigate how to obtain a good statistical guarantee for GANs through the lens of regularization. On the nonparametric end, we derive the optimal minimax rates for distribution estimation under the adversarial framework. On the parametric end, we establish a theory for general neural network classes (including deep leaky ReLU networks), that characterizes the interplay on the choice of generator and discriminator pair. We discover and isolate a new notion of regularization, called the generator-discriminator-pair regularization, that sheds light on the advantage of GANs compared to classical parametric and nonparametric approaches for explicit distribution estimation. We develop novel oracle inequalities as the main technical tools for analyzing GANs, which is of independent interest.

Suggested Citation

  • Tengyuan Liang, 2020. "How Well Generative Adversarial Networks Learn Distributions," Working Papers 2020-154, Becker Friedman Institute for Research In Economics.
  • Handle: RePEc:bfi:wpaper:2020-154
    as

    Download full text from publisher

    File URL: https://repec.bfi.uchicago.edu/RePEc/pdfs/BFI_WP_2020154.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hansen, Lars Peter, 1982. "Large Sample Properties of Generalized Method of Moments Estimators," Econometrica, Econometric Society, vol. 50(4), pages 1029-1054, July.
    2. Athey, Susan & Imbens, Guido W. & Metzger, Jonas & Munro, Evan, 2024. "Using Wasserstein Generative Adversarial Networks for the design of Monte Carlo simulations," Journal of Econometrics, Elsevier, vol. 240(2).
    3. Pakes, Ariel & Pollard, David, 1989. "Simulation and the Asymptotics of Optimization Estimators," Econometrica, Econometric Society, vol. 57(5), pages 1027-1057, September.
    4. Guido W. Imbens & Richard H. Spady & Phillip Johnson, 1998. "Information Theoretic Approaches to Inference in Moment Condition Models," Econometrica, Econometric Society, vol. 66(2), pages 333-358, March.
    5. Cai, T. Tony & Liang, Tengyuan & Zhou, Harrison H., 2015. "Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional Gaussian distributions," Journal of Multivariate Analysis, Elsevier, vol. 137(C), pages 161-172.
    6. Back, Kerry & Brown, David P, 1993. "Implied Probabilities in GMM Estimators," Econometrica, Econometric Society, vol. 61(4), pages 971-975, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antoine, Bertille & Bonnal, Helene & Renault, Eric, 2007. "On the efficient use of the informational content of estimating equations: Implied probabilities and Euclidean empirical likelihood," Journal of Econometrics, Elsevier, vol. 138(2), pages 461-487, June.
    2. Joachim Inkmann, 2000. "Finite Sample Properties of One-Step, Two-Step and Bootstrap Empirical Likelihood Approaches to Efficient GMM Estimation," Econometric Society World Congress 2000 Contributed Papers 0332, Econometric Society.
    3. Parente, Paulo M.D.C. & Smith, Richard J., 2011. "Gel Methods For Nonsmooth Moment Indicators," Econometric Theory, Cambridge University Press, vol. 27(1), pages 74-113, February.
    4. Shane M. Sherlund, 2004. "Quasi Empirical Likelihood Estimation of Moment Condition Models," Econometric Society 2004 North American Summer Meetings 507, Econometric Society.
    5. Giuseppe Ragusa, 2011. "Minimum Divergence, Generalized Empirical Likelihoods, and Higher Order Expansions," Econometric Reviews, Taylor & Francis Journals, vol. 30(4), pages 406-456, August.
    6. Hill, Jonathan B. & Prokhorov, Artem, 2016. "GEL estimation for heavy-tailed GARCH models with robust empirical likelihood inference," Journal of Econometrics, Elsevier, vol. 190(1), pages 18-45.
    7. Joachim Inkmann, 2010. "Estimating Firm Size Elasticities of Product and Process R&D," Economica, London School of Economics and Political Science, vol. 77(306), pages 384-402, April.
    8. Alain Guay & Florian Pelgrin, 2007. "Using Implied Probabilities to Improve Estimation with Unconditional Moment Restrictions," Cahiers de recherche 0747, CIRPEE.
    9. Hill, Jonathan B. & Aguilar, Mike, 2013. "Moment condition tests for heavy tailed time series," Journal of Econometrics, Elsevier, vol. 172(2), pages 255-274.
    10. Almeida, Caio & Garcia, René, 2012. "Assessing misspecified asset pricing models with empirical likelihood estimators," Journal of Econometrics, Elsevier, vol. 170(2), pages 519-537.
    11. Smith, Richard J., 2011. "Gel Criteria For Moment Condition Models," Econometric Theory, Cambridge University Press, vol. 27(6), pages 1192-1235, December.
    12. Bera, Anil K. & Bilias, Yannis, 2002. "The MM, ME, ML, EL, EF and GMM approaches to estimation: a synthesis," Journal of Econometrics, Elsevier, vol. 107(1-2), pages 51-86, March.
    13. Inkmann, J., 2005. "Inverse Probability Weighted Generalised Empirical Likelihood Estimators : Firm Size and R&D Revisited," Other publications TiSEM c39cff1f-16c1-4446-a83f-c, Tilburg University, School of Economics and Management.
    14. Jesús Fernández-Villaverde & Juan F. Rubio-Ramírez, 2007. "Estimating Macroeconomic Models: A Likelihood Approach," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 74(4), pages 1059-1087.
    15. Cui, Li-E & Zhao, Puying & Tang, Niansheng, 2022. "Generalized empirical likelihood for nonsmooth estimating equations with missing data," Journal of Multivariate Analysis, Elsevier, vol. 190(C).
    16. Inkmann, Joachim, 2000. "Misspecified heteroskedasticity in the panel probit model: A small sample comparison of GMM and SML estimators," Journal of Econometrics, Elsevier, vol. 97(2), pages 227-259, August.
    17. Smith, Richard J., 2007. "Efficient information theoretic inference for conditional moment restrictions," Journal of Econometrics, Elsevier, vol. 138(2), pages 430-460, June.
    18. Nevo, Aviv, 2003. "Using Weights to Adjust for Sample Selection When Auxiliary Information Is Available," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 43-52, January.
    19. Prosper Dovonon, 2016. "Large Sample Properties of the Three-Step Euclidean Likelihood Estimators under Model Misspecification," Econometric Reviews, Taylor & Francis Journals, vol. 35(4), pages 465-514, April.
    20. Cizek, P., 2009. "Generalized Methods of Trimmed Moments," Discussion Paper 2009-25, Tilburg University, Center for Economic Research.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bfi:wpaper:2020-154. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Toni Shears (email available below). General contact details of provider: https://edirc.repec.org/data/mfichus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.