Sample Compression, Support Vectors, and Generalization in Deep Learning

Submitted by admin on Mon, 06/10/2024 - 05:00

Even though Deep Neural Networks (DNNs) are widely celebrated for their practical performance, they possess many intriguing properties related to depth that are difficult to explain both theoretically and intuitively. Understanding how weights in deep networks coordinate together across layers to form useful learners has proven challenging, in part because the repeated composition of nonlinearities has proved intractable. This paper presents a reparameterization of DNNs as a linear function of a feature map that is locally independent of the weights.

A Fourier-Based Approach to Generalization and Optimization in Deep Learning

Submitted by admin on Mon, 06/10/2024 - 05:00

The success of deep neural networks stems from their ability to generalize well on real data; however, et al. have observed that neural networks can easily overfit randomly-generated labels. This observation highlights the following question: why do gradient methods succeed in finding generalizable solutions for neural networks while there exist solutions with poor generalization behavior?

PacGAN: The Power of Two Samples in Generative Adversarial Networks

Submitted by admin on Mon, 06/10/2024 - 05:00
Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little diversity, even when trained on diverse datasets. This phenomenon, known as mode collapse, has been the main focus of several recent advances in GANs.

Energy-Reliability Limits in Nanoscale Feedforward Neural Networks and Formulas

Submitted by admin on Mon, 06/10/2024 - 05:00

Due to energy-efficiency requirements, computational systems are now being implemented using noisy nanoscale semiconductor devices whose reliability depends on energy consumed. We study circuit-level energy-reliability limits for deep feedforward neural networks (multilayer perceptrons) built using such devices, and en route also establish the same limits for formulas (boolean tree-structured circuits).

An Information-Theoretic Approach to Unsupervised Feature Selection for High-Dimensional Data

Submitted by admin on Mon, 06/10/2024 - 05:00

In this paper, we propose an information-theoretic approach to design the functional representations to extract the hidden common structure shared by a set of random variables. The main idea is to measure the common information between the random variables by Watanabe's total correlation, and then find the hidden attributes of these random variables such that the common information is reduced the most given these attributes.

Information-Theoretic Lower Bounds for Compressive Sensing With Generative Models

Submitted by admin on Mon, 06/10/2024 - 05:00
It has recently been shown that for compressive sensing, significantly fewer measurements may be required if the sparsity assumption is replaced by the assumption the unknown vector lies near the range of a suitably-chosen generative model. In particular, in (Bora et al., 2017) it was shown roughly O(k log L) random Gaussian measurements suffice for accurate recovery when the generative model is an L-Lipschitz function with bounded k-dimensional inputs, and O(kdlogw) measurements suffice when the generative model is a k-input ReLU network with depth d and width w.

On the Fundamental Limit of Distributed Learning With Interchangable Constrained Statistics

Submitted by admin on Mon, 06/10/2024 - 05:00
In the popular federated learning scenarios, distributed nodes often represent and exchange information through functions or statistics of data, with communicative processes constrained by the dimensionality of transmitted information. This paper investigates the fundamental limits of distributed parameter estimation and model training problems under such constraints. Specifically, we assume that each node can observe a sequence of i.i.d. sampled data and communicate statistics of the observed data with dimensionality constraints.

Summary Statistic Privacy in Data Sharing

Submitted by admin on Mon, 06/10/2024 - 05:00
We study a setting where a data holder wishes to share data with a receiver, without revealing certain summary statistics of the data distribution (e.g., mean, standard deviation). It achieves this by passing the data through a randomization mechanism. We propose summary statistic privacy, a metric for quantifying the privacy risk of such a mechanism based on the worst-case probability of an adversary guessing the distributional secret within some threshold.