担当区分：筆頭著者,　責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

We propose the particle dual averaging (PDA) method, which generalizes the dual averaging method in convex optimization to the optimization over probability distributions with quantitative runtime guarantee. The algorithm consists of an inner loop and outer loop: the inner loop utilizes the Langevin algorithm to approximately solve for a stationary distribution, which is then optimized in the outer loop. The method can thus be interpreted as an extension of the Langevin algorithm to naturally handle nonlinear functional on the probability space. An important application of the proposed method is the optimization of neural network in the mean field regime, which is theoretically attractive due to the presence of nonlinear feature learning, but quantitative convergence rate can be challenging to obtain. By adapting finite-dimensional convex optimization theory into the space of measures, we analyze PDA in regularized empirical / expected risk minimization, and establish quantitative global convergence in learning two-layer mean field neural networks under more general settings. Our theoretical results are supported by numerical simulations on neural networks with reasonable size.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85131888963&origin=inward

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Deep learning has exhibited superior performance for various tasks, especially for high-dimensional datasets, such as images. To understand this property, we investigate the approximation and estimation ability of deep learning on anisotropic Besov spaces. The anisotropic Besov space is characterized by direction-dependent smoothness and includes several function classes that have been investigated thus far. We demonstrate that the approximation error and estimation error of deep learning only depend on the average value of the smoothness parameters in all directions. Consequently, the curse of dimensionality can be avoided if the smoothness of the target function is highly anisotropic. Unlike existing studies, our analysis does not require a low-dimensional structure of the input data. We also investigate the minimax optimality of deep learning and compare its performance with that of the kernel method (more generally, linear estimators). The results show that deep learning has better dependence on the input dimensionality if the target function possesses anisotropic smoothness, and it achieves an adaptive rate for functions with spatially inhomogeneous smoothness.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85124227156&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（学術雑誌）

The minibatching technique has been extensively adopted to facilitate stochastic first-order methods because of their computational efficiency in parallel computing for large-scale machine learning and data mining. Indeed, increasing the minibatch size decreases the iteration complexity (number of minibatch queries) to converge, resulting in the decrease of the running time by processing a minibatch in parallel. However, this gain is usually saturated for too large minibatch sizes and the total computational complexity (number of access to an example) is deteriorated. Hence, the determination of an appropriate minibatch size which controls the trade-off between the iteration and total computational complexities is important to maximize performance of the method with as few computational resources as possible. In this study, we define the optimal minibatch size as the minimum minibatch size with which there exists a stochastic first-order method that achieves the optimal iteration complexity and we call such a method the optimal minibatch method. Moreover, we show that Katyusha (in: Proceedings of annual ACM SIGACT symposium on theory of computing vol 49, pp 1200–1205, ACM, 2017), DASVRDA (Murata and Suzuki, in: Advances in neural information processing systems vol 30, pp 608–617, 2017), and the proposed method which is a combination of Acc-SVRG (Nitanda, in: Advances in neural information processing systems vol 27, pp 1574–1582, 2014) with APPA (Cotter et al. in: Advances in neural information processing systems vol 27, pp 3059–3067, 2014) are optimal minibatch methods. In experiments, we compare optimal minibatch methods with several competitors on L1-and L2-regularized logistic regression problems and observe that iteration complexities of optimal minibatch methods linearly decrease as minibatch sizes increase up to reasonable minibatch sizes and finally attain the best iteration complexities. This confirms the computational efficiency of optimal minibatch methods suggested by the theory.

DOI： 10.1007/s10115-021-01593-1

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85110769960&origin=inward

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

担当区分：筆頭著者   記述言語：日本語   掲載種別：記事・総説・解説・論説等（学術雑誌）

担当区分：筆頭著者   記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

担当区分：筆頭著者   記述言語：日本語   掲載種別：記事・総説・解説・論説等（商業誌、新聞、ウェブメディア）

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Data cleansing is a typical approach used to improve the accuracy of machine learning models, which, however, requires extensive domain knowledge to identify the influential instances that affect the models. In this paper, we propose an algorithm that can identify influential instances without using any domain knowledge. The proposed algorithm automatically cleans the data, which does not require any of the users' knowledge. Hence, even non-experts can improve the models. The existing methods require the loss function to be convex and an optimal model to be obtained, which is not always the case in modern machine learning. To overcome these limitations, we propose a novel approach specifically designed for the models trained with stochastic gradient descent (SGD). The proposed method infers the influential instances by retracing the steps of the SGD while incorporating intermediate models computed in each step. Through experiments, we demonstrate that the proposed method can accurately infer the influential instances. Moreover, we used MNIST and CIFAR10 to show that the models can be effectively improved by removing the influential instances suggested by the proposed method.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85086236097&origin=inward

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

The minibatching technique has been extensively adopted to facilitate stochastic first-order methods because of their computational efficiency in parallel computing for large-scale machine learning and data mining. However, the optimal minibatch size determination for accelerated stochastic gradient methods is not completely understood. Actually, there appears trade-off between the iteration complexity and the total computational complexity; that is, the number of iterations (minibatch queries) can be decreased by increasing the minibatch size, but too large minibatch size would result in an unnecessarily large total computational cost. In this study, we give a sharp characterization of the minimax optimal minibatch size to achieve the optimal iteration complexity by providing a reachable lower bound for minimizing finite sum of convex functions and, surprisingly, show that the optimal method with the minimax optimal minibatch size can achieve both of the optimal iteration complexity and the optimal total computational complexity simultaneously. Finally, this feature is verified experimentally.

DOI： 10.1109/ICDM.2019.00059

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85078881697&origin=inward

記述言語：日本語   掲載種別：記事・総説・解説・論説等（学術雑誌）

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

We consider stochastic gradient descent and its averaging variant for binary classification problems in a reproducing kernel Hilbert space. In traditional analysis using a consistency property of loss functions, it is known that the expected classification error converges more slowly than the expected risk even when assuming a low-noise condition on the conditional label probabilities. Consequently, the resulting rate is sublinear. Therefore, it is important to consider whether much faster convergence of the expected classification error can be achieved. In recent research, an exponential convergence rate for stochastic gradient descent was shown under a strong low-noise condition but provided theoretical analysis was limited to the squared loss function, which is somewhat inadequate for binary classification tasks. In this paper, we show an exponential convergence of the expected classification error in the final phase of the stochastic gradient descent for a wide class of differentiable convex loss functions under similar assumptions. As for the averaged stochastic gradient descent, we show that the same convergence rate holds from the early phase of training. In experiments, we verify our analyses on the L2-regularized logistic regression.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85085008325&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

We propose a new technique that boosts the convergence of training generative adversarial networks. Generally, the rate of training deep models reduces severely after multiple iterations. A key reason for this phenomenon is that a deep network is expressed using a highly non-convex finite-dimensional model, and thus the parameter gets stuck in a local optimum. Because of this, methods often suffer not only from degeneration of the convergence speed but also from limitations in the representational power of the trained network. To overcome this issue, we propose an additional layer called the gradient layer to seek a descent direction in an infinite-dimensional space. Because the layer is constructed in the infinite-dimensional space, we are not restricted by the specific model structure of finite-dimensional models. As a result, we can get out of the local optima in finite-dimensional models and move towards the global optimal function more directly. In this paper, this phenomenon is explained from the functional gradient method perspective of the gradient layer. Interestingly, the optimization procedure using the gradient layer naturally constructs the deep structure of the network. Moreover, we demonstrate that this procedure can be regarded as a discretization method of the gradient flow that naturally reduces the objective function. Finally, the method is tested using several numerical experiments, which show its fast convergence.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85067830953&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Residual Networks (ResNets) have become stateof-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well. One attractive viewpoint on ResNet is that it is optimizing the risk in a functional space by combining an ensemble of effective features. In this paper, we adopt this viewpoint to construct a new gradient boosting method, which is known to be very powerful in data analysis. To do so, we formalize the gradient boosting perspective of ResNet mathematically using the notion of functional gradients and propose a new method called ResFGB for classification tasks by leveraging ResNet perception. Two types of generalization guarantees are provided from the optimization perspective: one is the margin bound and the other is the expected risk bound by the sample-splitting technique. Experimental results show superior performance of the proposed method over state-of-the-art methods such as LightGBM.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85057224358&origin=inward

記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Difference of convex functions (DC) programming is an important approach to nonconvex optimization problems because these structures can be encountered in several fields. Effective optimization methods, called DC algorithms, have been developed in deterministic optimization literature. In machine learning, a lot of important learning problems such as the Boltzmann machines (BMs) can be formulated as DC programming. However, there is no DC-like algorithm guaranteed by convergence rate analysis for stochastic problems that are more suitable settings for machine learning tasks. In this paper, we propose a stochastic variant of DC algorithm and give computational complexities to converge to a stationary point under several situations. Moreover, we show our method includes expectation-maximization (EM) and Monte Carlo EM (MCEM) algorithm as special cases on training BMs. In other words, we extend EM/MCEM algorithm to more effective methods from DC viewpoint with theoretical convergence guarantees. Experimental results indicate that our method performs well for training binary restricted Boltzmann machines and deep Boltzmann machines without pre-training.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85083937970&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（研究会，シンポジウム資料等）

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（研究会，シンポジウム資料等）

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. An important feature of the method is that it can be directly applied to general convex and optimal strongly convex problems that is a weaker condition than strong convexity. We show that our method achieves a better overall complexity for the general convex problems and linear convergence for optimal strongly convex problems. Moreover we prove the fast iteration complexity of our method. Our experiments show the effectiveness of our method.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85067582313&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

担当区分：筆頭著者,　最終著者,　責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Proximal gradient descent (PGD) and stochastic proximal gradient descent (SPGD) are popular methods for solving regularized risk minimization problems in machine learning and statistics. In this paper, we propose and analyze an accelerated variant of these methods in the mini-batch setting. This method incorporates two acceleration techniques: one is Nesterov's acceleration method, and the other is a variance reduction for the stochastic gradient. Accelerated proximal gradient descent (APG) and proximal stochastic variance reduction gradient (Prox-SVRG) are in a trade-off relationship. We show that our method, with the appropriate mini-batch size, achieves lower overall complexity than both APG and Prox-SVRG.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=84937865260&origin=inward

記述言語：英語   掲載種別：研究論文（研究会，シンポジウム資料等）

担当区分：筆頭著者   記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

DOI： https://doi.org/10.20551/jscssymo.26.0_123

記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

An automatic scheduling system was developed using mathematical programming for LNG storage operations, for the stable conduct of LNG tank operations. This system can be used to derive daily schedules for LNG storage operations for the subsequent month in ∼ 15 min. The system can also be used to make daily schedules for LNG storage operations for an entire year. This automated system can be used to support swift decisions on the ability to receive LNG tankers when changes are made to LNG receiving schedules; capital investment decisions using the results of LNG storage operations simulations for preparing facilities plans; and decisions on participation and purchasing in new LNG projects using the results of LNG storage operations simulations. This is an abstract of a paper presented at the 25th World Gas Conference (Kuala Lumpur, Malaysia 6/4-8/2012).

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=84904414057&origin=inward

担当区分：筆頭著者,　最終著者,　責任著者   記述言語：英語   掲載種別：研究論文（学術雑誌）

Let f be a meromorphic mapping from Cn into a compact complex manifold M. In this paper we give some estimates of the growth of the proximity function mf(r,D) of f with respect to a divisor D. J.E. Littlewood [2] (cf. Hayman [1]) proved that every non-constant meromorphic function g on the complex plane C satisfies lim supr→∞ m g(r,a)/logT(r,g) ≤ 1/2 for almost all point a of the Riemann sphere. We extend this result to the case of a meromorphic mapping f: C n → M and a linear system P(E) on M. The main result is an estimate of the following type: For almost all divisor D € P(E), lim supr→∞ mf(r,D)-mf(r,I B(E))/logTfE(r,HE) ≤ 1/2.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=79954452156&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（学術雑誌）

開催期間： 2022年11月22日   記述言語：日本語  

開催期間： 2022年11月21日   記述言語：日本語  

開催期間： 2022年11月17日   記述言語：英語  

開催期間： 2022年06月27日   記述言語：日本語  

開催期間： 2021年11月10日   記述言語：日本語  

開催期間： 2021年09月08日   記述言語：日本語  

開催期間： 2021年09月08日   記述言語：日本語  

開催期間： 2021年06月25日   記述言語：英語  

開催期間： 2021年06月25日   記述言語：英語  

開催期間： 2021年03月13日   記述言語：日本語  

開催期間： 2020年12月11日   記述言語：英語  

開催期間： 2020年11月25日   記述言語：日本語  

開催期間： 2020年11月24日   記述言語：日本語  

開催期間： 2020年09月12日   記述言語：日本語  

開催期間： 2019年12月26日   記述言語：英語  

開催期間： 2019年11月21日   記述言語：日本語  

開催期間： 2019年11月21日   記述言語：日本語  

開催期間： 2019年11月20日   記述言語：日本語  

開催期間： 2019年09月12日   記述言語：日本語  

開催期間： 2019年09月11日   記述言語：日本語  

開催期間： 2019年09月05日   記述言語：日本語  

開催期間： 2019年08月13日   記述言語：英語  

開催期間： 2018年11月06日   記述言語：日本語  

開催期間： 2018年09月13日   記述言語：日本語  

開催期間： 2017年11月10日   記述言語：日本語  

開催期間： 2017年09月06日   記述言語：日本語  

開催期間： 2017年03月07日   記述言語：日本語  

開催期間： 2016年11月17日   記述言語：日本語  

開催期間： 2016年09月06日   記述言語：日本語  

開催期間： 2013年03月15日   記述言語：日本語  

開催期間： 2012年11月02日   記述言語：日本語  

開催期間： 2012年10月17日   記述言語：英語  

開催期間： 2012年03月27日   記述言語：日本語  

開催期間： 2008年12月16日   記述言語：日本語  

開催期間： 2019年09月07日   発表言語：日本語   講演種別：招待講演  

開催期間： 2019年06月05日   発表言語：英語   講演種別：招待講演  

開催期間： 2018年11月15日   発表言語：日本語   講演種別：招待講演  

開催期間： 2018年08月12日   発表言語：日本語   講演種別：招待講演  

開催期間： 2017年10月12日   発表言語：日本語   講演種別：招待講演  

受賞国：日本国

受賞国：日本国

受賞国：日本国