担当区分：筆頭著者   記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

<p>強化学習は様々な応用事例がある有用な手法であるが，一般に大量のデータが必要で，このことは改善すべき課題の一つである．本研究では，必ずしも質の高くない少量の模倣対象データを活用し，強化学習のサンプル効率を改善する手法Cross Bisimulation based Implicit Imitation Learning (CBI2L) を提案する．本研究では，模倣対象となるmentorと強化学習の実行者であるobserverのエージェントそれぞれのマルコフ決定過程に対し，その間での累積報酬差に関する擬距離cross bisimulation metricを定義する．そして，不動点定理によるcross bisimulation metricの一意存在性，mentorとobserverの累積報酬期待値の差とcross bisimulation metricの関係性について理論的に分析し，observerの学習にmentorの累積報酬を用いるCBI2Lの妥当性を示す．さらにはCBI2Lをsoft actor-critic (SAC)に組み込み，PointMaze環境でSACと比べてサンプル効率が改善することを実際に示す．</p>

DOI： 10.11517/pjsai.jsai2026.0_5m1gs2b02

CiNii Research

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

The cumulative regret of Q-learning, one of the representative methods in reinforcement learning, has been analyzed; however, most of them assume that the agent learns from scratch. If the agent has knowledge about the learning domain, its learning will be more efficient, as suggested in transfer learning research. Therefore, we modify an existing Q-learning method and propose Biased Exploration Q-learning (BEQ), which assumes that the agent can acquire domain knowledge in advance. We analyze BEQ theoretically and clarify the conditions under which pruning of suboptimal actions is possible and show that the upper bound of the regret of BEQ is significantly improved compared to the Q-learning. Furthermore, our experiments show that BEQ outperforms Potential-Based Reward Shaping (PBRS), a common method for introducing domain knowledge. Moreover, we apply BEQ in deep reinforcement learning and demonstrate its effectiveness in Atari games with simple domain knowledge.

DOI： 10.5220/0014246600004052

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=105041701330&origin=inward

担当区分：筆頭著者,　責任著者   記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

担当区分：責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

担当区分：責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

担当区分：責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

担当区分：責任著者   記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

Kyutacar

担当区分：筆頭著者,　責任著者   記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

Kyutacar

記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

Kyutacar

記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

Kyutacar

記述言語：日本語   掲載種別：研究論文（研究会，シンポジウム資料等）

Kyutacar

担当区分：最終著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85146871335&origin=inward

担当区分：筆頭著者   記述言語：英語   掲載種別：研究論文（学術雑誌）

Meta-reinforcement learning (RL) addresses the problem of sample inefficiency in deep RL by using experience obtained in past tasks for solving a new task. However, most existing meta-RL methods require partially or fully on-policy data, which hinders the improvement of sample efficiency. To alleviate this problem, we propose a novel off-policy meta-RL method, embedding learning and uncertainty evaluation (ELUE). An ELUE agent is characterized by the learning of what we call a task embedding space, an embedding space for representing the features of tasks. The agent learns a belief model over the task embedding space and trains a belief-conditional policy and Q-function. The belief model is designed to be agnostic to the order in which task information is obtained, thereby reducing the difficulty of task embedding learning. For a new task, the ELUE agent collects data by the pretrained policy, and updates its belief on the basis of the belief model. Thanks to the belief update, the performance of the agent improves with a small amount of data. In addition, the agent updates the parameters of its policy and Q-function so that it can adjust the pretrained relationships when there are enough data. We demonstrate that ELUE outperforms state-of-the-art meta RL methods through experiments on meta-RL benchmarks.

DOI： 10.1109/ACCESS.2022.3170582

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85129665822&origin=inward

担当区分：責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Model-based meta-reinforcement learning (RL) methods have recently been shown to be a promising approach to improving the sample efficiency of RL in multi-task settings. However, the theoretical understanding of those methods is yet to be established, and there is currently no theoretical guarantee of their performance in a real-world environment. In this paper, we analyze the performance guarantee of model-based meta-RL methods by extending the theorems proposed by Janner et al. (2019). On the basis of our theoretical results, we propose Meta-Model-Based Meta-Policy Optimization (M3PO), a model-based meta-RL method with a performance guarantee. We demonstrate that M3PO outperforms existing meta-RL methods in continuous-control benchmarks.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85137101456&origin=inward

担当区分：責任著者   記述言語：英語   掲載種別：研究論文（国際会議プロシーディングス）

Options are generally learned by using an inaccurate environment model (or simulator), which contains uncertain model parameters. While there are several methods to learn options that are robust against the uncertainty of model parameters, these methods only consider either the worst case or the average (ordinary) case for learning options. This limited consideration of the cases often produces options that do not work well in the unconsidered case. In this paper, we propose a conditional value at risk (CVaR)-based method to learn options that work well in both the average and worst cases. We extend the CVaR-based policy gradient method proposed by Chow and Ghavamzadeh (2014) to deal with robust Markov decision processes and then apply the extended method to learning robust options. We conduct experiments to evaluate our method in multi-joint robot control tasks (HopperIceBlock, Half-Cheetah, and Walker2D). Experimental results show that our method produces options that 1) give better worst-case performance than the options learned only to minimize the average-case loss, and 2) give better average-case performance than the options learned only to minimize the worst-case loss.

Scopus

その他リンク： https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85090171999&origin=inward