Abstract:
Recently, multi-hop reasoning over incomplete Knowledge Graphs (KGs) has attracted wide attention due to its desirable interpretability for downstream tasks, such as question answer and knowledge graph completion. Multi-Hop reasoning is a typical sequential decision problem, which can be formulated as a Markov decision process (MDP). Subsequently, some reinforcement learning (RL) based approaches are proposed and proven effective to train an agent for reasoning paths sequentially until reaching the target answer. However, these approaches assume that an entity/relation representation follows a one-point distribution. In fact, different entities and relations may contain different certainties. On the other hand, since REINFORCE used for updating the policy in these approaches is a biased policy gradients method, the agent is prone to be stuck in high reward paths rather than broad reasoning paths, which leads to premature and suboptimal exploitation. In this paper, we consider a Bayesian reinforcement learning paradigm to harness uncertainty into multi-hop reasoning. By incorporating uncertainty into the representation layer, the agent trained by RL has uncertainty in a region of the state space then it should be more efficient in exploring unknown or less known part of the KG. In our approach, we build a Bayesian Q-learning architecture as a state-action value function for estimating the expected long-term reward. As initialized by Gaussian prior or pre-trained prior distribution, the representation layer drives uncertainty that allows regularizing the training. We conducted extensive experiments on multiple KGs. Experimental results show a superior performance than other baselines, especially significant improvements on the automated extracted KG.
