End-to-End Reinforcement Learning of Dialogue Agents for Information Access 端對端加強學習對話代理資訊訪問

This paper proposes KB-InfoBot—a dialogueagent that provides users with an entity from a knowledge base (KB) byinteractively asking for its attributes. All components of the KBInfoBot aretrained in an end-to-end fashion using reinforcement learning. Goal-orienteddialogue systems typically need to interact with an external database to accessreal-world knowledge (e.g., movies playing in a city). Previous systemsachieved this by issuing a symbolic query to the database and adding retrievedresults to the dialogue state. However, such symbolic operations break thedifferentiability of the system and prevent end-to-end training of neuraldialogue agents. In this paper, we address this limitation by replacingsymbolic queries with an induced “soft” posterior distribution over the KB thatindicates which entities the user is interested in. We also provide a modifiedversion of the episodic REINFORCE algorithm, which allows the KBInfoBot toexplore and learn both the policy for selecting dialogue acts and the posteriorover the KB for retrieving the correct entities. Experimental results show thatthe end-to-end trained KB-InfoBot outperforms competitive rule-based baselines,as well as agents which are not end-to-end trainable.

本文提出KB-InfoBot - 一個對話代理，通過互動式詢問其屬性為使用者提供來自知識庫（KB）的實體。

KBInfoBot的所有組件都使用強化學習以端到端的方式進行培訓。面向目標的對話系統通常需要與外部資料庫互動以訪問現實世界的知識（例如，在城市中播放的電影）。以前的系統通過向資料庫發出符號查詢並將檢索到的結果添加到對話狀態來實現。然而，這種象徵性的操作打破了系統的可區分性，並阻止了神經對話代理的端到端訓練。在本文中，我們解決了這個限制，通過在KB上引用“軟”後驗分布來代替符號查詢來解決這個限制，這表明使用者感興趣的是哪個實體。我們還提供了一個修改版本的情景REINFORCE演算法，它允許KBInfoBot探索和學習選擇對話行為的政策和智慧財產權後驗以檢索正確的實體。

實驗結果表明，端到端培訓的KB-InfoBot勝過基於競爭規則的基準，以及不是端對端可訓練的代理。

guage. In this work, we present KB-InfoBot,a dialogue agent that identifies entities of interest to the user from aknowledge base (KB), by interactively asking for attributes of that entitywhich helps constrain the search. Such an agent finds application ininteractive search settings. Figure 1 shows a dialogue example between a usersearching for a movie and the proposed KB-InfoBot.

 

物件導向的對話系統可以協助使用者通過自然語言與他們互動來完成特定的任務，如預訂航班或搜尋資料庫。在這項工作中，我們提出KB-InfoBot，一個對話代理，通過互動式地詢問該實體的屬性來協助約束搜尋，從而從知識庫（KB）中識別使用者感興趣的實體。這樣的代理在互動式搜尋設定中尋找應用程式。圖1示出了使用者搜尋電影和所提出的KB-InfoBot之間的對話樣本。

 

 

A typical goal-oriented dialogue systemconsists of four basic components: a language understanding (LU) module foridentifying user intents and extracting associated slots (Yao et al., 2014;HakkaniTur et al., 2016; Chen et al., 2016), a dialogue ¨ state tracker whichtracks the user goal and dialogue history (Henderson et al., 2014; Henderson,2015), a dialogue policy which selects the next system action based on thecurrent state (Young et al., 2013), and a natural language generator (NLG) forconverting dialogue acts into natural language (Wen et al., 2015; Wen et al.,2016a). For successful completion of user goals, it is also necessary to equip thedialogue policy with real-world knowledge from a database. Previous end-to-endsystems achieved this by constructing a symbolic query from the current beliefstates of the agent and retrieving results from the database which match thequery (Wen et al., 2016b; Williams and Zweig, 2016; Zhao and Eskenazi, 2016).Unfortunately, such operations make the model non-differentiable, and variouscomponents in a dialogue system are usually trained separately.

一個典型的面向目標的對話系統由四個基本組成部分組成：

用於識別使用者意圖並提取相關時隙的語言理解（LU）模組，

跟蹤使用者目標和對話曆史的狀態跟蹤器（Henderson et al。，2014; Henderson，2015），

基於目前狀態選擇下一個系統動作的對話政策（Young等，2013），

和一種將對話行為轉化為自然語言的自然語言產生器（Wen et al。，2015; Wen et al。，2016a）。

為了順利完成使用者目標，還必須將對話政策與資料庫中的現實知識結合在一起。先前的端到端系統通過從代理的當前信念狀態構建符號查詢並從資料庫檢索與查詢匹配的結果來實現這一點（Wen等人，2016b; Williams和Zweig，2016; Zhao和Eskenazi， 2016）。不幸的是，這樣的操作使得模型不可區分，並且對話系統中的各種組件通常是分開訓練的。

 In our work, we replace SQL-like querieswith a probabilistic framework for inducing a posterior distribution of theuser target over KB entities. We build this distribution from the belieftracker multinomials over attribute-values and binomial probabilities of theuser not knowing the value of an attribute. The policy network receives asinput this full distribution to select its next action. In addition to makingthe model end-to-end trainable, this operation also provides a principledframework to propagate the uncertainty inherent in language understanding tothe dialogue policy making the agent robust to LU errors. Our entire model isdifferentiable, which means that in theory our system can be trained completelyend-to-end using only a reinforcement signal from the user that indicateswhether a dialogue is successful or not. However, in practice, we find thatwith random initialization the agent is unable to see any rewards if thedatabase is large; even when it does, credit assignment is tough. Hence, at thebeginning of training, we first have an imitation-learning phase (Argall etal., 2009) where both the belief tracker and policy network are trained tomimic a rule-based agent. Then, on switching to reinforcement learning, theagent is able to improve further and increase its average reward. Such abootstrapping approach has been shown effective when applying reinforcementlearning to solve hard problems, especially those with long decision horizons(Silver et al., 2016).