Student research opportunities

Generic Reinforcement Learning Agents (GRLA)

Project Code: CECS_122

This project is available at the following levels:
CS single semester, Engn4200, Engn R&D, Honours, Summer Scholar, Masters, PhD

Keywords:

reinforcement learning; feature selection; explore-exploit dilemma; information & complexity; rational agents.

Supervisors:

Dr Peter Sunehag
Professor Marcus Hutter

Outline:

Agent applications are ubiquitous in commerce and industry, and the sophistication, complexity, and importance of these applications is increasing rapidly; they include speech recognition systems, vision systems, search engines, planetary explorers, auto-pilots, spam filters, and robots [RN03]. Existing agent technology can be improved by developing systems that can automatically acquire during deployment much of the knowledge that would otherwise be required to be built in by agent designers. This greatly reduces the effort required for agent construction, and results in agents that are more adaptive and operate successfully in a wide variety of environments [LH07].

Goals of this project

Technically, the project is about a recent general approach to learning that bridges the gap between theory and practice in reinforcement learning (RL). General-purpose, intelligent, learning agents cycle through sequences of observations, actions, and rewards that are complex, uncertain, unknown, and non-Markovian [RN03]. On the other hand, RL is well-developed for small finite state Markov decision processes (MDPs) [SB98]. Extracting the right state representations out of bare observations, that is, reducing the general agent setup to the MDP framework, is an art that involves significant effort by designers. The project is to investigate (by simulations or theoretical) recent models [Hut09] that automate the reduction process and thereby significantly expand the scope of many existing RL algorithms and the agents that employ them.

Requirements/Prerequisites

• background in Artificial Intelligence and Machine Learning
  
• good programming skills
  
• performing (computer) experiments and analyzing results
  
• good math skills; linear algebra at the very minimum
  
• mastering elementary probability calculus

Student Gain

• getting acquainted with state-of-the art RL algorithms
  
• improving your math skills: linear algebra, statistics, probability, and information theory

Background Literature

• [RN03] S.J. Russell and P. Norvig. Artificial Intelligence: A Modern Approach Prentice-Hall, Englewood Cliffs, NJ, 2nd edition, 2003.
  
• [SB98] R.S. Sutton and A.G. Barto. Reinforcement Learning: An Introduction MIT Press, Cambridge, MA, 1998.
  
• [LH07] S. Legg and M. Hutter. Universal intelligence: A definition of machine intelligence. Minds & Machines, 17(4):391--444, 2007.
  
• [Hut09] M. Hutter. Feature reinforcement learning: Part I: Unstructured MDPs. Journal of Artificial General Intelligence, 1:3--24, 2009.

Contact: