Stock Trading Using Reinforcement Learning By Jonah Varon

Stock Trading Using Reinforcement Learning By Jonah Varon and Anthony Soroka CS 182: Artificial Intelligence Fall 2016 Introduction Considering the interest, there is surprisingly limited available research on reinforcement learning applied to stock trading. Hoping to either further public research or find a profitable strategy, we applied QLearning to stock trading. Abstract: We applied Q-Learning to stock trading. Specifically we attempted to place our agent in states to find profitable strategies through technical analysis and mean reversion. Our results were mixed, leading us to question the applicability of reinforcement learning in the stock market – a space that differs in significant ways from those where reinforcement learning has proven suitable. However, our multi-agent technical analysis approach highlighted specific stocks where Q-Learning proved effective for certain time periods. Results We used Quantopian to test our strategies. Quantopian is a crowdsourced investment firm which provides market data and a pythonbased development platform for algorithmic investors. Figure 1. YELP Tech-Analysis Agent Figure 2. ATML Tech-Analysis Agent Approach Q-Learning is defined by the following update function: For our project, we typically applied the below: a = {Buy, Sell} r = Stock Performance The stock market however has a few interesting characteristics that slightly alter Q-Learning: a) Future State: Due to the ability to rebalance, max Q(s’, a’) is the same regardless of Q(s, a) (presuming you are a small enough investor with minimal market impact). State Representation The state space for the stock market is vast. Hence, limiting the space to allow the agent to learn is crucial. Given such, we represented states in hopes of positioning our agent to find known trading strategies: 1) Technical Analysis: last 3 minutes of price and volume data {-1: Decreased, 0: Unchanged, 1: Increased}. # of States = 3^6 = 729 # of Actions per Year = 98, 280 2) Mean Reversion: Previous week’s stock performance {-1: Stock was in the bottom X Percentile, 1: Stock was in top X percentile} b) Fully Observable: Limiting the agent to actions of Buy and Sell, the reward from Q(s, a) is fully observable regardless of the actual action taken. We tried both approaches to updating Q(s, a): 1) Only for action taken 2) For both taken and not taken actions Conclusions Our attempts to find alpha using QLearning illustrate how competitive and efficient markets are. Additionally it highlights that reinforcement learning is perhaps not the best suited for stock trading given: a) Stochasticity of the market b) Limited amount of train data c) Agent can observe rewards without taking actions d) Agent has little impact on the environment That being said the possible methods to define a state space are limitless and should still be investigated. Moreover, our multi-agent approach proved useful in highlighting specific stocks where QLearning might be applicable. Data Quantopian provides a wealth of technical, fundamental, and event information from thousands of stocks. Results Summarized: Most of our strategies did not outperform reasonable benchmarks (S&P 500 / specific stock). However, using a multi-agent approach on the technical analysis agent, we were able to scan the market for specific stocks and specific periods where the agent was profitable. Example successful agents are illustrated above. Code Snippet. Updating Technical Analysis Agent’s State Lastly, we believe Deep Learning techniques such as LSTM might be better suited for this problem. John A. Paulson School of Engineering and Applied Sciences