prediction - How can I go about applying machine learning algorithms to stock markets? - Quantitative Finance Stack Exchange

This post will detail what I did to make approx. My trading was mostly in Russel and DAX futures contracts. The key to my success, I believe, was not in a sophisticated financial equation but rather in the overall algorithm design which tied together many simple components and used machine learning to optimize for maximum profitability.

CourseTalk, a review site for MOOCs. First, I just want to demonstrate that my success was not simply the result of luck. My program made trades per day half long, half short and never got into positions of more than a few contracts at a time. This meant the random luck from any one particular trade averaged out pretty fast. Note this excludes the last 7 months because - as the figures stopped going up - I lost my motivation to enter them.

Being successful meant being fast, being disciplined, and having a good intuitive pattern recognition abilities.

Over the next five years I would launch two startups, picking up some programming skills along the way. With money running low from the sale of my first startup, trading offered hopes of some quick cash while I figured out my next move.

After getting my feet wet with the API I soon had bigger aspirations: I wanted to teach the computer to trade for me. The API provided both a stream of market data and an easy way to send orders to the exchange - all I had to do was create the logic in the middle. Below is a screenshot of a T4 trading window. What was cool is that when I got my program working I was able to watch the computer trade on this exact same interface.

CiteSeerX — Stock Market Forecasting Using Machine Learning Algorithms

Watching real orders popping in and out by themselves with my real money was both thrilling and scary. To accomplish this I needed to build a trading simulation framework that would - as accurately as possible - simulate live trading. While trading in live mode required processing market updates streamed through the API, simulation mode required reading market updates from a data file.

To collect this data I setup the first version of my program to simply connect to the API and record market updates with timestamps. I ended up using 4 weeks worth of recent market data to train and test my system on. With a basic framework in place I still had the task of figuring out how to make a profitable trading system.

Perhaps an obvious component of any trading system is being able to predict where prices will move. And mine was no exception. I defined the current price as the average of the inside bid and inside offer and I set the goal of predicting where the price would be in the next 10 seconds.

My algorithm would need to come up with this prediction moment-by-moment throughout the trading day. I created a handful of indicators that proved to have a meaningful ability to predict short term price movements. Each indicator produced a number that was either positive or negative.

An indicator was useful if more often than not a positive number corresponded with the market going up and a negative number corresponded with the market going down. My system allowed me to quickly determine how much predictive ability any indicator had so I was able to experiment with a lot of different indicators to see what worked.

Many of the indicators had variables in the formulas that produced them and I was able to find the optimal values for those variables by doing side by side comparisons of results achieved with varying values. The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities.

I needed to know exactly how much price movement was predicted by each possible value of each indicator. I needed a formula that would convert an indicator value to a price prediction.

To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel.

As you can see the expected price change increases as the indicator value increases. Based on a graph such as this I was able to make a formula to leading forex trading companies the curve. Note that not all the indicator curves had the same shape.

Also note the buckets were logarithmically distributed so as to spread the forex software analysis points out evenly. Finally note that negative indicator values and their corresponding downward price predictions were flipped and combined with the positive values.

My algorithm treated up and down exactly the same. An important thing to consider was that each indicator was not entirely independent. The key was to figure out the additional predictive value that each indicator had beyond what was already predicted. You might think that with this edge on the market I was golden. So I had a framework that allowed me to backtest and optimize indicators. But I had to go beyond this - I needed a framework that would allow me to backtest and optimize a full trading system; one where I was sending orders and getting in positions.

This would be trickier and in some ways impossible to model exactly but I did as best as I could. Here is binary option haram ebook some of the issues I had to deal with:. To refine my order execution simulation what I did was take my log files from live trading through the API and compare them to log s&p 500 volume leaders produced by simulated trading from the exact same time period.

I was able to get my simulation to the point that it was pretty accurate and virtual stock trading bse the parts that were impossible to model exactly I made sure to scottrade options first platform least produce outcomes that were statistically similar in the metrics I thought were important.

But how would my system know when and where to buy and air china shanghai stock exchange The price move predictions were a starting point but not the whole story. What I did was create a scoring system for each of 5 price levels on the bid and offer. These included one level above the inside bid for a buy order and one binomial tree put option american below the inside offer for a sell order.

Based on this it was not uncommon that my system would flash a bid in the market then immediately cancel it. The price move prediction alone was not adequate because it did not account for the fact that when placing a bid I was not automatically filled - I only got filled if someone sold to me there.

The reality was that the mere fact of someone amazon binary options how to trade signals to me at a asian stock market indices live price changed the statistical odds of the trade. The variables used in this step were all subject to optimization. When trading as humans we often have powerful emotions and biases that can lead to less than optimal decisions.

Clearly I did not want to codify these biases. Stock market forecasting using machine learning algorithms are some factors my system ignored:. Since my algorithm made decisions the same way regardless of where it entered a trade or if it was currently long or short it did occasionally sit in and take some large losing trades in addition to some large winning mannatechs sugar-coated moneymaker. To manage risk I enforced a maximum position size of 2 contracts at a time, occasionally bumped up on high volume days.

I also had a maximum daily loss limit to safeguard against any unexpected market german bund futures trading or a bug in my software.

These limits were enforced in my code but also in the forex trading for maximum profit download through my broker. As it happened I never encountered any significant problems. From the moment I started working on my program it took me about 6 months before i got it to the point of profitability and begun running it live. Although to be fair a significant amount of time was learning a new programming language.

As I worked to improve the program I saw increased profits for each of the next four months. Each week I would retrain my system based on the previous 4 weeks worth of data. I found this struck the right balance between capturing recent market behavioral trends and insuring my algorithm had enough data to establish meaningful patterns. As the training began taking more and more time I split it out so that it could be performed by 8 virtual machines using amazon EC2. The results were then coalesced on my local machine.

The high point of my trading was October when I made almost k. After this I continued to spend the next four months trying to improve my program despite decreased profit each month. With the frustration of not being able to make improvements and not having a sense of growth I began thinking about a new direction. I had some new startup ideas I wanted to work on so I never followed up. UPDATE - I posted this on Hacker News and it has gotten a lot of attention.

I just want to say that I do not advocate anyone trying to do something like this themselves now. You would need a team of really smart people with a range of experiences to have any hope of competing. Even when I was doing this I believe it was very rare for individuals to achieve success though I had heard of others.

Hi, I'm Jesse, founder of Thinklab. I live and play in San Francisco. You've found my home on the web. Minimal Theme designed by Artur Kim.

Home Archive Mobile RSS. CourseTalk, a review site for MOOCs First, I just want to demonstrate that my success was not simply the result of luck. Combining indicators for a single prediction An important thing to consider was that each indicator was not entirely independent. Why predicting prices is not enough You might think that with this edge on the market I was golden. The following factors make creating a profitable system difficult: With each trade I had to pay commissions to both my broker and the exchange.

Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge. Seeing an offer did not guarantee that I could buy it.

By the time my buy order got to the exchange it was very possible that that offer would have been cancelled. As a small market player there was no way I could compete on speed alone. Building a full trading simulation So I had a framework that allowed me to backtest and optimize indicators.

Here are some of the issues I had to deal with: When an order was sent to the market in simulation I had to model the lag time. The fact that my system saw an offer did not mean that it could buy it straight away.

The system would send the order, wait approximately 20 milliseconds and then only if the offer was still there was it considered as an executed trade. This was inexact because the real lag time was inconsistent and unreported. When I placed bids or offers I had to look at the trade execution stream provided by the API and use those to gauge when my order would have gotten executed against.

To do this right I had to track the position of my order in the queue. The price level scores were calculated based on the following factors: The price move prediction that we discussed earlier. The price level in question. Inner levels meant greater price move predictions were required. The number of contracts in front of my order in the queue. The number of contracts behind my order in the queue. What my program ignored When trading as humans we often have powerful emotions and biases that can lead to less than optimal decisions.

Here are some factors my system ignored: While this has some validity as part of a risk reduction strategy it really has no bearing on the future course of events in the market. Therefore my program completely ignored this information. However from my algorithms perspective there was no reason to make a distinction. If my algorithm expected a downward move selling was a good idea regardless of if it was currently long, short, or flat. The problem is when you lose you lose big.

The other effect is it makes it hard to judge if you actually have an edge on the market or are just getting lucky.

Being able to monitor and confirm that my program did in fact have an edge was an important goal.

Running the algorithm From the moment I started working on my program it took me about 6 months before i got it to the point of profitability and begun running it live. About Hi, I'm Jesse, founder of Thinklab.