{"id":9276,"date":"2021-05-11T17:22:24","date_gmt":"2021-05-11T17:22:24","guid":{"rendered":"http:\/\/algotrading101.com\/learn\/?p=9276"},"modified":"2022-07-16T17:16:23","modified_gmt":"2022-07-16T17:16:23","slug":"cluster-analysis-guide","status":"publish","type":"post","link":"https:\/\/algotrading101.com\/learn\/cluster-analysis-guide\/","title":{"rendered":"Cluster Analysis – Machine Learning for Pairs Trading"},"content":{"rendered":"
<\/div>

<\/path><\/svg><\/i> \"Loading\"<\/p>

<\/div>\n

Table of contents:<\/h3>\n\n\n\n
  1. What is Cluster Analysis?<\/a><\/li>
  2. Is Cluster Analysis an Unsupervised Machine Learning task?<\/a><\/li>
  3. How can Cluster Analysis be used for Finance?<\/a><\/li>
  4. What is the Pairs Trading strategy? <\/a><\/li>
  5. Does Pair Trading succeed?<\/a><\/li>
  6. What are the main steps of a Machine Learning project?<\/a><\/li>
  7. Where to find stock data and how to load it?<\/a><\/li>
  8. How to explore stock data?<\/a><\/li>
  9. How to prepare stock data for clustering?<\/a><\/li>
  10. How to pick a good machine learning model?<\/a><\/li>
  11. What is k-Means Clustering?<\/a><\/li>
  12. How to use k-Means Clustering for Pairs Trading?<\/a><\/li>
  13. What is Hierarchical Clustering?<\/a><\/li>
  14. How to use Hierarchical Clustering for Pairs Trading?<\/a><\/li>
  15. What is Affinity Propagation Clustering?<\/a><\/li>
  16. How to use Affinity Propagation Clustering for Pairs Trading?<\/a><\/li>
  17. How to evaluate and compare clustering models?<\/a><\/li>
  18. How to extract the trading pairs?<\/a><\/li>
  19. How to efficiently present your findings?<\/a><\/li>
  20. What are the 3 Common Machine Learning Analysis\/Testing Mistakes?<\/a><\/li>
  21. Full code<\/a><\/li><\/ol>\n\n\n\n\n\n\n\n

    What is Cluster Analysis?<\/h2>\n\n\n\n

    Cluster Analysis is a group of methods that are used to classify phenomena into relative groups known as clusters. <\/p>\n\n\n\n

    Cluster Analysis doesn’t have any prior information about the groups our features inhabit.<\/p>\n\n\n\n

    \"cluster
    The result of a cluster analysis shown as the coloring of the squares into three clusters.<\/figcaption><\/figure>\n\n\n\n\n\n\n\n

    Is Cluster Analysis an Unsupervised Machine Learning task?<\/h2>\n\n\n\n

    Yes. In Unsupervised ML we feed inputs but there aren\u2019t any target outputs. This means that we don\u2019t tell the algorithm what to do and that it needs to figure out some sort of dependence or underlying logic of what to do.<\/p>\n\n\n\n

    For example, imagine a website where people posted pictures of dogs and cats. We want the said website to classify those pictures into two categories \u201cCats\u201d and \u201cDogs\u201d without us needing to label each picture.<\/p>\n\n\n\n

    Rather than teaching the clustering model what a cat or dog is, we will just say to the algorithm to group the pictures into two groups based on visual similarity. We will then get the result of two unlabeled groups:<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n\n\n\n\n

    How can Cluster Analysis be used for Finance?<\/h2>\n\n\n\n

    When it comes to Finance, Cluster Analysis can easily spot the underlying logic of our dataset without us needing to bang our head trying to figure it out for ourselves.<\/p>\n\n\n\n

    For example, imagine having the financial data for 200 stocks, and that you want the algorithm to divide them into groups. The algorithm gives us 4 groups (clusters) that we need to examine.<\/p>\n\n\n\n

    After examination, we conclude that the groups are: \u201cBestselling\u201d, \u201cSelling but mediocre\u201d, \u201cWorst selling\u201d, and \u201cStagnating\u201d. <\/p>\n\n\n\n

    In this article, we’ll explore how Cluster Analysis can help us in creating a Paris Trading Strategy. Let us first remind ourselves what a Pairs Trading strategy is.<\/p>\n\n\n\n\n\n\n\n

    What is the Pairs Trading strategy?<\/h2>\n\n\n\n

    Pairs trading is a strategy in which a trader buys one asset while shorting another. The main premise of the trade is that when the two pairs diverge, they will likely converge again resulting in profit for the trader.<\/p>\n\n\n\n

    A visual representation of this strategy might help you in understanding it better:<\/p>\n\n\n\n

    \"Pairs
    Pairs Trading<\/figcaption><\/figure>\n\n\n\n\n\n\n\n

    Does Pair Trading succeed?<\/h2>\n\n\n\n

    Pair Trading will work if you choose the right assets to form a pair. Moreover, you should add more parts to your pairs trading strategy like utilizing more pairs or coupling it with sentiment analysis.<\/p>\n\n\n\n

    As the fundamental building block of this strategy are the pairs we use, we don’t want to pick unreasonable ones. Finding them by hand might be too time-consuming and you might miss out on the underdogs.<\/p>\n\n\n\n

    This is where Machine Learning (ML) comes into play. In this article, we will go step-by-step through the pace of solving our problem with ML. We will define the problem as follows:<\/p>\n\n\n\n

    What trading assets work the best together to form a trading pair for the Pairs Trading strategy?<\/strong><\/p>\n\n\n\n

    Let’s begin!<\/p>\n\n\n\n

    \n\n\n\n

    Note that this article explores machine learning statistic methods to find assets that moved similarly historically. Pairs trading has been around for a long time and this strategy is common place among hedge funds and traders. <\/em><\/p>\n\n\n\n

    To succeed with pairs trading, you need market knowledge in addition to the statistical tools that you learnt here.<\/em><\/p>\n\n\n\n

    For more information about implementing pairs trading in real-life, check out the following article: https:\/\/algotrading101.com\/learn\/pairs-trading-guide\/<\/a><\/em><\/p>\n\n\n\n\n\n\n\n

    What are the main steps of a Machine Learning project?<\/h2>\n\n\n\n

    Before tackling the main problem that we defined, we need to remind ourselves of what the main steps of an ML project are. These steps can often be overlooked by novice practitioners, so be sure to have them in mind:<\/p>\n\n\n\n

    1. Define the Problem<\/strong> – be sure to deeply understand the problem you are trying to solve and elaborate it in a concise and understandable way.
      <\/li>
    2. Research the Problem<\/strong> – thoroughly research your problem by exploring if there are any proposed solutions, read papers, communicate with experts, etc.
      <\/li>
    3. Obtain the Data<\/strong> – <\/em>you can’t expect your model to perform without obtaining quality data that fits the problem. Remember, garbage in = garbage out.
      <\/li>
    4. Prepare the Data<\/strong> – things aren’t perfect in life as so goes for your data. It might have missing values, wrongly imputed values, undesirable values, and much more. Be sure to clean it!
      <\/li>
    5. Pick the right Model<\/strong> – picking the right model is one of the most important steps when trying to solve the problem. Think of how each model works and if it can provide a reasonable solution.

      Is your task to predict a value, cluster, or classify? Should you use supervised, unsupervised or reinforcement learning?
      <\/li>
    6. Evaluate the Model<\/strong> – evaluating your model is a no-brainer. You simply need to see how it performs on the main performance metrics like precision and recall.
      <\/li>
    7. Tune the Model<\/strong> – depending on the model you choose and its performance you might want to optimize it by tweaking it with its structure and hyperparameters.
      <\/li>
    8. Present your findings<\/strong> – after the model is tuned, you are ready to deploy it and present your findings. This is where your communication skills come into play so be sure to practice them.<\/li><\/ol>\n\n\n\n

      Have in mind that we covered the main steps, there are even more sub-steps and global steps that might and should arise. Be sure to think both wide and deep.<\/p>\n\n\n\n\n\n\n\n

      Where to find stock data and how to load it?<\/h2>\n\n\n\n

      Stock data can be easily obtained by using financial data providers like Quandl, Yahoo Finance, dxFeed, Bloomberg, or by utilizing online brokers like Interactive Brokers, Fidelity Investments, and more.<\/p>\n\n\n\n

      For this article, we will obtain 3 years’ worth of data for the S&P 500 stock by using Yahoo Finance. For more info on Yahoo Finance check out this article:<\/p>\n\n\n\n

      \n
      Yahoo Finance API – A Complete Guide<\/a><\/blockquote>