{"id":19429,"date":"2022-12-19T22:21:08","date_gmt":"2022-12-19T22:21:08","guid":{"rendered":"https:\/\/algotrading101.com\/learn\/?p=19429"},"modified":"2025-02-04T21:52:07","modified_gmt":"2025-02-04T21:52:07","slug":"conformal-prediction-guide","status":"publish","type":"post","link":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/","title":{"rendered":"Conformal Prediction &#8211; A Practical Guide with MAPIE"},"content":{"rendered":"<div class=\"pvc_clear\"><\/div><p id=\"pvc_stats_19429\" class=\"pvc_stats total_only  \" data-element-id=\"19429\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p><div class=\"pvc_clear\"><\/div><div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img fetchpriority=\"high\" decoding=\"async\" width=\"500\" height=\"500\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction.webp\" alt=\"\" class=\"wp-image-19282\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction.webp 500w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction-300x300.webp 300w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction-150x150.webp 150w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">Table of contents:<\/h2>\n\n\n\n<ol>\n<li><a href=\"#what-is-conformal-prediction\">What is Conformal Prediction?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-use\">What is Conformal Prediction used for?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-pros\">Why should I use Conformal Prediction?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-cons\">Why shouldn\u2019t I use Conformal Prediction?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-finance\">How can Conformal Prediction be used in Finance?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-algorithmic-trading\">How can Conformal Prediction be used in Algorithmic Trading?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-alternatives\">What are some Conformal Prediction alternatives?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-understand\">Understanding Conformal Prediction<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-mapie\">What is MAPIE?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-start\">How to get started with Conformal Prediction?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-mapie-python\">How to apply Conformal Prediction with MAPIE in Python?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-classification-mapie\">How to apply Conformal Prediction for Classification with MAPIE?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-regression-mapie\">How to apply Conformal Prediction for Regression with MAPIE?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-forecasting-mapie\">How to perform forecasting with Conformal Prediction in Python?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-learn-more\">Where can I learn more about Conformal Prediction?<\/a><\/li>\n\n\n\n<li><a href=\"#conformal-prediction-full-code\">Full code<\/a><\/li>\n<\/ol>\n\n\n\n<a name=\"what-is-conformal-prediction\">\n\n\n\n<h2 class=\"wp-block-heading\">What is Conformal Prediction?<\/h2>\n\n\n\n<p>Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model.<\/p>\n\n\n\n<a name=\"conformal-prediction-use\">\n\n\n\n<h2 class=\"wp-block-heading\">What is Conformal Prediction used for?<\/h2>\n\n\n\n<p>Conformal Prediction is often used for calibrating one&#8217;s machine learning models, estimating their uncertainty, comparing models, increasing the practical performance of the model, and more.<\/p>\n\n\n\n<a name=\"conformal-prediction-pros\">\n\n\n\n<h2 class=\"wp-block-heading\">Why should I use Conformal Prediction?<\/h2>\n\n\n\n<ul>\n<li>Conformal Prediction is not too computationally expensive<\/li>\n\n\n\n<li>It is easy to implement<\/li>\n\n\n\n<li>On average, it performs better than other calibration methods<\/li>\n\n\n\n<li>Can be applied to any machine learning model or neural network<\/li>\n\n\n\n<li>Can be used in a wide variety of tasks such as regression, classification, outlier detection, and more<\/li>\n\n\n\n<li>Doesn&#8217;t depend on the distribution of the data<\/li>\n\n\n\n<li>Tackles uncertainty in predictions<\/li>\n<\/ul>\n\n\n\n<a name=\"conformal-prediction-cons\">\n\n\n\n<h2 class=\"wp-block-heading\">Why shouldn&#8217;t I use Conformal Prediction?<\/h2>\n\n\n\n<ul>\n<li>Conformal Prediction is a fairly new approach<\/li>\n\n\n\n<li>There aren&#8217;t many out-of-the-box libraries that support it<\/li>\n\n\n\n<li>You &#8220;ditch&#8221; the standard way of viewing and interpreting your results (e.g. predictions sets instead of point predictions)<\/li>\n\n\n\n<li>At the time of writing this article, it isn&#8217;t widely adopted and there aren&#8217;t any books on it<\/li>\n\n\n\n<li>It is still &#8220;stuck&#8221; more in the academic place than in the practical space<\/li>\n<\/ul>\n\n\n\n<a name=\"conformal-prediction-finance\">\n\n\n\n<h2 class=\"wp-block-heading\">How can Conformal Prediction be used in Finance?<\/h2>\n\n\n\n<p>Conformal Prediction can be used in Finance in many ways. It seems to be most promising when it comes to forecasting, addressing repayment uncertainty, modeling algorithms in volatile market settings, and more.<\/p>\n\n\n\n<a name=\"conformal-prediction-algorithmic-trading\">\n\n\n\n<h2 class=\"wp-block-heading\">How can Conformal Prediction be used in Algorithmic Trading?<\/h2>\n\n\n\n<p>Conformal Prediction can help address the uncertainty surrounding trading hypotheses on which your algorithms might be based. Some might even use machine learning outputs as signals for potential trades, here too conformal prediction can help.<\/p>\n\n\n\n<a name=\"conformal-prediction-alternatives\">\n\n\n\n<h2 class=\"wp-block-heading\">What are some Conformal Prediction alternatives?<\/h2>\n\n\n\n<p>Conformal Prediction can be replaced with some other methods depending on what you&#8217;re doing. Here are some of the top 3 alternatives:<\/p>\n\n\n\n<ul>\n<li>Platt scaling<\/li>\n\n\n\n<li>Isotonic regression<\/li>\n\n\n\n<li>Spline calibration<\/li>\n<\/ul>\n\n\n\n<a name=\"conformal-prediction-understand\">\n\n\n\n<h2 class=\"wp-block-heading\">Understanding Conformal Prediction<\/h2>\n\n\n\n<p>When it comes to understanding Conformal Prediction, everyone has their preferences some like going into the math behind it and research papers, some like looking at code adaptions, and some like reading guides and tutorials like this one.<\/p>\n\n\n\n<p>I&#8217;ll approach this article by explaining it in very crude layman&#8217;s terms so that the intuition behind Conformal Prediction can be easily grasped and readily applied to your existing and future projects. <\/p>\n\n\n\n<p>Also, I&#8217;ll point you toward resources that feature the other approaches mentioned above in the &#8220;<a href=\"#conformal-prediction-learn-more\">Where can I learn more about Conformal Prediction<\/a>&#8221; section.<\/p>\n\n\n\n<p>Conformal Prediction usually &#8220;works&#8221; in the following way:<\/p>\n\n\n\n<ol>\n<li>Take a trained ML model<\/li>\n\n\n\n<li>Create an additional set from the data called the calibration set (unseen by the model)<\/li>\n\n\n\n<li>Pick an error metric (calibration score) and apply a cutoff point (e.g. 95% quantile cutoff point)<\/li>\n\n\n\n<li>Use the cutoff point to inform the width of your prediction interval<\/li>\n\n\n\n<li>Use this cutoff point to form the prediction sets for new examples<\/li>\n\n\n\n<li>Now, most of your predictions should fall inside the prediction interval<\/li>\n<\/ol>\n\n\n\n<p>That&#8217;s it! You now can adapt this for different approaches and models. For example, if you had an image classifier trained to classify dolphins the prediction interval wouldn&#8217;t hold labels of more uncertain classes that are being kept away by the chosen quantile level.<\/p>\n\n\n\n<p>Also, the more uncertain the model is (e.g. the harder the inputs are), the larger the prediction set will be and vice-versa which is exactly what we&#8217;re looking for. This can be further informed by the average set size, set spread, and coverage that we&#8217;ll explore further in this article.<\/p>\n\n\n\n<p>Finally, the error metric (also known as the calibration score) is a very important part and engineering choice that informs everything else when interpreting the results of conformal prediction. <\/p>\n\n\n\n<p>Let&#8217;s go into coding so that we can grasp this further.<\/p>\n\n\n\n<a name=\"conformal-prediction-mapie\">\n\n\n\n<h2 class=\"wp-block-heading\">What is MAPIE?<\/h2>\n\n\n\n<p>MAPIE (Model Agnostic Prediction Interval Estimator) is a Python library that&nbsp;allows you to estimate prediction intervals using any scikit-learn-compatible model for single-output regression or multi-class classification settings. All prediction sets are based on conformal prediction.<\/p>\n\n\n\n<a name=\"conformal-prediction-start\">\n\n\n\n<h2 class=\"wp-block-heading\">How to get started with Conformal Prediction?<\/h2>\n\n\n\n<p>To get started with Conformal Prediction, all we need is Python and an IDE such as VS Code, Google Colab, or the like. I&#8217;ll go with Google Colab this time and install MAPIE.<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>!pip install mapie<\/code><\/pre><\/div>\n\n\n\n<a name=\"conformal-prediction-mapie-python\">\n\n\n\n<h2 class=\"wp-block-heading\">How to apply Conformal Prediction with MAPIE in Python?<\/h2>\n\n\n\n<p>To apply Conformal Prediction with MAPIE in Python, we&#8217;ll need to use either the <code>MapieClassification<\/code> or <code>MapieRegression<\/code> algorithm to obtain the scores that we&#8217;ll use for implementing conformal prediction. <\/p>\n\n\n\n<p>Let&#8217;s start with a classification example:<\/p>\n\n\n\n<a name=\"conformal-prediction-classification-mapie\">\n\n\n\n<h2 class=\"wp-block-heading\">How to apply Conformal Prediction for Classification with MAPIE?<\/h2>\n\n\n\n<p>To apply Conformal Prediction for classification with MAPIE, we will estimate a prediction set of multiple classes such that the probability of a true label of a new test point is always higher than the target confidence level.<\/p>\n\n\n\n<p>We&#8217;ll use the classifier&#8217;s softmax score output as the conformity score on the Iris dataset. The classifier will be the Naive Bayes classifier from the <a href=\"https:\/\/algotrading101.com\/learn\/sklearn-guide\/\"><code>scikit-learn<\/code><\/a> library. <\/p>\n\n\n\n<p>To execute this properly, we&#8217;ll need to follow these steps:<\/p>\n\n\n\n<ul>\n<li>Craft a toy dataset and form train, calibration, and test sets.<\/li>\n\n\n\n<li>Fit the model on the train set.<\/li>\n\n\n\n<li>Set the conformal score to be the softmax output of the true class for each sample in the calibration set <code><img decoding=\"async\" src=\"https:\/\/mapie.readthedocs.io\/en\/latest\/_images\/math\/8980a1be91dffb15b6feeb9b1f4ca5c8a091627d.png\" alt=\"S_i = \\hat{f}(X_{i})_{y_i}\"><\/code><\/li>\n\n\n\n<li>Define <img decoding=\"async\" src=\"https:\/\/mapie.readthedocs.io\/en\/latest\/_images\/math\/7f76127ff352515f9827cde331e9acd7a9a5058c.png\" alt=\"\\hat{q}\"> as being the quantile of calibration scores (S1,&#8230;,Sn) which is <code><img decoding=\"async\" src=\"https:\/\/mapie.readthedocs.io\/en\/latest\/_images\/math\/0773be93a22f0067328ed61b54b247b50e598d08.png\" alt=\"(n + 1) (\\alpha) \/ n\"><\/code><\/li>\n\n\n\n<li>Form a prediction set for each new test data point (where X<sub>n+1<\/sub> is known but Y<sub>n+1<\/sub> isn&#8217;t) so that it includes all the labels with a sufficiently high softmax output. <code><img decoding=\"async\" src=\"https:\/\/mapie.readthedocs.io\/en\/latest\/_images\/math\/5d492088ba6a61788c6bedce392d37e56b9bbaf8.png\" alt=\"C(X_{n+1}) = \\{y: \\hat{f}(X_{n+1})_{y} > \\hat{q}\\}\"><\/code><\/li>\n<\/ul>\n\n\n\n<p>Prior to doing the first two steps, we will import the libraries we need:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>import numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import naive_bayes\n\nfrom mapie import MapieClassifier\nfrom mapie.metrics import classification_coverage_score<\/code><\/pre><\/div>\n\n\n\n<p>Now, the first two steps:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code># Create a toy dataset with 2 features and 3 classes (0, 1, 2) with a bit of noise\nn_samples = 1500\nn_features = 2\nn_classes = 3\n\nX = np.random.randn(n_samples, n_features)\ny = np.zeros(n_samples)\n\nfor i in range(n_classes):\n    X[y == i] += np.random.randn(1, n_features) * 1.2\n\ny = np.where(X[:, 0] &gt; 0, 0, 1)\ny = np.where(X[:, 1] &gt; 0, y, y + 1)\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=42)\nX_train, X_cal, y_train, y_cal = train_test_split(X_train, y_train, test_size=0.1, random_state=42)\n\n# Train a Gaussian Naive Bayes classifier\nclf = naive_bayes.GaussianNB()\nclf.fit(X_train, y_train)<\/code><\/pre><\/div>\n\n\n\n<p>Before moving on, let&#8217;s plot our training set data:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>plt.figure(figsize=(7, 7))\nplt.scatter(X_train[:, 1], X_train[:, 2], c=y_train, s=100, cmap=&#39;viridis&#39;, edgecolor=&#39;k&#39;, linewidth=2)\nplt.xlabel(&#39;Feature 1&#39;)\nplt.ylabel(&#39;Feature 2&#39;)\nplt.show()<\/code><\/pre><\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"612\" height=\"602\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/training_set.webp\" alt=\"\" class=\"wp-image-19270\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/training_set.webp 612w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/training_set-300x295.webp 300w\" sizes=\"(max-width: 612px) 100vw, 612px\" \/><\/figure>\n\n\n\n<p>Okay, now we proceed with computing the probabilities on the test set, computing the softmax scores, and passing the classifier as the estimator for  <code>MapieClassifier<\/code>. Then we can fit the <code>MapieClassifier<\/code> on the calibration sets.<\/p>\n\n\n\n<p>Here also comes the part where we pick the alpha metric (<code>alpha = 1 - target coverage<\/code>) which will suggest the quantile we are to set. In other words, the lower the alpha is, the higher the quantile will be. The higher the quantile, the less likely that a score satisfies it in highly uncertain areas.<\/p>\n\n\n\n<p>Let&#8217;s pick 3 different numbers for alpha so that we can visualize the difference it makes.<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code># Predict class labels and probabilities\ny_pred_proba = clf.predict_proba(X_test)\n\n# Calculate Softmax score\nsoftmax_score = np.max(y_pred_proba, axis=1)\n\n# Initialize the Conformal Prediction classifier\nmapie_score = MapieClassifier(estimator=clf, cv=&quot;prefit&quot;, method=&quot;score&quot;)\nmapie_score.fit(X_cal, y_cal)\n\nalpha = [0.2, 0.1, 0.05]\ny_pred_score, y_ps_score = mapie_score.predict(X_test, alpha=alpha)<\/code><\/pre><\/div>\n\n\n\n<ul>\n<li><code>method=\"score\"<\/code> &#8211; is based on the scores (i.e. 1 minus the softmax score of the true label) on the calibration set<\/li>\n\n\n\n<li><code>y_pred_score<\/code> &#8211; represents the prediction in the test set by the base estimator.<\/li>\n\n\n\n<li><code>y_ps_score<\/code> &#8211; represents the prediction sets estimated by MAPIE.<\/li>\n<\/ul>\n\n\n\n<p>To visualize the differences between different alpha scores, I&#8217;ll use a function kindly provided inside the MAPIE documentation:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>def plot_scores(n, alphas, scores, quantiles):\n    colors = {0: &quot;#1f77b4&quot;, 1: &quot;#ff7f0e&quot;, 2: &quot;#2ca02c&quot;}\n    plt.figure(figsize=(7, 5))\n    plt.hist(scores, bins=&quot;auto&quot;)\n    for i, quantile in enumerate(quantiles):\n        plt.vlines(\n            x=quantile,\n            ymin=0,\n            ymax=400,\n            color=colors[i],\n            ls=&quot;dashed&quot;,\n            label=f&quot;alpha = {alphas[i]}&quot;\n        )\n    plt.title(&quot;Distribution of scores&quot;)\n    plt.legend()\n    plt.xlabel(&quot;Scores&quot;)\n    plt.ylabel(&quot;Count&quot;)\n    plt.show()<\/code><\/pre><\/div>\n\n\n\n<p>Prior to using the function, we need to obtain the scores and quantiles from MAPIE. Then, we can plot the difference:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>scores = mapie_score.conformity_scores_\nn = len(mapie_score.conformity_scores_)\nquantiles = mapie_score.quantiles_\n\nplot_scores(n, alpha, scores, quantiles)<\/code><\/pre><\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"618\" height=\"470\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_score_distribution.webp\" alt=\"\" class=\"wp-image-19271\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_score_distribution.webp 618w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_score_distribution-300x228.webp 300w\" sizes=\"(max-width: 618px) 100vw, 618px\" \/><\/figure>\n\n\n\n<p>In the graph above we can observe how different alpha values affect the quantile and the overall width of our conformal prediction sets. We can also observe that the spread looks good and we&#8217;ll double down on this by exploring the coverage of each alpha score.<\/p>\n\n\n\n<p>Before that, it is worth repeating that a high alpha might lead to a high quantile that might not be reached by any class in regions where the uncertainty is high. For example, this would be the region where there is an overlap between the classes. Especially the very middle of the chart.<\/p>\n\n\n\n<p>Now, let&#8217;s plot the differences for different values of alpha. I&#8217;ll use the plotting function that is provided inside the MAPIE docs and adapt it to not show a mesh but rather individual points.<\/p>\n\n\n\n<p>On the top left, you will see the prediction made by the base estimator. The other 3 subplots will feature each alpha score. <\/p>\n\n\n\n<p>Points that are red had high uncertainty and the model abstained from giving a prediction for them (lower coverage), while points that are highly satiated in purple shared multiple possible classes in the prediction set (higher coverage).<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>plot_results(alpha, X_test, y_pred_score, y_ps_score)<\/code><\/pre><\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"821\" height=\"836\" src=\"http:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/predicted_labels_confromal.webp\" alt=\"\" class=\"wp-image-19274\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/predicted_labels_confromal.webp 821w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/predicted_labels_confromal-295x300.webp 295w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/predicted_labels_confromal-768x782.webp 768w\" sizes=\"(max-width: 821px) 100vw, 821px\" \/><\/figure>\n\n\n\n<p>Take note that there are alterations to this method (e.g. Adaptive Prediction Set) whose conformity score is the cumulated score of the softmax output until the true label is reached. This is for those that don&#8217;t want null predictions under high uncertainty.<\/p>\n\n\n\n<p>You have probably raised an important issue here. How does one pick the right alpha score? Well, to do this, MAPIE provided us with an example of testing a wide range of alpha values and observing their coverage scores and average prediction set sizes.<\/p>\n\n\n\n<p>The full code can be found in the GitHub repository and here&#8217;s the output featuring the graph of different alpha score coverages and average set sizes for each alpha score:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1001\" height=\"448\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_coverage_scores.webp\" alt=\"\" class=\"wp-image-19272\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_coverage_scores.webp 1001w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_coverage_scores-300x134.webp 300w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_coverage_scores-768x344.webp 768w\" sizes=\"(max-width: 1001px) 100vw, 1001px\" \/><\/figure>\n\n\n\n<a name=\"conformal-prediction-regression-mapie\">\n\n\n\n<h2 class=\"wp-block-heading\">How to apply Conformal Prediction for Regression with MAPIE?<\/h2>\n\n\n\n<p>To apply Conformal Prediction for Regression with MAPIE, we will use a conformalized quantile regression estimator (<code>LGBMRegressor<\/code>) with the <code>MapieQuantileRegressor<\/code> module on example tabular data.<\/p>\n\n\n\n<p>Have in mind that you can use the <code>MapieRegressor<\/code> for other regression types that aren&#8217;t quantile regressors. Also, some code will be hidden for brevity and can be found inside the GitHub repository.<\/p>\n\n\n\n<p>The steps that we&#8217;ll do are the following:<\/p>\n\n\n\n<ol>\n<li>Load the Diamonds dataset and preprocess it (we remove the categorical variables for simplicity)<\/li>\n\n\n\n<li>Split the dataset into train, calibration, and test sets<\/li>\n\n\n\n<li>Optimize and train the Quantile LGMBRegressor<\/li>\n\n\n\n<li>Pass the estimator to MAPIE and fit it with 80% coverage (alpha=0.2)<\/li>\n\n\n\n<li>Plot the conformal prediction sets and errors<\/li>\n<\/ol>\n\n\n\n<p>Let&#8217;s perform the first two steps:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code># import the diamonds dataset\ndf = pd.read_csv(&quot;https:\/\/raw.githubusercontent.com\/mwaskom\/seaborn-data\/master\/diamonds.csv&quot;)\n\n# Create a train, calibration and test set\nX = df.drop([&quot;price&quot;], axis=1)\n# drop categorical variables for brevity\nX = X.drop([&quot;cut&quot;, &quot;color&quot;, &quot;clarity&quot;], axis=1)\ny = df[&quot;price&quot;]\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\nX_cal, X_test, y_cal, y_test = train_test_split(X_test, y_test, test_size=0.5, random_state=42)<\/code><\/pre><\/div>\n\n\n\n<p>To observe the distribution of diamond prices we can do the following:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>plt.hist(y_train, bins=&quot;auto&quot;)\nplt.title(&quot;Distribution of prices&quot;)\nplt.xlabel(&quot;Price&quot;)\nplt.ylabel(&quot;Count&quot;)\nplt.show()<\/code><\/pre><\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"580\" height=\"455\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/diamonds_distribution_price.webp\" alt=\"\" class=\"wp-image-19276\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/diamonds_distribution_price.webp 580w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/diamonds_distribution_price-300x235.webp 300w\" sizes=\"(max-width: 580px) 100vw, 580px\" \/><\/figure>\n\n\n\n<p>Now, we optimize the model:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>from sklearn.model_selection import RandomizedSearchCV\nfrom sklearn.model_selection import KFold\n\n# fit a Quantile LightGBM model\nclf = LGBMRegressor(\n    objective=&quot;quantile&quot;,\n    alpha=0.05,\n    random_state=42\n)\n\n# optimize the model\nparams_distributions = dict(\n    num_leaves=np.random.randint(10, 50, 5),\n    max_depth=np.random.randint(3, 20, 1),\n    n_estimators=np.random.randint(50, 300, 10),\n    learning_rate=np.random.uniform(0.01, 0.1, 10),\n)\n\noptim_model = RandomizedSearchCV(\n    clf,\n    param_distributions=params_distributions,\n    n_jobs=-1,\n    n_iter=100,\n    cv=KFold(n_splits=5, shuffle=True),\n    verbose=-1\n)\n\noptim_model.fit(X_train, y_train)\nclf = optim_model.best_estimator_<\/code><\/pre><\/div>\n\n\n\n<p>After this, we pass the model onto MAPIE:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>mapie = MapieQuantileRegressor(\n    clf,\n    method=&quot;quantile&quot;,\n    cv=&quot;split&quot;,\n    alpha=0.2\n)\nmapie.fit(X_train, y_train)\n\n# sort the data\ny_pred, y_pis = mapie.predict(X_test)\n\n(\ny_test_sorted,\ny_pred_sorted,\nlower_bound,\nupper_bound\n) = sort_y_values(y_test, y_pred, y_pis)\n\n# Calculate the coverage and width of the prediction intervals\ncoverage = regression_coverage_score(\ny_test,\ny_pis[:, 0, 0],\ny_pis[:, 1, 0]\n)\nwidth = regression_mean_width_score(\ny_pis[:, 0, 0],\ny_pis[:, 1, 0]\n)<\/code><\/pre><\/div>\n\n\n\n<p>Now, we can plot our prediction intervals, the true value, and the error intervals. The prediction interval will be plotted where the true value was inside of it, and the error interval will be the interval inside of which the true value wasn&#8217;t present.<\/p>\n\n\n\n<p>I&#8217;ll only plot these for a small subset of the test set as the graph can get messy. I encourage everyone to experiment with different parts of the test set to observe the difference. For example, the model performs better on medium-to-high prices than on lower ones. <\/p>\n\n\n\n<p>As the prices go up, so do our prediction intervals.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"580\" height=\"455\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/prediction_intervals_coverage_diamonds.webp\" alt=\"\" class=\"wp-image-19278\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/prediction_intervals_coverage_diamonds.webp 580w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/prediction_intervals_coverage_diamonds-300x235.webp 300w\" sizes=\"(max-width: 580px) 100vw, 580px\" \/><\/figure>\n\n\n\n<a name=\"conformal-prediction-forecasting-mapie\">\n\n\n\n<h2 class=\"wp-block-heading\">How to perform forecasting with Conformal Prediction in Python?<\/h2>\n\n\n\n<p>To perform forecasting with Conformal Prediction in Python, you can use any forecasting algorithm and create training and test sets. Then, use a library such as MAPIE to calibrate the algorithm.<\/p>\n\n\n\n<p>You can also explore the option of coding your own conformal prediction implementations.<\/p>\n\n\n\n<p>Below, I&#8217;ll show an example of a monthly sales dataset I&#8217;ve found online. It will come with the GitHub repo.<\/p>\n\n\n\n<p>After creating the dataset we will proceed by engineering a few features and splitting it into the train and test sets. We also load in the MapieRegressor and XGBoost as our algorithms of choice.<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>from mapie.time_series_regression import MapieTimeSeriesRegressor\nfrom mapie.subsample import BlockBootstrap\nimport xgboost as xgb\n\ndf = pd.read_csv(&quot;sales.csv&quot;)\ndf[&quot;date&quot;] = pd.to_datetime(df[&quot;date&quot;])\ndf = df.set_index(&quot;date&quot;)\n\n# create features\ndf[&quot;month&quot;] = df.index.month\ndf[&quot;year&quot;] = df.index.year\n\n# plot the data\nplt.plot(df[&quot;sales&quot;])\nplt.title(&quot;Sales&quot;)\nplt.xlabel(&quot;Date&quot;)\nplt.ylabel(&quot;Sales&quot;)<\/code><\/pre><\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"571\" height=\"455\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/sales_data.webp\" alt=\"\" class=\"wp-image-19265\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/sales_data.webp 571w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/sales_data-300x239.webp 300w\" sizes=\"(max-width: 571px) 100vw, 571px\" \/><\/figure>\n\n\n\n<p>Now, let&#8217;s make this more fun by introducing a sudden change to the time series to see how the model will react. We&#8217;ll first split it into 80% train and 20% test after which we introduce an abrupt change in the label at the start of 1975 by applying a 10% drop rate.<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code># create train and test sets\nX = df.drop([&quot;sales&quot;], axis=1)\ny = df[&quot;sales&quot;]\n\n# split the data where test is eveyrthing after 1974\nX_train = X[X.index.year &lt; 1974]\nX_test = X[X.index.year &gt;= 1974]\ny_train = y[y.index.year &lt; 1974]\ny_test = y[y.index.year &gt;= 1974]\n\n# introduce a change in seasonality after 1975 in the y_test set by applying a 10% drop\ny_test.loc[y_test.index.year &gt;= 1975] = y_test.loc[y_test.index.year &gt;= 1975] * 0.9\n\n# plot the data whole data and colored train and test sets\nplt.plot(X_train.index, y_train, label=&quot;Train set&quot;)\nplt.plot(X_test.index, y_test, label=&quot;Test set&quot;)\nplt.title(&quot;Sales&quot;)\nplt.xlabel(&quot;Date&quot;)\nplt.ylabel(&quot;Sales&quot;)\nplt.legend()<\/code><\/pre><\/div>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"571\" height=\"455\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/sales_split_data.webp\" alt=\"\" class=\"wp-image-19266\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/sales_split_data.webp 571w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/sales_split_data-300x239.webp 300w\" sizes=\"(max-width: 571px) 100vw, 571px\" \/><\/figure>\n\n\n\n<p>Due to the introduction of the abrupt change, I expect the algorithm to struggle to capture the change. Here, we hope that conformal prediction will be able to adjust for the change by having the true value inside its prediction sets.<\/p>\n\n\n\n<p>Now, we will train the XGBoost model and optimize its hyperparameters:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>reg = xgb.XGBRegressor(\n    n_estimators=1000,\n    random_state=42\n)\n\n# optimize the model\nparams_distributions = dict(\n    max_depth=np.random.randint(3, 20, 1),\n    learning_rate=np.random.uniform(0.01, 0.1, 10),\n)\n\nreg = RandomizedSearchCV(\n    reg,\n    param_distributions=params_distributions,\n    n_jobs=-1,\n    n_iter=10,\n    cv=KFold(n_splits=5, shuffle=True),\n    verbose=-1\n)\n\nreg.fit(X_train, y_train)<\/code><\/pre><\/div>\n\n\n\n<p>Now, we can estimate the prediction intervals by using the MapieTimeSeriesRegresor. Here, we have two options to estimate the prediction intervals:<\/p>\n\n\n\n<ul>\n<li>the regular .fit and .predict process, limiting the use of training set residuals to create prediction intervals<\/li>\n\n\n\n<li>the use of <code>.partial_fit<\/code> in addition to <code>.fit<\/code> and <code>.predict<\/code> allowing MAPIE to use new residuals from the test points as new data are becoming available.<\/li>\n<\/ul>\n\n\n\n<p>The second approach is better for time series with unexpected oscillations. I&#8217;ll show you both versions so that you can see the difference between the two.<\/p>\n\n\n\n<p>Take note that we&#8217;ll be using the <code>BlockBootstrap<\/code> method which is shown to be better for time-series modeling. As for our coverage, we will go with 95% by having the alpha be 0.05. The method for the MAPIE regressor will be EnbPi which was shown to perform well on time series data.<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code># It&#39;s Confromal Prediction time!\nalpha = 0.05\ngap = 1\n\nmapie_cv = BlockBootstrap(\n    n_blocks=10,\n    overlapping=True,\n    random_state=42\n)\n\nmapie_enbpi = MapieTimeSeriesRegressor(\n    reg,\n    method=&quot;enbpi&quot;,\n    cv=mapie_cv,\n    agg_function=&quot;mean&quot;,\n    n_jobs=-1\n)<\/code><\/pre><\/div>\n\n\n\n<p>Now, we start by applying the first method:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>mapie_enbpi = mapie_enbpi.fit(X_train, y_train)\ny_pred_npfit, y_pis_npfit = mapie_enbpi.predict(\n    X_test, alpha=alpha, ensemble=True, optimize_beta=True\n)\ncoverage_npfit = regression_coverage_score(\n    y_test, y_pis_npfit[:, 0, 0], y_pis_npfit[:, 1, 0]\n)\nwidth_npfit = regression_mean_width_score(y_pis_npfit[:, 0, 0], y_pis_npfit[:, 1, 0])<\/code><\/pre><\/div>\n\n\n\n<p>Now, we apply the second method:<\/p>\n\n\n\n<div class=\"hcb_wrap\"><pre class=\"prism line-numbers lang-python\" data-lang=\"Python\"><code>mapie_enbpi = mapie_enbpi.fit(X_train, y_train)\n\ny_pred = np.zeros((len(test), 2))\ny_pis = np.zeros((len(test), 2, 2))\nconformity_scores_pfit = []\nlower_quantiles_pfit = []\nhigher_quantiles_pfit = []\n\ny_pred[:gap], y_pis[:gap, :, :] = mapie_enbpi.predict(\n    X_test.iloc[:gap, :], alpha=alpha, ensemble=True, optimize_beta=True\n)\n\nfor step in range(gap, len(X_test), gap):\n    mapie_enbpi.partial_fit(\n        X_test.iloc[(step - gap) : step, :],\n        y_test.iloc[(step - gap) : step],\n    )\n\n    (y_pred[step : step + gap], y_pis[step : step + gap, :, :],) = mapie_enbpi.predict(\n        X_test.iloc[step : (step + gap), :],\n        alpha=alpha,\n        ensemble=True,\n        optimize_beta=True,\n    )\n\n    conformity_scores_pfit.append(mapie_enbpi.conformity_scores_)\n    lower_quantiles_pfit.append(mapie_enbpi.lower_quantiles_)\n    higher_quantiles_pfit.append(mapie_enbpi.higher_quantiles_)\n\ncoverage_pfit = regression_coverage_score(y_test, y_pis[:, 0, 0], y_pis[:, 1, 0])\nwidth_pfit = regression_mean_width_score(y_pis[:, 0, 0], y_pis[:, 1, 0])<\/code><\/pre><\/div>\n\n\n\n<p>Finally, when we plot the prediction intervals and get the following chart:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"381\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_intervals_time_series-1024x381.webp\" alt=\"\" class=\"wp-image-19267\" srcset=\"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_intervals_time_series-1024x381.webp 1024w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_intervals_time_series-300x112.webp 300w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_intervals_time_series-768x286.webp 768w, https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction_intervals_time_series.webp 1264w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>The coverage in both examples isn&#8217;t great, but we can see how the introduction of the partial fit managed to &#8220;recover&#8221; after the abrupt change in the label by adjusting its prediction intervals.<\/p>\n\n\n\n<p>Please feel free to explore different alpha values and how they affect the output intervals of the algorithm.<\/p>\n\n\n\n<a name=\"conformal-prediction-learn-more\">\n\n\n\n<h2 class=\"wp-block-heading\">Where can I learn more about Conformal Prediction?<\/h2>\n\n\n\n<p>To learn more about conformal prediction, I suggest checking out the following resources:<\/p>\n\n\n\n<ul>\n<li><a href=\"https:\/\/github.com\/valeman\/awesome-conformal-prediction#Python\">Awesome Conformal Prediction<\/a> &#8211; A GitHub repo that is professionally curated by Valery Manokhin on everything that has to do with conformal prediction<\/li>\n\n\n\n<li><a href=\"https:\/\/arxiv.org\/pdf\/2107.07511.pdf\">A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification<\/a> &#8211; A paper written by Anastasios N. Angelopoulos and Stephen Bates<\/li>\n\n\n\n<li><a href=\"https:\/\/mapie.readthedocs.io\/en\/latest\/index.html\">MAPIE documentation<\/a> &#8211; Has very good tutorials and explanations<\/li>\n<\/ul>\n\n\n\n<a name=\"conformal-prediction-full-code\">\n\n\n\n<h2 class=\"wp-block-heading\">Full code<\/h2>\n\n\n\n<p><a href=\"https:\/\/github.com\/AlgoTrading101\/Conformal-Prediction-MAPIE-Algotrading101\">GitHub Link<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"pvc_clear\"><\/div>\n<p id=\"pvc_stats_19429\" class=\"pvc_stats total_only  \" data-element-id=\"19429\" style=\"\"><i class=\"pvc-stats-icon medium\" aria-hidden=\"true\"><svg aria-hidden=\"true\" focusable=\"false\" data-prefix=\"far\" data-icon=\"chart-bar\" role=\"img\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" viewBox=\"0 0 512 512\" class=\"svg-inline--fa fa-chart-bar fa-w-16 fa-2x\"><path fill=\"currentColor\" d=\"M396.8 352h22.4c6.4 0 12.8-6.4 12.8-12.8V108.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v230.4c0 6.4 6.4 12.8 12.8 12.8zm-192 0h22.4c6.4 0 12.8-6.4 12.8-12.8V140.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v198.4c0 6.4 6.4 12.8 12.8 12.8zm96 0h22.4c6.4 0 12.8-6.4 12.8-12.8V204.8c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v134.4c0 6.4 6.4 12.8 12.8 12.8zM496 400H48V80c0-8.84-7.16-16-16-16H16C7.16 64 0 71.16 0 80v336c0 17.67 14.33 32 32 32h464c8.84 0 16-7.16 16-16v-16c0-8.84-7.16-16-16-16zm-387.2-48h22.4c6.4 0 12.8-6.4 12.8-12.8v-70.4c0-6.4-6.4-12.8-12.8-12.8h-22.4c-6.4 0-12.8 6.4-12.8 12.8v70.4c0 6.4 6.4 12.8 12.8 12.8z\" class=\"\"><\/path><\/svg><\/i> <img decoding=\"async\" width=\"16\" height=\"16\" alt=\"Loading\" src=\"https:\/\/algotrading101.com\/learn\/wp-content\/plugins\/page-views-count\/ajax-loader-2x.gif\" border=0 \/><\/p>\n<div class=\"pvc_clear\"><\/div>\n<p>Table of contents: What is Conformal Prediction? Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model. What is Conformal Prediction used for? Conformal Prediction is often used for calibrating one&#8217;s machine learning models, estimating their uncertainty, comparing models, increasing the practical performance of the model, [&hellip;]<\/p>\n","protected":false},"author":14,"featured_media":19282,"comment_status":"closed","ping_status":"closed","sticky":true,"template":"","format":"standard","meta":{"_lmt_disableupdate":"no","_lmt_disable":"","_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[3],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Conformal Prediction - A Practical Guide with MAPIE - AlgoTrading101 Blog<\/title>\n<meta name=\"description\" content=\"Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Conformal Prediction - A Practical Guide with MAPIE - AlgoTrading101 Blog\" \/>\n<meta property=\"og:description\" content=\"Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/\" \/>\n<meta property=\"og:site_name\" content=\"Quantitative Trading Ideas and Guides - AlgoTrading101 Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-12-19T22:21:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-02-04T21:52:07+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction.webp\" \/>\n\t<meta property=\"og:image:width\" content=\"500\" \/>\n\t<meta property=\"og:image:height\" content=\"500\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/webp\" \/>\n<meta name=\"author\" content=\"Igor Radovanovic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Igor Radovanovic\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"17 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Conformal Prediction - A Practical Guide with MAPIE - AlgoTrading101 Blog","description":"Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/","og_locale":"en_US","og_type":"article","og_title":"Conformal Prediction - A Practical Guide with MAPIE - AlgoTrading101 Blog","og_description":"Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model.","og_url":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/","og_site_name":"Quantitative Trading Ideas and Guides - AlgoTrading101 Blog","article_published_time":"2022-12-19T22:21:08+00:00","article_modified_time":"2025-02-04T21:52:07+00:00","og_image":[{"width":500,"height":500,"url":"http:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2022\/12\/conformal_prediction.webp","type":"image\/webp"}],"author":"Igor Radovanovic","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Igor Radovanovic","Est. reading time":"17 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/#article","isPartOf":{"@id":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/"},"author":{"name":"Igor Radovanovic","@id":"https:\/\/algotrading101.com\/learn\/#\/schema\/person\/a7ae60c112a73b7c3fe14ac56726a0ae"},"headline":"Conformal Prediction &#8211; A Practical Guide with MAPIE","datePublished":"2022-12-19T22:21:08+00:00","dateModified":"2025-02-04T21:52:07+00:00","mainEntityOfPage":{"@id":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/"},"wordCount":2286,"publisher":{"@id":"https:\/\/algotrading101.com\/learn\/#organization"},"articleSection":["Programming"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/","url":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/","name":"Conformal Prediction - A Practical Guide with MAPIE - AlgoTrading101 Blog","isPartOf":{"@id":"https:\/\/algotrading101.com\/learn\/#website"},"datePublished":"2022-12-19T22:21:08+00:00","dateModified":"2025-02-04T21:52:07+00:00","description":"Conformal Prediction is a set of algorithms that assess the uncertainty of predictions produced by a machine learning model.","breadcrumb":{"@id":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/algotrading101.com\/learn\/conformal-prediction-guide\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/algotrading101.com\/learn\/"},{"@type":"ListItem","position":2,"name":"Conformal Prediction &#8211; A Practical Guide with MAPIE"}]},{"@type":"WebSite","@id":"https:\/\/algotrading101.com\/learn\/#website","url":"https:\/\/algotrading101.com\/learn\/","name":"Quantitative Trading Ideas and Guides - AlgoTrading101 Blog","description":"Authentic Stories about Algorithmic trading, coding and life.","publisher":{"@id":"https:\/\/algotrading101.com\/learn\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/algotrading101.com\/learn\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/algotrading101.com\/learn\/#organization","name":"AlgoTrading101","url":"https:\/\/algotrading101.com\/learn\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/algotrading101.com\/learn\/#\/schema\/logo\/image\/","url":"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2020\/11\/AlgoTrading101-Lucas-Liew.jpg","contentUrl":"https:\/\/algotrading101.com\/learn\/wp-content\/uploads\/2020\/11\/AlgoTrading101-Lucas-Liew.jpg","width":1200,"height":627,"caption":"AlgoTrading101"},"image":{"@id":"https:\/\/algotrading101.com\/learn\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/algotrading101.com\/learn\/#\/schema\/person\/a7ae60c112a73b7c3fe14ac56726a0ae","name":"Igor Radovanovic","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/algotrading101.com\/learn\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d46175c509b3ee240a1e2bbe735a4d1e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d46175c509b3ee240a1e2bbe735a4d1e?s=96&d=mm&r=g","caption":"Igor Radovanovic"},"sameAs":["https:\/\/igorradovanovic.com","https:\/\/www.linkedin.com\/in\/igor-radovanovic-profile"],"url":"https:\/\/algotrading101.com\/learn\/author\/igor\/"}]}},"modified_by":"Lucas Liew","_links":{"self":[{"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/posts\/19429"}],"collection":[{"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/comments?post=19429"}],"version-history":[{"count":7,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/posts\/19429\/revisions"}],"predecessor-version":[{"id":22837,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/posts\/19429\/revisions\/22837"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/media\/19282"}],"wp:attachment":[{"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/media?parent=19429"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/categories?post=19429"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/algotrading101.com\/learn\/wp-json\/wp\/v2\/tags?post=19429"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}