🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 57 (from laksa199)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa057.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://www.quantstart.com/articles/calculating-realised-volatility-with-polygon-forex-data/\')), getAhrefsUnparsedNoserviceFromURL(\'https://www.quantstart.com/articles/calculating-realised-volatility-with-polygon-forex-data/\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/www.quantstart.com\/articles\/calculating-realised-volatility-with-polygon-forex-data\/","crawl_time":1776155621,"first_indexed_time":1687968136,"http_code":200,"src_unparsed":"com,quantstart!www,\/articles\/calculating-realised-volatility-with-polygon-forex-data\/ s443","src_root_hash":"15201169621511058857","history_drop_reason":null,"meta_title":"Calculating Realised Volatility with Polygon Forex data | QuantStart","meta_descriptions":["In this article we carry out an exploratory data analysis of minutely Forex pairs data"],"attrs_boilerpipe_text":"In the previous\narticle\nwe wrote a Python function which utilised the Polygon API to extract a month of minutely data for both a major (EURUSD) and exotic (MZXZAR) FX pair. We plotted the returns series and looked at some of the issues that can occur when working with this type of data. This article is part of series where we will be creating a machine learning model which uses realised volatility to predict market regime change. In this article we will be learning how to calculate realised volatility. We will also be conducting a exploratory data analysis to prepare the feature set for our machine learning model.\nIn order to follow along with the code in this tutorial you will need:\nPython 3.9\nMatplotlib 3.5\nPandas 1.4\nRequests 2.27\nSeaborn 0.12.\nWe also recommend that you take a look at our\nearly career research series\nto create an algorithmic trading environment inside a Jupyter Notebook.\nIn order to calculate realised volatility we first need to obtain and format the data. In the previous article we created Python functions to contact the Polygon API and obtain a month of minutely data for EURUSD and MZXZAR. You can view the full article\nhere\n. Below you will find the code to obtain the data.\nimport\njson\nimport\nmatplotlib\n.\npyplot\nas\nplt\nimport\npandas\nas\npd\nimport\nrequests\nimport\nseaborn\nas\nsns\n\n\nPOLYGON_API_KEY\n=\nos\n.\ngetenv\n(\n\"POLYGON_API_KEY\"\n)\nHEADERS\n=\n{\n'Authorization'\n:\n'Bearer '\n+\nPOLYGON_API_KEY\n}\nBASE_URL\n=\n'https:\/\/api.polygon.io\/v2\/aggs\/'\ndef\nget_fx_pairs_data\n(\npg_tickers\n)\n:\nstart\n=\n'2023-01-01'\nend\n=\n'2023-01-31'\nmultiplier\n=\n'1'\ntimespan\n=\n'minute'\nfx_url\n=\nf\"range\/\n{\nmultiplier\n}\n\/\n{\ntimespan\n}\n\/\n{\nstart\n}\n\/\n{\nend\n}\n?adjusted=true&sort=asc&limit=50000\"\nfx_pairs_dict\n=\n{\n}\nfor\npair\nin\npg_tickers\n:\nresponse\n=\nrequests\n.\nget\n(\nf\"\n{\nBASE_URL\n}\nticker\/\n{\npair\n}\n\/\n{\nfx_url\n}\n\"\n,\nheaders\n=\nHEADERS\n)\n.\njson\n(\n)\nfx_pairs_dict\n[\npair\n]\n=\npd\n.\nDataFrame\n(\nresponse\n[\n'results'\n]\n)\nreturn\nfx_pairs_dict\ndef\nformat_fx_pairs\n(\nfx_pair_dict\n)\n:\nfor\npair\nin\nfx_pairs_dict\n.\nkeys\n(\n)\n:\nfx_pairs_dict\n[\npair\n]\n[\n't'\n]\n=\npd\n.\nto_datetime\n(\nfx_pairs_dict\n[\npair\n]\n[\n't'\n]\n,\nunit\n=\n'ms'\n)\nfx_pairs_dict\n[\npair\n]\n=\nfx_pairs_dict\n[\npair\n]\n.\nset_index\n(\n't'\n)\nreturn\nfx_pairs_dict\ndef\ncreate_str_index\n(\nfx_pairs_dict\n)\n:\nfx_pairs_str_ind\n=\n{\n}\nfor\npair\nin\nfx_pairs_dict\n.\nkeys\n(\n)\n:\nfx_pairs_str_ind\n[\npair\n]\n=\nfx_pairs_dict\n[\npair\n]\n.\nset_index\n(\nfx_pairs_dict\n[\npair\n]\n.\nindex\n.\nto_series\n(\n)\n.\ndt\n.\nstrftime\n(\n'%Y-%m-%d-%H-%M-%S'\n)\n)\nreturn\nfx_pairs_str_ind\ndef\ncreate_returns_series\n(\nfx_pairs_str_ind\n)\n:\nfor\npair\nin\nfx_pairs_str_ind\n.\nkeys\n(\n)\n:\nfx_pairs_str_ind\n[\npair\n]\n[\n'rets'\n]\n=\nfx_pairs_str_ind\n[\npair\n]\n[\n'c'\n]\n.\npct_change\n(\n)\nreturn\nfx_pair_str_ind\nPython\nThe functions can be called in a new cell or in a main if running the code outside a notebook environment. The code below will store the returns value of each function in a variable.\npg_tickers\n=\n[\n\"C:EURUSD\"\n,\n\"C:MXNZAR\"\n]\nfx_pairs_dict\n=\nget_fx_pairs_data\n(\npg_tickers\n)\nformatted_fx_dict\n=\nformat_fx_pairs\n(\nfx_pairs_dict\n)\nfx_pairs_str_ind\n=\ncreate_str_index\n(\nformatted_fx_dict\n)\nfx_returns_dict\n=\ncreate_returns_series\n(\nfx_pairs_str_ind\n)\nPython\nThe FX data for each pair can be accessed as below:\nfx_returns_dict\n[\n'C:EURUSD'\n]\nPython\nEURUSD formatted returns series\nRealized volatility is a measure of risk. It measures the variability in an investment over a defined period of time. It is calculated as the square root of the sum of the squared return for a particular period of time. In contrast to\nimplied volatility\n, realised volatility shows the actual change in historical prices, rather than a prediction of future volatility. However, it is possible to use the data to forecast the volatility in returns.\nSo why is this useful? If you can reliably predict volatility for a window of future time you can make an assessment of the market regime. This can help you determine the types of trading strategies you may want to use. For example in periods of low volatility you might look at trend following strategies on stocks. In a high volatility environment you might want to consider mean reverting strategies or options trading strategies such as\nStraddle or Strangle\n. By quantifying the level of risk present in your trading universe you can make better trading decisions.\nRealised volatility is a way of understanding the degree of price movement for a given period. It is calculated as follows:\nCollect the price data and calculate the returns\nSquare the returns to give more weight to larger changes\nCalculate the average of the squared returns by adding them up and dividing by the number of periods (this is also known as the variance)\nTake the square root of the variance (also know as the standard deviation). This is your measure of realised volatility\nUltimately we will be training a Support Vector Regressor to help us identify market regime change. In order to do this we will calculate realised volatility across a particular window of time. The window you select will depend upon the frequency of you data and the liquidity of the asset. As we are considering both a major and minor FX pair we will calculate our realised volatility over 30 data points. The Pandas library contains a rolling function that can be applied to any statistical aggregator using\nmethod chaining\n. Here we use\nrolling()\nwith\nstd()\n, to calculate rolling standard deviation. The rolling window size can be set to any number of fixed observations or timedeltas, such as business days.\nCalculating Realised Volatility\nThe code below will create a new column in each of the fx pairs DataFrames which will contain the realised vol calculated as the rolling standard deviation of the last 30 data points. This will equate approximately to the last 30 minutes of data, with some variation depending on the trading frequency of the FX pair. Let's take a look at the realised volatility on a chart.\ndef\ncreate_realised_vol\n:\nfor\npair\nin\npairs_rets_dict\n:\npairs_rets_dict\n[\npair\n]\n[\n'realised_vol'\n]\n=\npairs_rets_dict\n[\npair\n]\n[\n'rets'\n]\n.\nrolling\n(\n30\n)\n.\nstd\n(\n)\nreturn\npairs_rets_dict\nPython\nRealised volatility for EURUSD and MXNZAR\nAs you can see the exotic pair MXNZAR has fewer data points than EURUSD, as it is thinly traded this is to be expected. In fact if we look at the length of both our DataFrames we can see that EURUSD has 31745 rows, whereas MXNZAR has 16865 rows almost half as many. In the previous article we discussed different options available to us that would allow us to handle the missing data points in the MXNZAR pair.\nExploring the Features of the Data\nBefore we begin to construct a machine learning model it is a good idea to have a look at the data and understand how its features could affect model performance. Why should you do this, why can't you just get going? Well, firstly the distribution of the data should influence your choice of model. For example, if your data was linearly distributed the use of linear regression model or a linear kernel in an SVM would be appropriate. If the data is non-lineaer a polynomial regression or radial basis function kernel would be more appropriate. You might also want to consider\ntransforming or scaling\nyour data to make it more compatible with your chosen model. Perharps your data contains outliers or missing data that you need to be aware of. There could be correlation (or linear association) between the features you have selected to train your model. There are many reasons why it is important to get to know your data before you start training a model. In fact it's estimated that between 70-80% of an engineer's time if spend toiling with data. Let's take a look at a simple exploratory data analysis:\nFirst we will look at the distribution of the returns for both our FX pairs. We use the Seaborn function\nkdeplot()\nto visualise the distributions.\nfig\n,\nax\n=\nplt\n.\nsubplots\n(\n2\n,\n2\n,\nfigsize\n=\n(\n16\n,\n10\n)\n,\nsqueeze\n=\nFalse\n)\ny_data\n=\n[\n'rets'\n,\n'realised_vol'\n]\nfor\nidx\n,\nfxpair\nin\nenumerate\n(\nfx_pairs_str_ind\n.\nkeys\n(\n)\n)\n:\nfor\nidx2\n,\ndfcol\nin\nenumerate\n(\ny_data\n)\n:\nrow\n=\n(\nidx\n)\ncol\n=\n(\nidx2\n%\n2\n)\nsns\n.\nkdeplot\n(\nfx_pairs_str_ind\n[\nfxpair\n]\n[\ndfcol\n]\n,\nax\n=\nax\n[\nrow\n]\n[\ncol\n]\n,\nbw_adjust\n=\n0.5\n)\nax\n[\nrow\n]\n[\ncol\n]\n.\nset_title\n(\nfxpair\n)\nfig\n.\nsuptitle\n(\n\"KDE of Returns & Realised Volatility\"\n)\nplt\n.\nshow\n(\n)\nPython\nReturns and realised Volatility distributions\nFirstly it's important to note that the scale on the x axis for EURUSD and MXNZAR is different. The range of returns is larger for MXNZAR in both the poistive and negative direction when compare to EURUSD. This is reflected in the realised volatility. We can also see that we have positive kurtosis of all distributions when compared to a standard normal distribution. The realised volatility also have positive skew. This information will be helpful when we are trying to choose or improve any machine learning models that we might want to use. Finally in the exoctic FX pair MXNZAR, we can see two small humps just before -0.01 and 0.01. This indicates the presence of spikes in our returns data.\nTo look into these with more detail we can use\nViolin plot\n. This will allows us to see how many instances of these spikes there are in our data and how far from the bulk of the distribution they are. The following code will create the violin plot.\nfig\n,\nax\n=\nplt\n.\nsubplots\n(\n1\n,\n2\n,\nfigsize\n=\n(\n16\n,\n10\n)\n,\nsqueeze\n=\nFalse\n)\nfor\nidx\n,\nfxpair\nin\nenumerate\n(\nfx_pairs_str_ind\n.\nkeys\n(\n)\n)\n:\nrow\n=\n(\nidx\n\/\/\n2\n)\ncol\n=\n(\nidx\n%\n2\n)\nsns\n.\nviolinplot\n(\nx\n=\n'rets'\n,\ndata\n=\nfx_pairs_str_ind\n[\nfxpair\n]\n,\nax\n=\nax\n[\nrow\n]\n[\ncol\n]\n,\ninner\n=\n'point'\n)\nax\n[\nrow\n]\n[\ncol\n]\n.\nset_title\n(\nfxpair\n)\nfig\n.\nsuptitle\n(\n\"Violin plot of rets\"\n)\nplt\n.\nshow\n(\n)\nPython\nViolin plot of returns\nAs you can see there are more points further from the bulk of the distribution in the MXNZAR FX pair. This means that there are more occassions where there is a higher or lower return when compared to the EURUSD. This information alone could be useful when you are considering trading strategies.\nAs we are planning to put our data into a machine learning model we also need to think about correlation. We want to use the realised volatility and the returns as features in a training set for a Support Vector Regressor. We need to ensure that these two features aren't correlated, otherwise we would be giving the machine learning model the same information twice. The model would become overspecified which could lead to poor model performance. Below we will look at the correlation between our two chosen features. In order to use both features we would be looking for a correlation close to zero.\nThe following function will display a scatter plot of the two columns; returns and realised vol. It also uses the Pandas\npd.DataFrame.corr\nmethod\nto calculate the correlation between the two.\nfig\n,\nax\n=\nplt\n.\nsubplots\n(\n1\n,\n2\n,\nfigsize\n=\n(\n16\n,\n10\n)\n,\nsqueeze\n=\nFalse\n)\nfor\nidx\n,\nfxpair\nin\nenumerate\n(\nfx_pairs_str_ind\n.\nkeys\n(\n)\n)\n:\nrow\n=\n(\nidx\n\/\/\n2\n)\ncol\n=\n(\nidx\n%\n2\n)\nprint\n(\nf\"vol and rets correlation for\n{\nfxpair\n}\nis \"\nf\"\n{\nfx_pairs_str_ind\n[\nfxpair\n]\n[\n'rets'\n]\n.\ncorr\n(\nfx_pairs_str_ind\n[\nfxpair\n]\n[\n'realised_vol'\n]\n)\n}\n\"\n)\nsns\n.\nscatterplot\n(\ndata\n=\nfx_pairs_str_ind\n[\nfxpair\n]\n,\nx\n=\n'rets'\n,\ny\n=\n'realised_vol'\n,\nax\n=\nax\n[\nrow\n]\n[\ncol\n]\n)\nax\n[\nrow\n]\n[\ncol\n]\n.\nset_title\n(\nfxpair\n)\nfig\n.\nsuptitle\n(\n\"Correlation between realised vol and returns\"\n)\nplt\n.\nshow\n(\n)\nPython\nvol and rets correlation for C:EURUSD is 0.008572524273933109\nvol and rets correlation for C:MXNZAR is 0.018750753882674935\nNone\nFX pairs returns and realised vol correlation\nNow that we know our returns series and realised vol have little correlation we can use them as separate features in our machine learning model. In the next article we will be building a Support Vector Regressor to determine how reliably we can predict the next value of realised volatility using the returns and realised volatility values from the previous time point. This will allow us to prepare a pipeline for our data, understand some of the model parameters and prepare for the final specification of the model.\nRelated Articles\nCreating a Returns Series with Polygon's Forex Data","attrs_markdown":"- [QuantStart](https:\/\/www.quantstart.com\/)\n\n- [Quantcademy](https:\/\/www.quantstart.com\/quantcademy\/)\n- [Books](https:\/\/www.quantstart.com\/articles\/calculating-realised-volatility-with-polygon-forex-data\/)\n  [Successful Algorithmic Trading](https:\/\/www.quantstart.com\/successful-algorithmic-trading-ebook\/)\n  \n  [Advanced Algorithmic Trading](https:\/\/www.quantstart.com\/advanced-algorithmic-trading-ebook\/)\n  \n  [C++ For Quantitative Finance](https:\/\/www.quantstart.com\/cpp-for-quantitative-finance-ebook\/)\n- [QSTrader](https:\/\/www.quantstart.com\/qstrader\/)\n- [Articles](https:\/\/www.quantstart.com\/articles\/)\n- [Login](https:\/\/www.quantstart.com\/members\/login\/)\n\n- [QuantStart](https:\/\/www.quantstart.com\/)\n- [Quantcademy](https:\/\/www.quantstart.com\/quantcademy\/)\n- [Books](https:\/\/www.quantstart.com\/articles\/calculating-realised-volatility-with-polygon-forex-data\/)\n- [Successful Algorithmic Trading](https:\/\/www.quantstart.com\/successful-algorithmic-trading-ebook\/)\n- [Advanced Algorithmic Trading](https:\/\/www.quantstart.com\/advanced-algorithmic-trading-ebook\/)\n- [C++ For Quantitative Finance](https:\/\/www.quantstart.com\/cpp-for-quantitative-finance-ebook\/)\n- [QSTrader](https:\/\/www.quantstart.com\/qstrader\/)\n- [Articles](https:\/\/www.quantstart.com\/articles\/)\n- [Login](https:\/\/www.quantstart.com\/members\/login\/)\n\nCalculating Realised Volatility with Polygon Forex data\n\nIn this article we carry out an exploratory data analysis of minutely Forex pairs data\n\nIn the previous [article](https:\/\/www.quantstart.com\/articles\/creating-a-returns-series-with-polygons-forex-data\/) we wrote a Python function which utilised the Polygon API to extract a month of minutely data for both a major (EURUSD) and exotic (MZXZAR) FX pair. We plotted the returns series and looked at some of the issues that can occur when working with this type of data. This article is part of series where we will be creating a machine learning model which uses realised volatility to predict market regime change. In this article we will be learning how to calculate realised volatility. We will also be conducting a exploratory data analysis to prepare the feature set for our machine learning model.\n\nIn order to follow along with the code in this tutorial you will need:\n\n- Python 3.9\n- Matplotlib 3.5\n- Pandas 1.4\n- Requests 2.27\n- Seaborn 0.12.\n\nWe also recommend that you take a look at our [early career research series](https:\/\/www.quantstart.com\/articles\/creating-an-algorithmic-trading-prototyping-environment-with-jupyter-notebooks-and-plotly\/) to create an algorithmic trading environment inside a Jupyter Notebook.\n\n## Realised Volatility\nIn order to calculate realised volatility we first need to obtain and format the data. In the previous article we created Python functions to contact the Polygon API and obtain a month of minutely data for EURUSD and MZXZAR. You can view the full article [here](https:\/\/www.quantstart.com\/articles\/creating-a-returns-series-with-polygons-forex-data\/). Below you will find the code to obtain the data.\n\n```\n\n```\nPython\n\nCopy\n\nThe functions can be called in a new cell or in a main if running the code outside a notebook environment. The code below will store the returns value of each function in a variable.\n\n```\n\n```\nPython\n\nCopy\n\nThe FX data for each pair can be accessed as below:\n\n```\nfx_returns_dict['C:EURUSD']\n```\nPython\n\nCopy\n\n![Formatted DataFrame of EURUSD](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/fx_returns_dict.png)\n\nEURUSD formatted returns series\n\nRealized volatility is a measure of risk. It measures the variability in an investment over a defined period of time. It is calculated as the square root of the sum of the squared return for a particular period of time. In contrast to [implied volatility](https:\/\/www.quantstart.com\/articles\/Implied-Volatility-in-C-using-Template-Functions-and-Interval-Bisection\/), realised volatility shows the actual change in historical prices, rather than a prediction of future volatility. However, it is possible to use the data to forecast the volatility in returns.\n\nSo why is this useful? If you can reliably predict volatility for a window of future time you can make an assessment of the market regime. This can help you determine the types of trading strategies you may want to use. For example in periods of low volatility you might look at trend following strategies on stocks. In a high volatility environment you might want to consider mean reverting strategies or options trading strategies such as [Straddle or Strangle](https:\/\/www.investopedia.com\/articles\/active-trading\/040515\/how-profit-volatility.asp). By quantifying the level of risk present in your trading universe you can make better trading decisions.\n\nRealised volatility is a way of understanding the degree of price movement for a given period. It is calculated as follows:\n\n1. Collect the price data and calculate the returns\n2. Square the returns to give more weight to larger changes\n3. Calculate the average of the squared returns by adding them up and dividing by the number of periods (this is also known as the variance)\n4. Take the square root of the variance (also know as the standard deviation). This is your measure of realised volatility\n\nUltimately we will be training a Support Vector Regressor to help us identify market regime change. In order to do this we will calculate realised volatility across a particular window of time. The window you select will depend upon the frequency of you data and the liquidity of the asset. As we are considering both a major and minor FX pair we will calculate our realised volatility over 30 data points. The Pandas library contains a rolling function that can be applied to any statistical aggregator using [method chaining](https:\/\/practicaldatascience.co.uk\/data-science\/how-to-use-method-chaining-in-pandas). Here we use `rolling()` with `std()`, to calculate rolling standard deviation. The rolling window size can be set to any number of fixed observations or timedeltas, such as business days.\n\n## Calculating Realised Volatility\nThe code below will create a new column in each of the fx pairs DataFrames which will contain the realised vol calculated as the rolling standard deviation of the last 30 data points. This will equate approximately to the last 30 minutes of data, with some variation depending on the trading frequency of the FX pair. Let's take a look at the realised volatility on a chart.\n\n```\n\n```\nPython\n\nCopy\n\n![Distribution of calculated realised volatility for EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/realised_vol.png)\n\nRealised volatility for EURUSD and MXNZAR\n\nAs you can see the exotic pair MXNZAR has fewer data points than EURUSD, as it is thinly traded this is to be expected. In fact if we look at the length of both our DataFrames we can see that EURUSD has 31745 rows, whereas MXNZAR has 16865 rows almost half as many. In the previous article we discussed different options available to us that would allow us to handle the missing data points in the MXNZAR pair.\n\n## Exploring the Features of the Data\nBefore we begin to construct a machine learning model it is a good idea to have a look at the data and understand how its features could affect model performance. Why should you do this, why can't you just get going? Well, firstly the distribution of the data should influence your choice of model. For example, if your data was linearly distributed the use of linear regression model or a linear kernel in an SVM would be appropriate. If the data is non-lineaer a polynomial regression or radial basis function kernel would be more appropriate. You might also want to consider [transforming or scaling](https:\/\/scikit-learn.org\/stable\/modules\/preprocessing.html) your data to make it more compatible with your chosen model. Perharps your data contains outliers or missing data that you need to be aware of. There could be correlation (or linear association) between the features you have selected to train your model. There are many reasons why it is important to get to know your data before you start training a model. In fact it's estimated that between 70-80% of an engineer's time if spend toiling with data. Let's take a look at a simple exploratory data analysis:\n\nFirst we will look at the distribution of the returns for both our FX pairs. We use the Seaborn function [kdeplot()](https:\/\/seaborn.pydata.org\/generated\/seaborn.kdeplot.html) to visualise the distributions.\n\n```\n\n```\nPython\n\nCopy\n\n![Distribution of returns and volatility for EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/kde_rets_vol.png)\n\nReturns and realised Volatility distributions\n\nFirstly it's important to note that the scale on the x axis for EURUSD and MXNZAR is different. The range of returns is larger for MXNZAR in both the poistive and negative direction when compare to EURUSD. This is reflected in the realised volatility. We can also see that we have positive kurtosis of all distributions when compared to a standard normal distribution. The realised volatility also have positive skew. This information will be helpful when we are trying to choose or improve any machine learning models that we might want to use. Finally in the exoctic FX pair MXNZAR, we can see two small humps just before -0.01 and 0.01. This indicates the presence of spikes in our returns data.\n\nTo look into these with more detail we can use [Violin plot](https:\/\/seaborn.pydata.org\/generated\/seaborn.violinplot.html). This will allows us to see how many instances of these spikes there are in our data and how far from the bulk of the distribution they are. The following code will create the violin plot.\n\n```\n\n```\nPython\n\nCopy\n\n![Violin Plot of the returns from EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/returns_violin.png)\n\nViolin plot of returns\n\nAs you can see there are more points further from the bulk of the distribution in the MXNZAR FX pair. This means that there are more occassions where there is a higher or lower return when compared to the EURUSD. This information alone could be useful when you are considering trading strategies.\n\nAs we are planning to put our data into a machine learning model we also need to think about correlation. We want to use the realised volatility and the returns as features in a training set for a Support Vector Regressor. We need to ensure that these two features aren't correlated, otherwise we would be giving the machine learning model the same information twice. The model would become overspecified which could lead to poor model performance. Below we will look at the correlation between our two chosen features. In order to use both features we would be looking for a correlation close to zero.\n\nThe following function will display a scatter plot of the two columns; returns and realised vol. It also uses the Pandas `pd.DataFrame.corr` [method](https:\/\/pandas.pydata.org\/docs\/reference\/api\/pandas.DataFrame.corr.html) to calculate the correlation between the two.\n\n```\n\n```\nPython\n\nCopy\n\n```\n\n```\nNone\n\nCopy\n\n![Correlation between realised vol and returns for EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/correlation.png)\n\nFX pairs returns and realised vol correlation\n\nNow that we know our returns series and realised vol have little correlation we can use them as separate features in our machine learning model. In the next article we will be building a Support Vector Regressor to determine how reliably we can predict the next value of realised volatility using the returns and realised volatility values from the previous time point. This will allow us to prepare a pipeline for our data, understand some of the model parameters and prepare for the final specification of the model.\n\n## Related Articles\n- [Creating a Returns Series with Polygon's Forex Data](https:\/\/www.quantstart.com\/articles\/creating-a-returns-series-with-polygons-forex-data\/)\n\n[![Quantcademy](https:\/\/www.quantstart.com\/static\/images\/quantcademy-sidebar-advert-small.png)](https:\/\/www.quantstart.com\/quantcademy\/?ref=art)\n\n### [The Quantcademy](https:\/\/www.quantstart.com\/quantcademy\/?ref=art)\nJoin the Quantcademy membership portal that caters to the rapidly-growing retail quant trader community and learn how to increase your strategy profitability.\n\n[Find Out More](https:\/\/www.quantstart.com\/quantcademy\/?ref=art)\n\n[![Successful Algorithmic Trading](https:\/\/www.quantstart.com\/static\/images\/sat-sidebar-advert-small.png)](https:\/\/www.quantstart.com\/successful-algorithmic-trading-ebook\/)\n\n### [Successful Algorithmic Trading](https:\/\/www.quantstart.com\/successful-algorithmic-trading-ebook\/)\nHow to find new trading strategy ideas and objectively assess them for your portfolio using a Python-based backtesting engine.\n\n[Find Out More](https:\/\/www.quantstart.com\/successful-algorithmic-trading-ebook\/)\n\n[![Advanced Algorithmic Trading](https:\/\/www.quantstart.com\/static\/images\/aat-sidebar-advert-small.png)](https:\/\/www.quantstart.com\/advanced-algorithmic-trading-ebook\/)\n\n### [Advanced Algorithmic Trading](https:\/\/www.quantstart.com\/advanced-algorithmic-trading-ebook\/)\nHow to implement advanced trading strategies using time series analysis, machine learning and Bayesian statistics with R and Python.\n\n[Find Out More](https:\/\/www.quantstart.com\/advanced-algorithmic-trading-ebook\/)\n\n- QuantStart\n- [About](https:\/\/www.quantstart.com\/about\/)\n- [Articles](https:\/\/www.quantstart.com\/articles\/)\n- [Sitemap](https:\/\/www.quantstart.com\/sitemap\/)\n\n- Products\n- [Quantcademy](https:\/\/www.quantstart.com\/quantcademy\/)\n- [QSTrader](https:\/\/www.quantstart.com\/qstrader\/)\n- [Successful Algorithmic Trading](https:\/\/www.quantstart.com\/successful-algorithmic-trading-ebook\/)\n- [Advanced Algorithmic Trading](https:\/\/www.quantstart.com\/advanced-algorithmic-trading-ebook\/)\n- [C++ For Quantitative Finance](https:\/\/www.quantstart.com\/cpp-for-quantitative-finance-ebook\/)\n\n- Legal\n- [Privacy Policy](https:\/\/www.quantstart.com\/privacy-policy\/)\n- [Terms & Conditions](https:\/\/www.quantstart.com\/terms-and-conditions\/)\n\n- Social\n- [Twitter](https:\/\/twitter.com\/quantstart)\n- [YouTube](https:\/\/www.youtube.com\/channel\/UCmVnnZ6Y2TrJtY1eQJN6kWA)\n\n©2012-2026 QuarkGluon Ltd. All rights reserved.","attrs_readable_markdown":"In the previous [article](https:\/\/www.quantstart.com\/articles\/creating-a-returns-series-with-polygons-forex-data\/) we wrote a Python function which utilised the Polygon API to extract a month of minutely data for both a major (EURUSD) and exotic (MZXZAR) FX pair. We plotted the returns series and looked at some of the issues that can occur when working with this type of data. This article is part of series where we will be creating a machine learning model which uses realised volatility to predict market regime change. In this article we will be learning how to calculate realised volatility. We will also be conducting a exploratory data analysis to prepare the feature set for our machine learning model.\n\nIn order to follow along with the code in this tutorial you will need:\n\n- Python 3.9\n- Matplotlib 3.5\n- Pandas 1.4\n- Requests 2.27\n- Seaborn 0.12.\n\nWe also recommend that you take a look at our [early career research series](https:\/\/www.quantstart.com\/articles\/creating-an-algorithmic-trading-prototyping-environment-with-jupyter-notebooks-and-plotly\/) to create an algorithmic trading environment inside a Jupyter Notebook.\n\nIn order to calculate realised volatility we first need to obtain and format the data. In the previous article we created Python functions to contact the Polygon API and obtain a month of minutely data for EURUSD and MZXZAR. You can view the full article [here](https:\/\/www.quantstart.com\/articles\/creating-a-returns-series-with-polygons-forex-data\/). Below you will find the code to obtain the data.\n\n```\n\n```\nPython\n\nThe functions can be called in a new cell or in a main if running the code outside a notebook environment. The code below will store the returns value of each function in a variable.\n\n```\n\n```\nPython\n\nThe FX data for each pair can be accessed as below:\n\n```\nfx_returns_dict['C:EURUSD']\n```\nPython\n\n![Formatted DataFrame of EURUSD](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/fx_returns_dict.png)\n\nEURUSD formatted returns series\n\nRealized volatility is a measure of risk. It measures the variability in an investment over a defined period of time. It is calculated as the square root of the sum of the squared return for a particular period of time. In contrast to [implied volatility](https:\/\/www.quantstart.com\/articles\/Implied-Volatility-in-C-using-Template-Functions-and-Interval-Bisection\/), realised volatility shows the actual change in historical prices, rather than a prediction of future volatility. However, it is possible to use the data to forecast the volatility in returns.\n\nSo why is this useful? If you can reliably predict volatility for a window of future time you can make an assessment of the market regime. This can help you determine the types of trading strategies you may want to use. For example in periods of low volatility you might look at trend following strategies on stocks. In a high volatility environment you might want to consider mean reverting strategies or options trading strategies such as [Straddle or Strangle](https:\/\/www.investopedia.com\/articles\/active-trading\/040515\/how-profit-volatility.asp). By quantifying the level of risk present in your trading universe you can make better trading decisions.\n\nRealised volatility is a way of understanding the degree of price movement for a given period. It is calculated as follows:\n\n1. Collect the price data and calculate the returns\n2. Square the returns to give more weight to larger changes\n3. Calculate the average of the squared returns by adding them up and dividing by the number of periods (this is also known as the variance)\n4. Take the square root of the variance (also know as the standard deviation). This is your measure of realised volatility\n\nUltimately we will be training a Support Vector Regressor to help us identify market regime change. In order to do this we will calculate realised volatility across a particular window of time. The window you select will depend upon the frequency of you data and the liquidity of the asset. As we are considering both a major and minor FX pair we will calculate our realised volatility over 30 data points. The Pandas library contains a rolling function that can be applied to any statistical aggregator using [method chaining](https:\/\/practicaldatascience.co.uk\/data-science\/how-to-use-method-chaining-in-pandas). Here we use `rolling()` with `std()`, to calculate rolling standard deviation. The rolling window size can be set to any number of fixed observations or timedeltas, such as business days.\n\n## Calculating Realised Volatility\nThe code below will create a new column in each of the fx pairs DataFrames which will contain the realised vol calculated as the rolling standard deviation of the last 30 data points. This will equate approximately to the last 30 minutes of data, with some variation depending on the trading frequency of the FX pair. Let's take a look at the realised volatility on a chart.\n\n```\n\n```\nPython\n\n![Distribution of calculated realised volatility for EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/realised_vol.png)\n\nRealised volatility for EURUSD and MXNZAR\n\nAs you can see the exotic pair MXNZAR has fewer data points than EURUSD, as it is thinly traded this is to be expected. In fact if we look at the length of both our DataFrames we can see that EURUSD has 31745 rows, whereas MXNZAR has 16865 rows almost half as many. In the previous article we discussed different options available to us that would allow us to handle the missing data points in the MXNZAR pair.\n\n## Exploring the Features of the Data\nBefore we begin to construct a machine learning model it is a good idea to have a look at the data and understand how its features could affect model performance. Why should you do this, why can't you just get going? Well, firstly the distribution of the data should influence your choice of model. For example, if your data was linearly distributed the use of linear regression model or a linear kernel in an SVM would be appropriate. If the data is non-lineaer a polynomial regression or radial basis function kernel would be more appropriate. You might also want to consider [transforming or scaling](https:\/\/scikit-learn.org\/stable\/modules\/preprocessing.html) your data to make it more compatible with your chosen model. Perharps your data contains outliers or missing data that you need to be aware of. There could be correlation (or linear association) between the features you have selected to train your model. There are many reasons why it is important to get to know your data before you start training a model. In fact it's estimated that between 70-80% of an engineer's time if spend toiling with data. Let's take a look at a simple exploratory data analysis:\n\nFirst we will look at the distribution of the returns for both our FX pairs. We use the Seaborn function [kdeplot()](https:\/\/seaborn.pydata.org\/generated\/seaborn.kdeplot.html) to visualise the distributions.\n\n```\n\n```\nPython\n\n![Distribution of returns and volatility for EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/kde_rets_vol.png)\n\nReturns and realised Volatility distributions\n\nFirstly it's important to note that the scale on the x axis for EURUSD and MXNZAR is different. The range of returns is larger for MXNZAR in both the poistive and negative direction when compare to EURUSD. This is reflected in the realised volatility. We can also see that we have positive kurtosis of all distributions when compared to a standard normal distribution. The realised volatility also have positive skew. This information will be helpful when we are trying to choose or improve any machine learning models that we might want to use. Finally in the exoctic FX pair MXNZAR, we can see two small humps just before -0.01 and 0.01. This indicates the presence of spikes in our returns data.\n\nTo look into these with more detail we can use [Violin plot](https:\/\/seaborn.pydata.org\/generated\/seaborn.violinplot.html). This will allows us to see how many instances of these spikes there are in our data and how far from the bulk of the distribution they are. The following code will create the violin plot.\n\n```\n\n```\nPython\n\n![Violin Plot of the returns from EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/returns_violin.png)\n\nViolin plot of returns\n\nAs you can see there are more points further from the bulk of the distribution in the MXNZAR FX pair. This means that there are more occassions where there is a higher or lower return when compared to the EURUSD. This information alone could be useful when you are considering trading strategies.\n\nAs we are planning to put our data into a machine learning model we also need to think about correlation. We want to use the realised volatility and the returns as features in a training set for a Support Vector Regressor. We need to ensure that these two features aren't correlated, otherwise we would be giving the machine learning model the same information twice. The model would become overspecified which could lead to poor model performance. Below we will look at the correlation between our two chosen features. In order to use both features we would be looking for a correlation close to zero.\n\nThe following function will display a scatter plot of the two columns; returns and realised vol. It also uses the Pandas `pd.DataFrame.corr` [method](https:\/\/pandas.pydata.org\/docs\/reference\/api\/pandas.DataFrame.corr.html) to calculate the correlation between the two.\n\n```\n\n```\nPython\n\n```\n\n```\nNone\n\n![Correlation between realised vol and returns for EURUSD and MXNZAR](https:\/\/quantstartmedia.s3.amazonaws.com\/images\/article-images\/articles\/calculating-realised-volatility-with-polygon-forex-data\/correlation.png)\n\nFX pairs returns and realised vol correlation\n\nNow that we know our returns series and realised vol have little correlation we can use them as separate features in our machine learning model. In the next article we will be building a Support Vector Regressor to determine how reliably we can predict the next value of realised volatility using the returns and realised volatility values from the previous time point. This will allow us to prepare a pipeline for our data, understand some of the model parameters and prepare for the final specification of the model.\n\n## Related Articles\n- [Creating a Returns Series with Polygon's Forex Data](https:\/\/www.quantstart.com\/articles\/creating-a-returns-series-with-polygons-forex-data\/)","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

1 day ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.1 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://www.quantstart.com/articles/calculating-realised-volatility-with-polygon-forex-data/
Last Crawled	2026-04-14 08:33:41 (1 day ago)
First Indexed	2023-06-28 16:02:16 (2 years ago)
HTTP Status Code	200
Meta Title	Calculating Realised Volatility with Polygon Forex data \| QuantStart
Meta Description	In this article we carry out an exploratory data analysis of minutely Forex pairs data
Meta Canonical	null
Boilerpipe Text	In the previous article we wrote a Python function which utilised the Polygon API to extract a month of minutely data for both a major (EURUSD) and exotic (MZXZAR) FX pair. We plotted the returns series and looked at some of the issues that can occur when working with this type of data. This article is part of series where we will be creating a machine learning model which uses realised volatility to predict market regime change. In this article we will be learning how to calculate realised volatility. We will also be conducting a exploratory data analysis to prepare the feature set for our machine learning model. In order to follow along with the code in this tutorial you will need: Python 3.9 Matplotlib 3.5 Pandas 1.4 Requests 2.27 Seaborn 0.12. We also recommend that you take a look at our early career research series to create an algorithmic trading environment inside a Jupyter Notebook. In order to calculate realised volatility we first need to obtain and format the data. In the previous article we created Python functions to contact the Polygon API and obtain a month of minutely data for EURUSD and MZXZAR. You can view the full article here . Below you will find the code to obtain the data. import json import matplotlib . pyplot as plt import pandas as pd import requests import seaborn as sns POLYGON_API_KEY = os . getenv ( "POLYGON_API_KEY" ) HEADERS = { 'Authorization' : 'Bearer ' + POLYGON_API_KEY } BASE_URL = 'https://api.polygon.io/v2/aggs/' def get_fx_pairs_data ( pg_tickers ) : start = '2023-01-01' end = '2023-01-31' multiplier = '1' timespan = 'minute' fx_url = f"range/ { multiplier } / { timespan } / { start } / { end } ?adjusted=true&sort=asc&limit=50000" fx_pairs_dict = { } for pair in pg_tickers : response = requests . get ( f" { BASE_URL } ticker/ { pair } / { fx_url } " , headers = HEADERS ) . json ( ) fx_pairs_dict [ pair ] = pd . DataFrame ( response [ 'results' ] ) return fx_pairs_dict def format_fx_pairs ( fx_pair_dict ) : for pair in fx_pairs_dict . keys ( ) : fx_pairs_dict [ pair ] [ 't' ] = pd . to_datetime ( fx_pairs_dict [ pair ] [ 't' ] , unit = 'ms' ) fx_pairs_dict [ pair ] = fx_pairs_dict [ pair ] . set_index ( 't' ) return fx_pairs_dict def create_str_index ( fx_pairs_dict ) : fx_pairs_str_ind = { } for pair in fx_pairs_dict . keys ( ) : fx_pairs_str_ind [ pair ] = fx_pairs_dict [ pair ] . set_index ( fx_pairs_dict [ pair ] . index . to_series ( ) . dt . strftime ( '%Y-%m-%d-%H-%M-%S' ) ) return fx_pairs_str_ind def create_returns_series ( fx_pairs_str_ind ) : for pair in fx_pairs_str_ind . keys ( ) : fx_pairs_str_ind [ pair ] [ 'rets' ] = fx_pairs_str_ind [ pair ] [ 'c' ] . pct_change ( ) return fx_pair_str_ind Python The functions can be called in a new cell or in a main if running the code outside a notebook environment. The code below will store the returns value of each function in a variable. pg_tickers = [ "C:EURUSD" , "C:MXNZAR" ] fx_pairs_dict = get_fx_pairs_data ( pg_tickers ) formatted_fx_dict = format_fx_pairs ( fx_pairs_dict ) fx_pairs_str_ind = create_str_index ( formatted_fx_dict ) fx_returns_dict = create_returns_series ( fx_pairs_str_ind ) Python The FX data for each pair can be accessed as below: fx_returns_dict [ 'C:EURUSD' ] Python EURUSD formatted returns series Realized volatility is a measure of risk. It measures the variability in an investment over a defined period of time. It is calculated as the square root of the sum of the squared return for a particular period of time. In contrast to implied volatility , realised volatility shows the actual change in historical prices, rather than a prediction of future volatility. However, it is possible to use the data to forecast the volatility in returns. So why is this useful? If you can reliably predict volatility for a window of future time you can make an assessment of the market regime. This can help you determine the types of trading strategies you may want to use. For example in periods of low volatility you might look at trend following strategies on stocks. In a high volatility environment you might want to consider mean reverting strategies or options trading strategies such as Straddle or Strangle . By quantifying the level of risk present in your trading universe you can make better trading decisions. Realised volatility is a way of understanding the degree of price movement for a given period. It is calculated as follows: Collect the price data and calculate the returns Square the returns to give more weight to larger changes Calculate the average of the squared returns by adding them up and dividing by the number of periods (this is also known as the variance) Take the square root of the variance (also know as the standard deviation). This is your measure of realised volatility Ultimately we will be training a Support Vector Regressor to help us identify market regime change. In order to do this we will calculate realised volatility across a particular window of time. The window you select will depend upon the frequency of you data and the liquidity of the asset. As we are considering both a major and minor FX pair we will calculate our realised volatility over 30 data points. The Pandas library contains a rolling function that can be applied to any statistical aggregator using method chaining . Here we use rolling() with std() , to calculate rolling standard deviation. The rolling window size can be set to any number of fixed observations or timedeltas, such as business days. Calculating Realised Volatility The code below will create a new column in each of the fx pairs DataFrames which will contain the realised vol calculated as the rolling standard deviation of the last 30 data points. This will equate approximately to the last 30 minutes of data, with some variation depending on the trading frequency of the FX pair. Let's take a look at the realised volatility on a chart. def create_realised_vol : for pair in pairs_rets_dict : pairs_rets_dict [ pair ] [ 'realised_vol' ] = pairs_rets_dict [ pair ] [ 'rets' ] . rolling ( 30 ) . std ( ) return pairs_rets_dict Python Realised volatility for EURUSD and MXNZAR As you can see the exotic pair MXNZAR has fewer data points than EURUSD, as it is thinly traded this is to be expected. In fact if we look at the length of both our DataFrames we can see that EURUSD has 31745 rows, whereas MXNZAR has 16865 rows almost half as many. In the previous article we discussed different options available to us that would allow us to handle the missing data points in the MXNZAR pair. Exploring the Features of the Data Before we begin to construct a machine learning model it is a good idea to have a look at the data and understand how its features could affect model performance. Why should you do this, why can't you just get going? Well, firstly the distribution of the data should influence your choice of model. For example, if your data was linearly distributed the use of linear regression model or a linear kernel in an SVM would be appropriate. If the data is non-lineaer a polynomial regression or radial basis function kernel would be more appropriate. You might also want to consider transforming or scaling your data to make it more compatible with your chosen model. Perharps your data contains outliers or missing data that you need to be aware of. There could be correlation (or linear association) between the features you have selected to train your model. There are many reasons why it is important to get to know your data before you start training a model. In fact it's estimated that between 70-80% of an engineer's time if spend toiling with data. Let's take a look at a simple exploratory data analysis: First we will look at the distribution of the returns for both our FX pairs. We use the Seaborn function kdeplot() to visualise the distributions. fig , ax = plt . subplots ( 2 , 2 , figsize = ( 16 , 10 ) , squeeze = False ) y_data = [ 'rets' , 'realised_vol' ] for idx , fxpair in enumerate ( fx_pairs_str_ind . keys ( ) ) : for idx2 , dfcol in enumerate ( y_data ) : row = ( idx ) col = ( idx2 % 2 ) sns . kdeplot ( fx_pairs_str_ind [ fxpair ] [ dfcol ] , ax = ax [ row ] [ col ] , bw_adjust = 0.5 ) ax [ row ] [ col ] . set_title ( fxpair ) fig . suptitle ( "KDE of Returns & Realised Volatility" ) plt . show ( ) Python Returns and realised Volatility distributions Firstly it's important to note that the scale on the x axis for EURUSD and MXNZAR is different. The range of returns is larger for MXNZAR in both the poistive and negative direction when compare to EURUSD. This is reflected in the realised volatility. We can also see that we have positive kurtosis of all distributions when compared to a standard normal distribution. The realised volatility also have positive skew. This information will be helpful when we are trying to choose or improve any machine learning models that we might want to use. Finally in the exoctic FX pair MXNZAR, we can see two small humps just before -0.01 and 0.01. This indicates the presence of spikes in our returns data. To look into these with more detail we can use Violin plot . This will allows us to see how many instances of these spikes there are in our data and how far from the bulk of the distribution they are. The following code will create the violin plot. fig , ax = plt . subplots ( 1 , 2 , figsize = ( 16 , 10 ) , squeeze = False ) for idx , fxpair in enumerate ( fx_pairs_str_ind . keys ( ) ) : row = ( idx // 2 ) col = ( idx % 2 ) sns . violinplot ( x = 'rets' , data = fx_pairs_str_ind [ fxpair ] , ax = ax [ row ] [ col ] , inner = 'point' ) ax [ row ] [ col ] . set_title ( fxpair ) fig . suptitle ( "Violin plot of rets" ) plt . show ( ) Python Violin plot of returns As you can see there are more points further from the bulk of the distribution in the MXNZAR FX pair. This means that there are more occassions where there is a higher or lower return when compared to the EURUSD. This information alone could be useful when you are considering trading strategies. As we are planning to put our data into a machine learning model we also need to think about correlation. We want to use the realised volatility and the returns as features in a training set for a Support Vector Regressor. We need to ensure that these two features aren't correlated, otherwise we would be giving the machine learning model the same information twice. The model would become overspecified which could lead to poor model performance. Below we will look at the correlation between our two chosen features. In order to use both features we would be looking for a correlation close to zero. The following function will display a scatter plot of the two columns; returns and realised vol. It also uses the Pandas pd.DataFrame.corr method to calculate the correlation between the two. fig , ax = plt . subplots ( 1 , 2 , figsize = ( 16 , 10 ) , squeeze = False ) for idx , fxpair in enumerate ( fx_pairs_str_ind . keys ( ) ) : row = ( idx // 2 ) col = ( idx % 2 ) print ( f"vol and rets correlation for { fxpair } is " f" { fx_pairs_str_ind [ fxpair ] [ 'rets' ] . corr ( fx_pairs_str_ind [ fxpair ] [ 'realised_vol' ] ) } " ) sns . scatterplot ( data = fx_pairs_str_ind [ fxpair ] , x = 'rets' , y = 'realised_vol' , ax = ax [ row ] [ col ] ) ax [ row ] [ col ] . set_title ( fxpair ) fig . suptitle ( "Correlation between realised vol and returns" ) plt . show ( ) Python vol and rets correlation for C:EURUSD is 0.008572524273933109 vol and rets correlation for C:MXNZAR is 0.018750753882674935 None FX pairs returns and realised vol correlation Now that we know our returns series and realised vol have little correlation we can use them as separate features in our machine learning model. In the next article we will be building a Support Vector Regressor to determine how reliably we can predict the next value of realised volatility using the returns and realised volatility values from the previous time point. This will allow us to prepare a pipeline for our data, understand some of the model parameters and prepare for the final specification of the model. Related Articles Creating a Returns Series with Polygon's Forex Data
Markdown	- [QuantStart](https://www.quantstart.com/) - [Quantcademy](https://www.quantstart.com/quantcademy/) - [Books](https://www.quantstart.com/articles/calculating-realised-volatility-with-polygon-forex-data/) [Successful Algorithmic Trading](https://www.quantstart.com/successful-algorithmic-trading-ebook/) [Advanced Algorithmic Trading](https://www.quantstart.com/advanced-algorithmic-trading-ebook/) [C++ For Quantitative Finance](https://www.quantstart.com/cpp-for-quantitative-finance-ebook/) - [QSTrader](https://www.quantstart.com/qstrader/) - [Articles](https://www.quantstart.com/articles/) - [Login](https://www.quantstart.com/members/login/) - [QuantStart](https://www.quantstart.com/) - [Quantcademy](https://www.quantstart.com/quantcademy/) - [Books](https://www.quantstart.com/articles/calculating-realised-volatility-with-polygon-forex-data/) - [Successful Algorithmic Trading](https://www.quantstart.com/successful-algorithmic-trading-ebook/) - [Advanced Algorithmic Trading](https://www.quantstart.com/advanced-algorithmic-trading-ebook/) - [C++ For Quantitative Finance](https://www.quantstart.com/cpp-for-quantitative-finance-ebook/) - [QSTrader](https://www.quantstart.com/qstrader/) - [Articles](https://www.quantstart.com/articles/) - [Login](https://www.quantstart.com/members/login/) Calculating Realised Volatility with Polygon Forex data In this article we carry out an exploratory data analysis of minutely Forex pairs data In the previous [article](https://www.quantstart.com/articles/creating-a-returns-series-with-polygons-forex-data/) we wrote a Python function which utilised the Polygon API to extract a month of minutely data for both a major (EURUSD) and exotic (MZXZAR) FX pair. We plotted the returns series and looked at some of the issues that can occur when working with this type of data. This article is part of series where we will be creating a machine learning model which uses realised volatility to predict market regime change. In this article we will be learning how to calculate realised volatility. We will also be conducting a exploratory data analysis to prepare the feature set for our machine learning model. In order to follow along with the code in this tutorial you will need: - Python 3.9 - Matplotlib 3.5 - Pandas 1.4 - Requests 2.27 - Seaborn 0.12. We also recommend that you take a look at our [early career research series](https://www.quantstart.com/articles/creating-an-algorithmic-trading-prototyping-environment-with-jupyter-notebooks-and-plotly/) to create an algorithmic trading environment inside a Jupyter Notebook. ## Realised Volatility In order to calculate realised volatility we first need to obtain and format the data. In the previous article we created Python functions to contact the Polygon API and obtain a month of minutely data for EURUSD and MZXZAR. You can view the full article [here](https://www.quantstart.com/articles/creating-a-returns-series-with-polygons-forex-data/). Below you will find the code to obtain the data. ``` ``` Python Copy The functions can be called in a new cell or in a main if running the code outside a notebook environment. The code below will store the returns value of each function in a variable. ``` ``` Python Copy The FX data for each pair can be accessed as below: ``` fx_returns_dict['C:EURUSD'] ``` Python Copy ![Formatted DataFrame of EURUSD](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/fx_returns_dict.png) EURUSD formatted returns series Realized volatility is a measure of risk. It measures the variability in an investment over a defined period of time. It is calculated as the square root of the sum of the squared return for a particular period of time. In contrast to [implied volatility](https://www.quantstart.com/articles/Implied-Volatility-in-C-using-Template-Functions-and-Interval-Bisection/), realised volatility shows the actual change in historical prices, rather than a prediction of future volatility. However, it is possible to use the data to forecast the volatility in returns. So why is this useful? If you can reliably predict volatility for a window of future time you can make an assessment of the market regime. This can help you determine the types of trading strategies you may want to use. For example in periods of low volatility you might look at trend following strategies on stocks. In a high volatility environment you might want to consider mean reverting strategies or options trading strategies such as [Straddle or Strangle](https://www.investopedia.com/articles/active-trading/040515/how-profit-volatility.asp). By quantifying the level of risk present in your trading universe you can make better trading decisions. Realised volatility is a way of understanding the degree of price movement for a given period. It is calculated as follows: 1. Collect the price data and calculate the returns 2. Square the returns to give more weight to larger changes 3. Calculate the average of the squared returns by adding them up and dividing by the number of periods (this is also known as the variance) 4. Take the square root of the variance (also know as the standard deviation). This is your measure of realised volatility Ultimately we will be training a Support Vector Regressor to help us identify market regime change. In order to do this we will calculate realised volatility across a particular window of time. The window you select will depend upon the frequency of you data and the liquidity of the asset. As we are considering both a major and minor FX pair we will calculate our realised volatility over 30 data points. The Pandas library contains a rolling function that can be applied to any statistical aggregator using [method chaining](https://practicaldatascience.co.uk/data-science/how-to-use-method-chaining-in-pandas). Here we use `rolling()` with `std()`, to calculate rolling standard deviation. The rolling window size can be set to any number of fixed observations or timedeltas, such as business days. ## Calculating Realised Volatility The code below will create a new column in each of the fx pairs DataFrames which will contain the realised vol calculated as the rolling standard deviation of the last 30 data points. This will equate approximately to the last 30 minutes of data, with some variation depending on the trading frequency of the FX pair. Let's take a look at the realised volatility on a chart. ``` ``` Python Copy ![Distribution of calculated realised volatility for EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/realised_vol.png) Realised volatility for EURUSD and MXNZAR As you can see the exotic pair MXNZAR has fewer data points than EURUSD, as it is thinly traded this is to be expected. In fact if we look at the length of both our DataFrames we can see that EURUSD has 31745 rows, whereas MXNZAR has 16865 rows almost half as many. In the previous article we discussed different options available to us that would allow us to handle the missing data points in the MXNZAR pair. ## Exploring the Features of the Data Before we begin to construct a machine learning model it is a good idea to have a look at the data and understand how its features could affect model performance. Why should you do this, why can't you just get going? Well, firstly the distribution of the data should influence your choice of model. For example, if your data was linearly distributed the use of linear regression model or a linear kernel in an SVM would be appropriate. If the data is non-lineaer a polynomial regression or radial basis function kernel would be more appropriate. You might also want to consider [transforming or scaling](https://scikit-learn.org/stable/modules/preprocessing.html) your data to make it more compatible with your chosen model. Perharps your data contains outliers or missing data that you need to be aware of. There could be correlation (or linear association) between the features you have selected to train your model. There are many reasons why it is important to get to know your data before you start training a model. In fact it's estimated that between 70-80% of an engineer's time if spend toiling with data. Let's take a look at a simple exploratory data analysis: First we will look at the distribution of the returns for both our FX pairs. We use the Seaborn function [kdeplot()](https://seaborn.pydata.org/generated/seaborn.kdeplot.html) to visualise the distributions. ``` ``` Python Copy ![Distribution of returns and volatility for EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/kde_rets_vol.png) Returns and realised Volatility distributions Firstly it's important to note that the scale on the x axis for EURUSD and MXNZAR is different. The range of returns is larger for MXNZAR in both the poistive and negative direction when compare to EURUSD. This is reflected in the realised volatility. We can also see that we have positive kurtosis of all distributions when compared to a standard normal distribution. The realised volatility also have positive skew. This information will be helpful when we are trying to choose or improve any machine learning models that we might want to use. Finally in the exoctic FX pair MXNZAR, we can see two small humps just before -0.01 and 0.01. This indicates the presence of spikes in our returns data. To look into these with more detail we can use [Violin plot](https://seaborn.pydata.org/generated/seaborn.violinplot.html). This will allows us to see how many instances of these spikes there are in our data and how far from the bulk of the distribution they are. The following code will create the violin plot. ``` ``` Python Copy ![Violin Plot of the returns from EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/returns_violin.png) Violin plot of returns As you can see there are more points further from the bulk of the distribution in the MXNZAR FX pair. This means that there are more occassions where there is a higher or lower return when compared to the EURUSD. This information alone could be useful when you are considering trading strategies. As we are planning to put our data into a machine learning model we also need to think about correlation. We want to use the realised volatility and the returns as features in a training set for a Support Vector Regressor. We need to ensure that these two features aren't correlated, otherwise we would be giving the machine learning model the same information twice. The model would become overspecified which could lead to poor model performance. Below we will look at the correlation between our two chosen features. In order to use both features we would be looking for a correlation close to zero. The following function will display a scatter plot of the two columns; returns and realised vol. It also uses the Pandas `pd.DataFrame.corr` [method](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html) to calculate the correlation between the two. ``` ``` Python Copy ``` ``` None Copy ![Correlation between realised vol and returns for EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/correlation.png) FX pairs returns and realised vol correlation Now that we know our returns series and realised vol have little correlation we can use them as separate features in our machine learning model. In the next article we will be building a Support Vector Regressor to determine how reliably we can predict the next value of realised volatility using the returns and realised volatility values from the previous time point. This will allow us to prepare a pipeline for our data, understand some of the model parameters and prepare for the final specification of the model. ## Related Articles - [Creating a Returns Series with Polygon's Forex Data](https://www.quantstart.com/articles/creating-a-returns-series-with-polygons-forex-data/) [![Quantcademy](https://www.quantstart.com/static/images/quantcademy-sidebar-advert-small.png)](https://www.quantstart.com/quantcademy/?ref=art) ### [The Quantcademy](https://www.quantstart.com/quantcademy/?ref=art) Join the Quantcademy membership portal that caters to the rapidly-growing retail quant trader community and learn how to increase your strategy profitability. [Find Out More](https://www.quantstart.com/quantcademy/?ref=art) [![Successful Algorithmic Trading](https://www.quantstart.com/static/images/sat-sidebar-advert-small.png)](https://www.quantstart.com/successful-algorithmic-trading-ebook/) ### [Successful Algorithmic Trading](https://www.quantstart.com/successful-algorithmic-trading-ebook/) How to find new trading strategy ideas and objectively assess them for your portfolio using a Python-based backtesting engine. [Find Out More](https://www.quantstart.com/successful-algorithmic-trading-ebook/) [![Advanced Algorithmic Trading](https://www.quantstart.com/static/images/aat-sidebar-advert-small.png)](https://www.quantstart.com/advanced-algorithmic-trading-ebook/) ### [Advanced Algorithmic Trading](https://www.quantstart.com/advanced-algorithmic-trading-ebook/) How to implement advanced trading strategies using time series analysis, machine learning and Bayesian statistics with R and Python. [Find Out More](https://www.quantstart.com/advanced-algorithmic-trading-ebook/) - QuantStart - [About](https://www.quantstart.com/about/) - [Articles](https://www.quantstart.com/articles/) - [Sitemap](https://www.quantstart.com/sitemap/) - Products - [Quantcademy](https://www.quantstart.com/quantcademy/) - [QSTrader](https://www.quantstart.com/qstrader/) - [Successful Algorithmic Trading](https://www.quantstart.com/successful-algorithmic-trading-ebook/) - [Advanced Algorithmic Trading](https://www.quantstart.com/advanced-algorithmic-trading-ebook/) - [C++ For Quantitative Finance](https://www.quantstart.com/cpp-for-quantitative-finance-ebook/) - Legal - [Privacy Policy](https://www.quantstart.com/privacy-policy/) - [Terms & Conditions](https://www.quantstart.com/terms-and-conditions/) - Social - [Twitter](https://twitter.com/quantstart) - [YouTube](https://www.youtube.com/channel/UCmVnnZ6Y2TrJtY1eQJN6kWA) ©2012-2026 QuarkGluon Ltd. All rights reserved.
Readable Markdown	In the previous [article](https://www.quantstart.com/articles/creating-a-returns-series-with-polygons-forex-data/) we wrote a Python function which utilised the Polygon API to extract a month of minutely data for both a major (EURUSD) and exotic (MZXZAR) FX pair. We plotted the returns series and looked at some of the issues that can occur when working with this type of data. This article is part of series where we will be creating a machine learning model which uses realised volatility to predict market regime change. In this article we will be learning how to calculate realised volatility. We will also be conducting a exploratory data analysis to prepare the feature set for our machine learning model. In order to follow along with the code in this tutorial you will need: - Python 3.9 - Matplotlib 3.5 - Pandas 1.4 - Requests 2.27 - Seaborn 0.12. We also recommend that you take a look at our [early career research series](https://www.quantstart.com/articles/creating-an-algorithmic-trading-prototyping-environment-with-jupyter-notebooks-and-plotly/) to create an algorithmic trading environment inside a Jupyter Notebook. In order to calculate realised volatility we first need to obtain and format the data. In the previous article we created Python functions to contact the Polygon API and obtain a month of minutely data for EURUSD and MZXZAR. You can view the full article [here](https://www.quantstart.com/articles/creating-a-returns-series-with-polygons-forex-data/). Below you will find the code to obtain the data. ``` ``` Python The functions can be called in a new cell or in a main if running the code outside a notebook environment. The code below will store the returns value of each function in a variable. ``` ``` Python The FX data for each pair can be accessed as below: ``` fx_returns_dict['C:EURUSD'] ``` Python ![Formatted DataFrame of EURUSD](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/fx_returns_dict.png) EURUSD formatted returns series Realized volatility is a measure of risk. It measures the variability in an investment over a defined period of time. It is calculated as the square root of the sum of the squared return for a particular period of time. In contrast to [implied volatility](https://www.quantstart.com/articles/Implied-Volatility-in-C-using-Template-Functions-and-Interval-Bisection/), realised volatility shows the actual change in historical prices, rather than a prediction of future volatility. However, it is possible to use the data to forecast the volatility in returns. So why is this useful? If you can reliably predict volatility for a window of future time you can make an assessment of the market regime. This can help you determine the types of trading strategies you may want to use. For example in periods of low volatility you might look at trend following strategies on stocks. In a high volatility environment you might want to consider mean reverting strategies or options trading strategies such as [Straddle or Strangle](https://www.investopedia.com/articles/active-trading/040515/how-profit-volatility.asp). By quantifying the level of risk present in your trading universe you can make better trading decisions. Realised volatility is a way of understanding the degree of price movement for a given period. It is calculated as follows: 1. Collect the price data and calculate the returns 2. Square the returns to give more weight to larger changes 3. Calculate the average of the squared returns by adding them up and dividing by the number of periods (this is also known as the variance) 4. Take the square root of the variance (also know as the standard deviation). This is your measure of realised volatility Ultimately we will be training a Support Vector Regressor to help us identify market regime change. In order to do this we will calculate realised volatility across a particular window of time. The window you select will depend upon the frequency of you data and the liquidity of the asset. As we are considering both a major and minor FX pair we will calculate our realised volatility over 30 data points. The Pandas library contains a rolling function that can be applied to any statistical aggregator using [method chaining](https://practicaldatascience.co.uk/data-science/how-to-use-method-chaining-in-pandas). Here we use `rolling()` with `std()`, to calculate rolling standard deviation. The rolling window size can be set to any number of fixed observations or timedeltas, such as business days. ## Calculating Realised Volatility The code below will create a new column in each of the fx pairs DataFrames which will contain the realised vol calculated as the rolling standard deviation of the last 30 data points. This will equate approximately to the last 30 minutes of data, with some variation depending on the trading frequency of the FX pair. Let's take a look at the realised volatility on a chart. ``` ``` Python ![Distribution of calculated realised volatility for EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/realised_vol.png) Realised volatility for EURUSD and MXNZAR As you can see the exotic pair MXNZAR has fewer data points than EURUSD, as it is thinly traded this is to be expected. In fact if we look at the length of both our DataFrames we can see that EURUSD has 31745 rows, whereas MXNZAR has 16865 rows almost half as many. In the previous article we discussed different options available to us that would allow us to handle the missing data points in the MXNZAR pair. ## Exploring the Features of the Data Before we begin to construct a machine learning model it is a good idea to have a look at the data and understand how its features could affect model performance. Why should you do this, why can't you just get going? Well, firstly the distribution of the data should influence your choice of model. For example, if your data was linearly distributed the use of linear regression model or a linear kernel in an SVM would be appropriate. If the data is non-lineaer a polynomial regression or radial basis function kernel would be more appropriate. You might also want to consider [transforming or scaling](https://scikit-learn.org/stable/modules/preprocessing.html) your data to make it more compatible with your chosen model. Perharps your data contains outliers or missing data that you need to be aware of. There could be correlation (or linear association) between the features you have selected to train your model. There are many reasons why it is important to get to know your data before you start training a model. In fact it's estimated that between 70-80% of an engineer's time if spend toiling with data. Let's take a look at a simple exploratory data analysis: First we will look at the distribution of the returns for both our FX pairs. We use the Seaborn function [kdeplot()](https://seaborn.pydata.org/generated/seaborn.kdeplot.html) to visualise the distributions. ``` ``` Python ![Distribution of returns and volatility for EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/kde_rets_vol.png) Returns and realised Volatility distributions Firstly it's important to note that the scale on the x axis for EURUSD and MXNZAR is different. The range of returns is larger for MXNZAR in both the poistive and negative direction when compare to EURUSD. This is reflected in the realised volatility. We can also see that we have positive kurtosis of all distributions when compared to a standard normal distribution. The realised volatility also have positive skew. This information will be helpful when we are trying to choose or improve any machine learning models that we might want to use. Finally in the exoctic FX pair MXNZAR, we can see two small humps just before -0.01 and 0.01. This indicates the presence of spikes in our returns data. To look into these with more detail we can use [Violin plot](https://seaborn.pydata.org/generated/seaborn.violinplot.html). This will allows us to see how many instances of these spikes there are in our data and how far from the bulk of the distribution they are. The following code will create the violin plot. ``` ``` Python ![Violin Plot of the returns from EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/returns_violin.png) Violin plot of returns As you can see there are more points further from the bulk of the distribution in the MXNZAR FX pair. This means that there are more occassions where there is a higher or lower return when compared to the EURUSD. This information alone could be useful when you are considering trading strategies. As we are planning to put our data into a machine learning model we also need to think about correlation. We want to use the realised volatility and the returns as features in a training set for a Support Vector Regressor. We need to ensure that these two features aren't correlated, otherwise we would be giving the machine learning model the same information twice. The model would become overspecified which could lead to poor model performance. Below we will look at the correlation between our two chosen features. In order to use both features we would be looking for a correlation close to zero. The following function will display a scatter plot of the two columns; returns and realised vol. It also uses the Pandas `pd.DataFrame.corr` [method](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html) to calculate the correlation between the two. ``` ``` Python ``` ``` None ![Correlation between realised vol and returns for EURUSD and MXNZAR](https://quantstartmedia.s3.amazonaws.com/images/article-images/articles/calculating-realised-volatility-with-polygon-forex-data/correlation.png) FX pairs returns and realised vol correlation Now that we know our returns series and realised vol have little correlation we can use them as separate features in our machine learning model. In the next article we will be building a Support Vector Regressor to determine how reliably we can predict the next value of realised volatility using the returns and realised volatility values from the previous time point. This will allow us to prepare a pipeline for our data, understand some of the model parameters and prepare for the final specification of the model. ## Related Articles - [Creating a Returns Series with Polygon's Forex Data](https://www.quantstart.com/articles/creating-a-returns-series-with-polygons-forex-data/)
Shard	57 (laksa)
Root Hash	15201169621511058857
Unparsed URL	com,quantstart!www,/articles/calculating-realised-volatility-with-polygon-forex-data/ s443