🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 37 (from laksa184)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa037.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://apmonitor.com/pds/index.php/Main/CleanseData\')), getAhrefsUnparsedNoserviceFromURL(\'https://apmonitor.com/pds/index.php/Main/CleanseData\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData","crawl_time":1775349048,"first_indexed_time":1613091513,"http_code":200,"src_unparsed":"com,apmonitor!\/pds\/index.php\/Main\/CleanseData s443","src_root_hash":"13934051142679362237","history_drop_reason":null,"meta_title":"Data Cleansing","meta_descriptions":["Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data."],"attrs_boilerpipe_text":"Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data.\nData cleansing is the process of removing bad data that may include outliers, missing entries, failed sensors, or other types of missing or corrupted information.\nBad data can be detected with\nsummary statistics\nand\ndata visualization\n. An effective way to remove bad data is with filters that segregate based on conditions that remove outliers, replace bad values such as\nNaN\n(Not a Number), or remove the data row that contains an\nNaN\nvalue.\nRemove Bad Data with Numpy\nNaN values are removed with numpy by identifying rows\nix\nthat contain NaN. Next, the rows are removed with\nz=z[~iz]\nwhere ~ is a bitwise not operator.\nimport\nnumpy\nas\nnp\nz\n=\nnp.\narray\n(\n[\n[\n     \n1\n,\n     \n2\n]\n,\n             \n[\nnp.\nnan\n,\n     \n3\n]\n,\n             \n[\n     \n4\n,\nnp.\nnan\n]\n,\n             \n[\n     \n5\n,\n     \n6\n]\n]\n)\niz\n=\nnp.\nany\n(\nnp.\nisnan\n(\nz\n)\n,\naxis\n=\n1\n)\nprint\n(\n~\niz\n)\nz\n=\nz\n[\n~\niz\n]\nprint\n(\nz\n)\nThe result of the selection\niz\nis\nTrue\nif\nNaN\nis found anywhere in the row of\nFalse\nif\nNaN\nis not found in that row. The ~ is the not operator to reverse\nTrue\nand\nFalse\n.\nprint(~iz)\n\n   > [ True False False  True]\nOnly the rows with\nTrue\nare kept in the final\nz\nresult.\nprint(z)\n\n   > [[1. 2.]\n   >  [5. 6.]]\nRemove Bad Data with Pandas\nPandas\nmanipulates data tables called DataFrames. There are many functions to efficiently manipulate data and remove bad data.\nimport\nnumpy\nas\nnp\nimport\npandas\nas\npd\nz\n=\npd.\nDataFrame\n(\n{\n'x'\n:\n[\n1\n,\nnp.\nnan\n,\n4\n,\n5\n]\n,\n'y'\n:\n[\n2\n,\n3\n,\nnp.\nnan\n,\n6\n]\n}\n)\nprint\n(\nz\n)\nThis produces the same values as shown with the Numpy example.\nx    y\n   0  1.0  2.0\n   1  NaN  3.0\n   2  4.0  NaN\n   3  5.0  6.0\nThere are two common ways to deal with\nNaN\nvalues: drop the rows or fill in values. The first is to remove the rows with\ndropna\n. Use\ninplace=True\nto avoid the extra assignment of\nresult=z.dropna()\nto modify z directly.\nresult\n=\nz.\ndropna\n(\n)\nx    y\n   0  1.0  2.0\n   3  5.0  6.0\nReplace Bad Data with Pandas\nIf data is very limited then it may be better to keep the row and fill-in values with methods\ninterpolate\nfor time-series data or\nfillna\nto replace\nNaN\nwith a default value.\nresult\n=\nz.\nfillna\n(\nz.\nmean\n(\n)\n)\nA common replacement is the mean value for each column with\nz.mean()\n. In this case, the mean (average) of column\nx\nis 3.333 and column\ny\nis 3.667.\nx         y\n   0  1.000000  2.000000\n   1  3.333333  3.000000\n   2  4.000000  3.666667\n   3  5.000000  6.000000\nFilters with Pandas\nData visualization\ncan help identify outliers, especially with box plots. Statistical information such as standard deviation can also help to identify outliers such as eliminating data points that are more than 5 standard deviations away from the mean. A conditional filter can be created to eliminate bad values. The conditional statement\nz['y']<5.5\ncreates a Logical array of\nTrue\nfor\nz['y']<5.5\nand\nFalse\nfor\nz['y']>=5.5\n.\nresult\n=\nz\n[\nz\n[\n'y'\n]\n<\n5.5\n]\nThis filter eliminates the last row of the DataFrame and the\nNaN\nin that column.\nx    y\n   0  1.0  2.0\n   1  NaN  3.0\nFilters can be combined with the\nand\nbitwise operator\n&\nor the\nor\nbitwise operator\n|\n.\nresult\n=\nz\n[\n(\nz\n[\n'y'\n]\n<\n5.5\n)\n&\n(\nz\n[\n'y'\n]\n>=\n1.0\n)\n]\nAnother way to combine filters is to operate on the object with successive methods such as\nz['x'].notnull()\nto eliminate\nNaN\nvalues in the\nx\ncolumn, the\n.fillna(0)\nto replace\nNaN\nwith zero in the\ny\ncolumn, and\n.reset_index(drop=True)\nto reset the DataFrame index.\nresult\n=\nz\n[\nz\n[\n'x'\n]\n.\nnotnull\n(\n)\n]\n.\nfillna\n(\n0\n)\n.\nreset_index\n(\ndrop\n=\nTrue\n)\nx    y\n   0  1.0  2.0\n   1  4.0  0.0\n   2  5.0  6.0\nFurther Reading\nBrownlee, J.,\nHow to Remove Outliers for Machine Learning\n, April 2018.\n✅ Knowledge Check\n1.\nWhat is the primary purpose of data cleansing?\nA.\nTo add bad data into machine learning models.\nIncorrect.\nData cleansing aims to remove bad data, not add it.\nB.\nTo identify and remove bad data.\nCorrect.\nData cleansing is about removing bad data such as outliers, missing entries, and corrupted information.\nC.\nTo improve the efficiency of data storage.\nIncorrect.\nWhile data cleansing can reduce storage needs by removing unnecessary data, its primary purpose is to improve data quality for analysis.\nD.\nTo make data visualization more attractive.\nIncorrect.\nWhile clean data can lead to clearer visualizations, the main goal of data cleansing is to improve data quality for analysis.\n2.\nHow can NaN values be removed from a Numpy array?\nA.\nBy using the dropna() method.\nIncorrect.\nThe dropna() method is associated with Pandas, not Numpy.\nB.\nBy replacing NaN values with the mean of the array.\nIncorrect.\nWhile this method is used to replace NaN values, it doesn't remove them.\nC.\nBy using the command: z=z[~iz] where iz is the selection of NaN rows.\nCorrect.\nThis method uses bitwise not operator (~) and a selection to remove rows containing NaN values from a Numpy array.\nD.\nBy using the fillna() method with the parameter 0.\nIncorrect.\nThe fillna() method is associated with Pandas, and it replaces NaN values rather than removing them.","attrs_markdown":"# [Machine Learning for Engineers](https:\/\/apmonitor.com\/pds\/index.php)\n- [Home](https:\/\/apmonitor.com\/pds\/index.php\/Main\/HomePage)\n- [Schedule](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CourseSchedule)\n\n- [View](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData)\n- [Edit](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=edit)\n- [History](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=diff)\n- [Print](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=print)\n\n# [Data Cleansing](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData)\nMeasurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data.\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/analyze.png)\n\nData cleansing is the process of removing bad data that may include outliers, missing entries, failed sensors, or other types of missing or corrupted information.\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/python.png) [Data Cleansing Python Jupyter Notebook](https:\/\/github.com\/APMonitor\/pds\/blob\/main\/Cleanse_Data.ipynb)\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/colab.png) [Jupyter Notebook in Google Colab](https:\/\/colab.research.google.com\/github\/APMonitor\/pds\/blob\/main\/Cleanse_Data.ipynb)\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/matlab.png) [Data Cleansing MATLAB Live Script](https:\/\/github.com\/APMonitor\/mds\/blob\/main\/Cleanse.mlx)\n\nBad data can be detected with [summary statistics](https:\/\/apmonitor.com\/pds\/index.php\/Main\/StatisticsMath) and [data visualization](https:\/\/apmonitor.com\/pds\/index.php\/Main\/VisualizeData). An effective way to remove bad data is with filters that segregate based on conditions that remove outliers, replace bad values such as *NaN* (Not a Number), or remove the data row that contains an *NaN* value.\n***\n#### Remove Bad Data with Numpy\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/python_numpy.png)\n\nNaN values are removed with numpy by identifying rows *ix* that contain NaN. Next, the rows are removed with *z=z\\[~iz\\]* where ~ is a bitwise not operator.\n\nimport numpy as np  \n z \\= np.array(\\[\\[ 1, 2\\],  \n \\[ np.nan, 3\\],  \n \\[ 4, np.nan\\],  \n \\[ 5, 6\\]\\])  \n iz \\= np.any(np.isnan(z), axis\\=1)  \n print(~iz)  \n z \\= z\\[~iz\\]  \n print(z)\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=1)\n\nThe result of the selection *iz* is *True* if *NaN* is found anywhere in the row of *False* if *NaN* is not found in that row. The ~ is the not operator to reverse *True* and *False*.\n```\n   print(~iz)\n\n   > [ True False False  True]\n```\nOnly the rows with *True* are kept in the final *z* result.\n```\n   print(z)\n\n   > [[1. 2.]\n   >  [5. 6.]]\n```\n***\n#### Remove Bad Data with Pandas\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/python_pandas.png)\n\n[Pandas](https:\/\/pandas.pydata.org\/) manipulates data tables called DataFrames. There are many functions to efficiently manipulate data and remove bad data.\n\nimport numpy as np  \n import pandas as pd  \n z \\= pd.DataFrame({'x':\\[1,np.nan,4,5\\],'y':\\[2,3,np.nan,6\\]})  \n print(z)\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=2)\n\nThis produces the same values as shown with the Numpy example.\n```\n        x    y\n   0  1.0  2.0\n   1  NaN  3.0\n   2  4.0  NaN\n   3  5.0  6.0\n```\nThere are two common ways to deal with *NaN* values: drop the rows or fill in values. The first is to remove the rows with *dropna*. Use *inplace=True* to avoid the extra assignment of *result=z.dropna()* to modify z directly.\n\nresult \\= z.dropna()\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=3)\n```\n        x    y\n   0  1.0  2.0\n   3  5.0  6.0\n```\n#### Replace Bad Data with Pandas\nIf data is very limited then it may be better to keep the row and fill-in values with methods *interpolate* for time-series data or *fillna* to replace *NaN* with a default value.\n\nresult \\= z.fillna(z.mean())\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=4)\n\nA common replacement is the mean value for each column with *z.mean()*. In this case, the mean (average) of column *x* is 3.333 and column *y* is 3.667.\n```\n             x         y\n   0  1.000000  2.000000\n   1  3.333333  3.000000\n   2  4.000000  3.666667\n   3  5.000000  6.000000\n```\n#### Filters with Pandas\n[Data visualization](https:\/\/apmonitor.com\/pds\/index.php\/Main\/VisualizeData) can help identify outliers, especially with box plots. Statistical information such as standard deviation can also help to identify outliers such as eliminating data points that are more than 5 standard deviations away from the mean. A conditional filter can be created to eliminate bad values. The conditional statement *z\\['y'\\]\\<5.5* creates a Logical array of *True* for *z\\['y'\\]\\<5.5* and *False* for *z\\['y'\\]\\>=5.5*.\n\nresult \\= z\\[z\\['y'\\]\\<5\\.5\\]\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=5)\n\nThis filter eliminates the last row of the DataFrame and the *NaN* in that column.\n```\n        x    y\n   0  1.0  2.0\n   1  NaN  3.0\n```\nFilters can be combined with the *and* bitwise operator *&* or the *or* bitwise operator *\\|*.\n\nresult \\= z\\[(z\\['y'\\]\\<5\\.5) & (z\\['y'\\]\\>=1\\.0)\\]\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=6)\n\nAnother way to combine filters is to operate on the object with successive methods such as *z\\['x'\\].notnull()* to eliminate *NaN* values in the *x* column, the *.fillna(0)* to replace *NaN* with zero in the *y* column, and *.reset\\_index(drop=True)* to reset the DataFrame index.\n\nresult \\= z\\[z\\['x'\\].notnull()\\].fillna(0).reset\\_index(drop\\=True)\n\n[\\[\\$\\[Get Code\\]\\]](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData?action=sourceblock&num=7)\n```\n        x    y\n   0  1.0  2.0\n   1  4.0  0.0\n   2  5.0  6.0\n```\n#### Further Reading\n- Brownlee, J., [How to Remove Outliers for Machine Learning](https:\/\/machinelearningmastery.com\/how-to-use-statistics-to-identify-outliers-in-data\/), April 2018.\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/data_cleansing.png)\n***\n#### ✅ Knowledge Check\n**1\\.** What is the primary purpose of data cleansing?\n\n**A.** To add bad data into machine learning models.\n\nIncorrect. Data cleansing aims to remove bad data, not add it.\n\n**B.** To identify and remove bad data.\n\nCorrect. Data cleansing is about removing bad data such as outliers, missing entries, and corrupted information.\n\n**C.** To improve the efficiency of data storage.\n\nIncorrect. While data cleansing can reduce storage needs by removing unnecessary data, its primary purpose is to improve data quality for analysis.\n\n**D.** To make data visualization more attractive.\n\nIncorrect. While clean data can lead to clearer visualizations, the main goal of data cleansing is to improve data quality for analysis.\n\n**2\\.** How can NaN values be removed from a Numpy array?\n\n**A.** By using the dropna() method.\n\nIncorrect. The dropna() method is associated with Pandas, not Numpy.\n\n**B.** By replacing NaN values with the mean of the array.\n\nIncorrect. While this method is used to replace NaN values, it doesn't remove them.\n\n**C.** By using the command: z=z\\[~iz\\] where iz is the selection of NaN rows.\n\nCorrect. This method uses bitwise not operator (~) and a selection to remove rows containing NaN values from a Numpy array.\n\n**D.** By using the fillna() method with the parameter 0.\n\nIncorrect. The fillna() method is associated with Pandas, and it replaces NaN values rather than removing them.\n\nCourse on ![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/github.png) [GitHub](https:\/\/github.com\/APMonitor\/pds)\n\n- [Overview](https:\/\/apmonitor.com\/pds\/index.php\/Main\/HomePage)\n- [Syllabus](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CourseSyllabus)\n- [Schedule](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CourseSchedule)\n- [4-Day Course](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ShortCourse)\n- [Course Project](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CourseProject)\n- [Startup Incubator](https:\/\/apmonitor.com\/pds\/index.php\/Main\/StartupIncubator)\n- [Resources](https:\/\/apmonitor.com\/pds\/index.php\/Main\/LearningResources)\n- [Install Python](https:\/\/apmonitor.com\/pds\/index.php\/Main\/InstallPython)\n- [Install Packages](https:\/\/apmonitor.com\/pds\/index.php\/Main\/InstallPythonPackages)\n- [AI Ethics](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AIEthics)\n\nExams\n\n- [Midterm Exam](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CourseExam1)\n\nData Engineering\n\n- [Overview](https:\/\/apmonitor.com\/pds\/index.php\/Main\/DataPreparation)\n- [1️⃣ Gather Data](https:\/\/apmonitor.com\/pds\/index.php\/Main\/GatherData)\n- [2️⃣ Statistics](https:\/\/apmonitor.com\/pds\/index.php\/Main\/StatisticsMath)\n- [3️⃣ Visualize](https:\/\/apmonitor.com\/pds\/index.php\/Main\/VisualizeData)\n- [4️⃣ Cleanse](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CleanseData)\n- [5️⃣ Features](https:\/\/apmonitor.com\/pds\/index.php\/Main\/FeatureEngineering)\n- [6️⃣ Balance](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ImbalancedData)\n- [7️⃣ Scale](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ScaleData)\n- [8️⃣ Split](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SplitData)\n- [9️⃣ Deploy](https:\/\/apmonitor.com\/pds\/index.php\/Main\/DeployMachineLearning)\n- [🔟 Apps](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AppDevelopment)\n\nAgentic Engineering\n\n- [1️⃣ Workplan](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AgenticWorkplan)\n- [2️⃣ Coding](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AgenticCoding)\n- [3️⃣ Coordination](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AgenticCoordination)\n- [4️⃣ Visualization](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AgenticVisualization)\n- [5️⃣ Communication](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AgenticReports)\n\nClassification\n\n- [Overview](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ClassificationOverview)\n- [Hyperparameters](https:\/\/apmonitor.com\/pds\/index.php\/Main\/HyperparameterOptimization)\n- [Cybersecurity](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CyberSecurity)\n\n**Supervised Learning**\n\n- [AdaBoost](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AdaBoost)\n- [Decision Tree](https:\/\/apmonitor.com\/pds\/index.php\/Main\/DecisionTree)\n- [k-Nearest Neighbors](https:\/\/apmonitor.com\/pds\/index.php\/Main\/KNearestNeighbors)\n- [Logistic Regression](https:\/\/apmonitor.com\/pds\/index.php\/Main\/LogisticRegression)\n- [Naïve Bayes](https:\/\/apmonitor.com\/pds\/index.php\/Main\/NaiveBayes)\n- [Neural Network Classifier](https:\/\/apmonitor.com\/pds\/index.php\/Main\/DeepLearningNeuralNetwork)\n- [Random Forest](https:\/\/apmonitor.com\/pds\/index.php\/Main\/RandomForest)\n- [Stochastic Gradient Descent](https:\/\/apmonitor.com\/pds\/index.php\/Main\/StochasticGradientDescent)\n- [Support Vector Classifier](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SupportVectorClassifier)\n- [XGBoost Classifier](https:\/\/apmonitor.com\/pds\/index.php\/Main\/XGBoostClassifier)\n\n**Unsupervised Learning**\n\n- [Gaussian Mixture Model](https:\/\/apmonitor.com\/pds\/index.php\/Main\/GaussianMixtureModel)\n- [K-Means Clustering](https:\/\/apmonitor.com\/pds\/index.php\/Main\/KMeansClustering)\n- [Spectral Clustering](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SpectralClustering)\n\nRegression\n\n- [Overview](https:\/\/apmonitor.com\/pds\/index.php\/Main\/RegressionOverview)\n- [Linear Regression](https:\/\/apmonitor.com\/pds\/index.php\/Main\/LinearRegression)\n- [k-Nearest Neighbors](https:\/\/apmonitor.com\/pds\/index.php\/Main\/KNearestNeighborsRegression)\n- [Support Vector Regressor](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SupportVectorRegressor)\n- [Gaussian Processes](https:\/\/apmonitor.com\/pds\/index.php\/Main\/GaussianProcessRegression)\n- [Neural Network Regressor](https:\/\/apmonitor.com\/pds\/index.php\/Main\/MultilayerPerceptronNeuralNetwork)\n- [XGBoost Regressor](https:\/\/apmonitor.com\/pds\/index.php\/Main\/XGBoostRegressor)\n\nTime-Series\n\n- [ARX Model](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ARXTimeSeries)\n- [LSTM Network](https:\/\/apmonitor.com\/pds\/index.php\/Main\/LongShortTermMemory)\n\nComputer Vision\n\n- [Introduction](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ComputerVisionIntro)\n- [Cascade Classifier](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CascadeClassifier)\n- [Deep Learning](https:\/\/apmonitor.com\/pds\/index.php\/Main\/VisionDeepLearning)\n\nApplications\n\n[3D Print](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AdditiveManufacturing) 📈📊\n\n[Automation LSTM](https:\/\/apmonitor.com\/pds\/index.php\/Main\/LSTMAutomation) ⏱️\n\n[Automotive](https:\/\/apmonitor.com\/pds\/index.php\/Main\/AutomotiveMonitoring) 📈📊\n\n[Battery Life](https:\/\/apmonitor.com\/pds\/index.php\/Main\/BatteryLife)⏱️📈\n\n[Bit Classification](https:\/\/apmonitor.com\/pds\/index.php\/Main\/BitClassification) 👁️📊\n\n[Concrete Strength](https:\/\/apmonitor.com\/pds\/index.php\/Main\/CementStrength) 📈📊\n\n[Chemical Properties](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ThermophysicalProperties) 📈\n\n[Draw Classification](https:\/\/apmonitor.com\/pds\/index.php\/Main\/DrawClassification) 📊\n\n[Facial Recognition](https:\/\/apmonitor.com\/pds\/index.php\/Main\/FacialRecognition) 👁️📊\n\n[Glass Type](https:\/\/apmonitor.com\/pds\/index.php\/Main\/GlassCharacterization)⏱️📈\n\n[Hand Tracking](https:\/\/apmonitor.com\/pds\/index.php\/Main\/HandTracking) 👁️\n\n[OT Cybersecurity](https:\/\/apmonitor.com\/pds\/index.php\/Main\/ActuatorMonitor) ⏱️📊\n\n[Batteries](https:\/\/apmonitor.com\/pds\/index.php\/Main\/LithiumIonBatteries) 📊\n\n[Polymers](https:\/\/apmonitor.com\/pds\/index.php\/Main\/PolymerMeltFlowRate) 📈\n\n[Road Detection](https:\/\/apmonitor.com\/pds\/index.php\/Main\/RoadDetection) 👁️📊\n\n[Safety](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SafetySurveillance) 👁️\n\n[Soils](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SoilClassification) 👁️📊\n\n[Sonar](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SonarDetection) 📊\n\n[Steel Plate](https:\/\/apmonitor.com\/pds\/index.php\/Main\/SteelPlateFaults) 📊\n\n[Texture](https:\/\/apmonitor.com\/pds\/index.php\/Main\/TextureClassification) 👁️📊\n\n[Wind Power](https:\/\/apmonitor.com\/pds\/index.php\/Main\/WindPower) ⏱️📈\n\n📈=Regression\n\n📊=Classification\n\n⏱️=Time Series\n\n👁️=Computer Vision\n\n🎧=Audio\n\n[TCLab Project](https:\/\/apmonitor.github.io\/data_science)\n\n- [Overview](https:\/\/apmonitor.com\/pds\/index.php\/Main\/TCLabIntro)\n- [TCLab Help](https:\/\/apmonitor.com\/pds\/notebooks\/TCLab_Help.html)\n- [1\\. Overview](https:\/\/apmonitor.com\/pds\/notebooks\/01_overview.html)\n- [2\\. Import Data](https:\/\/apmonitor.com\/pds\/notebooks\/02_import_export.html)\n- [3\\. Statistics](https:\/\/apmonitor.com\/pds\/notebooks\/03_analyze.html)\n- [4\\. Visualize](https:\/\/apmonitor.com\/pds\/notebooks\/04_visualize.html)\n- [5\\. Prepare Data](https:\/\/apmonitor.com\/pds\/notebooks\/05_prepare_data.html)\n- [6\\. Regression](https:\/\/apmonitor.com\/pds\/notebooks\/06_regression.html)\n- [7\\. Features](https:\/\/apmonitor.com\/pds\/notebooks\/07_features.html)\n- [8\\. Classification](https:\/\/apmonitor.com\/pds\/notebooks\/08_classification.html)\n- [9\\. Interpolation](https:\/\/apmonitor.com\/pds\/notebooks\/09_interpolation.html)\n- [10\\. Solve Equations](https:\/\/apmonitor.com\/pds\/notebooks\/10_solve_equations.html)\n- [11\\. Differential Equations](https:\/\/apmonitor.com\/pds\/notebooks\/11_differential_equations.html)\n- [12\\. Time Series](https:\/\/apmonitor.com\/pds\/notebooks\/12_time_series.html)\n- [Final TCLab Project](https:\/\/apmonitor.com\/pds\/notebooks\/XX_final.html)\n\nRelated Courses\n\n- [🎓 Begin Python](https:\/\/apmonitor.com\/che263\/index.php\/Main\/CourseProjects)\n- [🎓 Begin Matlab](https:\/\/apmonitor.com\/che263\/index.php\/Main\/BeginMatlab)\n- [🎓 Begin Java](https:\/\/apmonitor.com\/che263\/index.php\/Main\/BeginJava)\n- [🎓 Engineering Computing](https:\/\/apmonitor.com\/che263)\n- [🎓 Data Science](https:\/\/apmonitor.com\/pds\/index.php\/Main\/TCLabIntro)\n- [🎓 Data-Driven Engineering](https:\/\/apmonitor.com\/dde)\n- [🎓 Machine Learning](https:\/\/apmonitor.com\/pds)\n- [🎓 Control (MATLAB)](https:\/\/apmonitor.com\/che436)\n- [🎓 Control (Python)](https:\/\/apmonitor.com\/pdc)\n- [🎓 Optimization](https:\/\/apmonitor.com\/me575)\n- [🎓 Dynamic Optimization](https:\/\/apmonitor.com\/do)\n\n[Admin](https:\/\/apmonitor.com\/pds\/index.php\/PmWiki\/PmWiki)\n\nPage last modified on November 22, 2023, at 12:20 am\n\n💬\n\nSend\n\nMinimize","attrs_readable_markdown":"Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data.\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/analyze.png)\n\nData cleansing is the process of removing bad data that may include outliers, missing entries, failed sensors, or other types of missing or corrupted information.\n\nBad data can be detected with [summary statistics](https:\/\/apmonitor.com\/pds\/index.php\/Main\/StatisticsMath) and [data visualization](https:\/\/apmonitor.com\/pds\/index.php\/Main\/VisualizeData). An effective way to remove bad data is with filters that segregate based on conditions that remove outliers, replace bad values such as *NaN* (Not a Number), or remove the data row that contains an *NaN* value.\n***\n#### Remove Bad Data with Numpy\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/python_numpy.png)\n\nNaN values are removed with numpy by identifying rows *ix* that contain NaN. Next, the rows are removed with *z=z\\[~iz\\]* where ~ is a bitwise not operator.\n\nimport numpy as np  \n z \\= np.array(\\[\\[ 1, 2\\],  \n \\[ np.nan, 3\\],  \n \\[ 4, np.nan\\],  \n \\[ 5, 6\\]\\])  \n iz \\= np.any(np.isnan(z), axis\\=1)  \n print(~iz)  \n z \\= z\\[~iz\\]  \n print(z)\n\nThe result of the selection *iz* is *True* if *NaN* is found anywhere in the row of *False* if *NaN* is not found in that row. The ~ is the not operator to reverse *True* and *False*.\n```\n   print(~iz)\n\n   > [ True False False  True]\n```\nOnly the rows with *True* are kept in the final *z* result.\n```\n   print(z)\n\n   > [[1. 2.]\n   >  [5. 6.]]\n```\n***\n#### Remove Bad Data with Pandas\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/python_pandas.png)\n\n[Pandas](https:\/\/pandas.pydata.org\/) manipulates data tables called DataFrames. There are many functions to efficiently manipulate data and remove bad data.\n\nimport numpy as np  \n import pandas as pd  \n z \\= pd.DataFrame({'x':\\[1,np.nan,4,5\\],'y':\\[2,3,np.nan,6\\]})  \n print(z)\n\nThis produces the same values as shown with the Numpy example.\n```\n        x    y\n   0  1.0  2.0\n   1  NaN  3.0\n   2  4.0  NaN\n   3  5.0  6.0\n```\nThere are two common ways to deal with *NaN* values: drop the rows or fill in values. The first is to remove the rows with *dropna*. Use *inplace=True* to avoid the extra assignment of *result=z.dropna()* to modify z directly.\n\nresult \\= z.dropna()\n```\n        x    y\n   0  1.0  2.0\n   3  5.0  6.0\n```\n#### Replace Bad Data with Pandas\nIf data is very limited then it may be better to keep the row and fill-in values with methods *interpolate* for time-series data or *fillna* to replace *NaN* with a default value.\n\nresult \\= z.fillna(z.mean())\n\nA common replacement is the mean value for each column with *z.mean()*. In this case, the mean (average) of column *x* is 3.333 and column *y* is 3.667.\n```\n             x         y\n   0  1.000000  2.000000\n   1  3.333333  3.000000\n   2  4.000000  3.666667\n   3  5.000000  6.000000\n```\n#### Filters with Pandas\n[Data visualization](https:\/\/apmonitor.com\/pds\/index.php\/Main\/VisualizeData) can help identify outliers, especially with box plots. Statistical information such as standard deviation can also help to identify outliers such as eliminating data points that are more than 5 standard deviations away from the mean. A conditional filter can be created to eliminate bad values. The conditional statement *z\\['y'\\]\\<5.5* creates a Logical array of *True* for *z\\['y'\\]\\<5.5* and *False* for *z\\['y'\\]\\>=5.5*.\n\nresult \\= z\\[z\\['y'\\]\\<5\\.5\\]\n\nThis filter eliminates the last row of the DataFrame and the *NaN* in that column.\n```\n        x    y\n   0  1.0  2.0\n   1  NaN  3.0\n```\nFilters can be combined with the *and* bitwise operator *&* or the *or* bitwise operator *\\|*.\n\nresult \\= z\\[(z\\['y'\\]\\<5\\.5) & (z\\['y'\\]\\>=1\\.0)\\]\n\nAnother way to combine filters is to operate on the object with successive methods such as *z\\['x'\\].notnull()* to eliminate *NaN* values in the *x* column, the *.fillna(0)* to replace *NaN* with zero in the *y* column, and *.reset\\_index(drop=True)* to reset the DataFrame index.\n\nresult \\= z\\[z\\['x'\\].notnull()\\].fillna(0).reset\\_index(drop\\=True)\n```\n        x    y\n   0  1.0  2.0\n   1  4.0  0.0\n   2  5.0  6.0\n```\n#### Further Reading\n- Brownlee, J., [How to Remove Outliers for Machine Learning](https:\/\/machinelearningmastery.com\/how-to-use-statistics-to-identify-outliers-in-data\/), April 2018.\n\n![](https:\/\/apmonitor.com\/pds\/uploads\/Main\/data_cleansing.png)\n***\n#### ✅ Knowledge Check\n**1\\.** What is the primary purpose of data cleansing?\n\n**A.** To add bad data into machine learning models.\n\nIncorrect. Data cleansing aims to remove bad data, not add it.\n\n**B.** To identify and remove bad data.\n\nCorrect. Data cleansing is about removing bad data such as outliers, missing entries, and corrupted information.\n\n**C.** To improve the efficiency of data storage.\n\nIncorrect. While data cleansing can reduce storage needs by removing unnecessary data, its primary purpose is to improve data quality for analysis.\n\n**D.** To make data visualization more attractive.\n\nIncorrect. While clean data can lead to clearer visualizations, the main goal of data cleansing is to improve data quality for analysis.\n\n**2\\.** How can NaN values be removed from a Numpy array?\n\n**A.** By using the dropna() method.\n\nIncorrect. The dropna() method is associated with Pandas, not Numpy.\n\n**B.** By replacing NaN values with the mean of the array.\n\nIncorrect. While this method is used to replace NaN values, it doesn't remove them.\n\n**C.** By using the command: z=z\\[~iz\\] where iz is the selection of NaN rows.\n\nCorrect. This method uses bitwise not operator (~) and a selection to remove rows containing NaN values from a Numpy array.\n\n**D.** By using the fillna() method with the parameter 0.\n\nIncorrect. The fillna() method is associated with Pandas, and it replaces NaN values rather than removing them.","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

4 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.1 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://apmonitor.com/pds/index.php/Main/CleanseData
Last Crawled	2026-04-05 00:30:48 (4 days ago)
First Indexed	2021-02-12 00:58:33 (5 years ago)
HTTP Status Code	200
Meta Title	Data Cleansing
Meta Description	Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data.
Meta Canonical	null
Boilerpipe Text	Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data. Data cleansing is the process of removing bad data that may include outliers, missing entries, failed sensors, or other types of missing or corrupted information. Bad data can be detected with summary statistics and data visualization . An effective way to remove bad data is with filters that segregate based on conditions that remove outliers, replace bad values such as NaN (Not a Number), or remove the data row that contains an NaN value. Remove Bad Data with Numpy NaN values are removed with numpy by identifying rows ix that contain NaN. Next, the rows are removed with z=z[~iz] where ~ is a bitwise not operator. import numpy as np z = np. array ( [ [ 1 , 2 ] , [ np. nan , 3 ] , [ 4 , np. nan ] , [ 5 , 6 ] ] ) iz = np. any ( np. isnan ( z ) , axis = 1 ) print ( ~ iz ) z = z [ ~ iz ] print ( z ) The result of the selection iz is True if NaN is found anywhere in the row of False if NaN is not found in that row. The ~ is the not operator to reverse True and False . print(~iz) > [ True False False True] Only the rows with True are kept in the final z result. print(z) > [[1. 2.] > [5. 6.]] Remove Bad Data with Pandas Pandas manipulates data tables called DataFrames. There are many functions to efficiently manipulate data and remove bad data. import numpy as np import pandas as pd z = pd. DataFrame ( { 'x' : [ 1 , np. nan , 4 , 5 ] , 'y' : [ 2 , 3 , np. nan , 6 ] } ) print ( z ) This produces the same values as shown with the Numpy example. x y 0 1.0 2.0 1 NaN 3.0 2 4.0 NaN 3 5.0 6.0 There are two common ways to deal with NaN values: drop the rows or fill in values. The first is to remove the rows with dropna . Use inplace=True to avoid the extra assignment of result=z.dropna() to modify z directly. result = z. dropna ( ) x y 0 1.0 2.0 3 5.0 6.0 Replace Bad Data with Pandas If data is very limited then it may be better to keep the row and fill-in values with methods interpolate for time-series data or fillna to replace NaN with a default value. result = z. fillna ( z. mean ( ) ) A common replacement is the mean value for each column with z.mean() . In this case, the mean (average) of column x is 3.333 and column y is 3.667. x y 0 1.000000 2.000000 1 3.333333 3.000000 2 4.000000 3.666667 3 5.000000 6.000000 Filters with Pandas Data visualization can help identify outliers, especially with box plots. Statistical information such as standard deviation can also help to identify outliers such as eliminating data points that are more than 5 standard deviations away from the mean. A conditional filter can be created to eliminate bad values. The conditional statement z['y']<5.5 creates a Logical array of True for z['y']<5.5 and False for z['y']>=5.5 . result = z [ z [ 'y' ] < 5.5 ] This filter eliminates the last row of the DataFrame and the NaN in that column. x y 0 1.0 2.0 1 NaN 3.0 Filters can be combined with the and bitwise operator & or the or bitwise operator \| . result = z [ ( z [ 'y' ] < 5.5 ) & ( z [ 'y' ] >= 1.0 ) ] Another way to combine filters is to operate on the object with successive methods such as z['x'].notnull() to eliminate NaN values in the x column, the .fillna(0) to replace NaN with zero in the y column, and .reset_index(drop=True) to reset the DataFrame index. result = z [ z [ 'x' ] . notnull ( ) ] . fillna ( 0 ) . reset_index ( drop = True ) x y 0 1.0 2.0 1 4.0 0.0 2 5.0 6.0 Further Reading Brownlee, J., How to Remove Outliers for Machine Learning , April 2018. ✅ Knowledge Check 1. What is the primary purpose of data cleansing? A. To add bad data into machine learning models. Incorrect. Data cleansing aims to remove bad data, not add it. B. To identify and remove bad data. Correct. Data cleansing is about removing bad data such as outliers, missing entries, and corrupted information. C. To improve the efficiency of data storage. Incorrect. While data cleansing can reduce storage needs by removing unnecessary data, its primary purpose is to improve data quality for analysis. D. To make data visualization more attractive. Incorrect. While clean data can lead to clearer visualizations, the main goal of data cleansing is to improve data quality for analysis. 2. How can NaN values be removed from a Numpy array? A. By using the dropna() method. Incorrect. The dropna() method is associated with Pandas, not Numpy. B. By replacing NaN values with the mean of the array. Incorrect. While this method is used to replace NaN values, it doesn't remove them. C. By using the command: z=z[~iz] where iz is the selection of NaN rows. Correct. This method uses bitwise not operator (~) and a selection to remove rows containing NaN values from a Numpy array. D. By using the fillna() method with the parameter 0. Incorrect. The fillna() method is associated with Pandas, and it replaces NaN values rather than removing them.
Markdown	# [Machine Learning for Engineers](https://apmonitor.com/pds/index.php) - [Home](https://apmonitor.com/pds/index.php/Main/HomePage) - [Schedule](https://apmonitor.com/pds/index.php/Main/CourseSchedule) - [View](https://apmonitor.com/pds/index.php/Main/CleanseData) - [Edit](https://apmonitor.com/pds/index.php/Main/CleanseData?action=edit) - [History](https://apmonitor.com/pds/index.php/Main/CleanseData?action=diff) - [Print](https://apmonitor.com/pds/index.php/Main/CleanseData?action=print) # [Data Cleansing](https://apmonitor.com/pds/index.php/Main/CleanseData) Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data. ![](https://apmonitor.com/pds/uploads/Main/analyze.png) Data cleansing is the process of removing bad data that may include outliers, missing entries, failed sensors, or other types of missing or corrupted information. ![](https://apmonitor.com/pds/uploads/Main/python.png) [Data Cleansing Python Jupyter Notebook](https://github.com/APMonitor/pds/blob/main/Cleanse_Data.ipynb) ![](https://apmonitor.com/pds/uploads/Main/colab.png) [Jupyter Notebook in Google Colab](https://colab.research.google.com/github/APMonitor/pds/blob/main/Cleanse_Data.ipynb) ![](https://apmonitor.com/pds/uploads/Main/matlab.png) [Data Cleansing MATLAB Live Script](https://github.com/APMonitor/mds/blob/main/Cleanse.mlx) Bad data can be detected with [summary statistics](https://apmonitor.com/pds/index.php/Main/StatisticsMath) and [data visualization](https://apmonitor.com/pds/index.php/Main/VisualizeData). An effective way to remove bad data is with filters that segregate based on conditions that remove outliers, replace bad values such as NaN (Not a Number), or remove the data row that contains an NaN value. *** #### Remove Bad Data with Numpy ![](https://apmonitor.com/pds/uploads/Main/python_numpy.png) NaN values are removed with numpy by identifying rows ix that contain NaN. Next, the rows are removed with z=z\[~iz\] where ~ is a bitwise not operator. import numpy as np z \= np.array(\[\[ 1, 2\], \[ np.nan, 3\], \[ 4, np.nan\], \[ 5, 6\]\]) iz \= np.any(np.isnan(z), axis\=1) print(~iz) z \= z\[~iz\] print(z) [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=1) The result of the selection iz is True if NaN is found anywhere in the row of False if NaN is not found in that row. The ~ is the not operator to reverse True and False. ``` print(~iz) > [ True False False True] ``` Only the rows with True are kept in the final z result. ``` print(z) > [[1. 2.] > [5. 6.]] ``` *** #### Remove Bad Data with Pandas ![](https://apmonitor.com/pds/uploads/Main/python_pandas.png) [Pandas](https://pandas.pydata.org/) manipulates data tables called DataFrames. There are many functions to efficiently manipulate data and remove bad data. import numpy as np import pandas as pd z \= pd.DataFrame({'x':\[1,np.nan,4,5\],'y':\[2,3,np.nan,6\]}) print(z) [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=2) This produces the same values as shown with the Numpy example. ``` x y 0 1.0 2.0 1 NaN 3.0 2 4.0 NaN 3 5.0 6.0 ``` There are two common ways to deal with NaN values: drop the rows or fill in values. The first is to remove the rows with dropna. Use inplace=True to avoid the extra assignment of result=z.dropna() to modify z directly. result \= z.dropna() [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=3) ``` x y 0 1.0 2.0 3 5.0 6.0 ``` #### Replace Bad Data with Pandas If data is very limited then it may be better to keep the row and fill-in values with methods interpolate for time-series data or fillna to replace NaN with a default value. result \= z.fillna(z.mean()) [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=4) A common replacement is the mean value for each column with z.mean(). In this case, the mean (average) of column x is 3.333 and column y is 3.667. ``` x y 0 1.000000 2.000000 1 3.333333 3.000000 2 4.000000 3.666667 3 5.000000 6.000000 ``` #### Filters with Pandas [Data visualization](https://apmonitor.com/pds/index.php/Main/VisualizeData) can help identify outliers, especially with box plots. Statistical information such as standard deviation can also help to identify outliers such as eliminating data points that are more than 5 standard deviations away from the mean. A conditional filter can be created to eliminate bad values. The conditional statement z\['y'\]\<5.5 creates a Logical array of True for z\['y'\]\<5.5 and False for z\['y'\]\>=5.5. result \= z\[z\['y'\]\<5\.5\] [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=5) This filter eliminates the last row of the DataFrame and the NaN in that column. ``` x y 0 1.0 2.0 1 NaN 3.0 ``` Filters can be combined with the and bitwise operator & or the or bitwise operator \\|. result \= z\[(z\['y'\]\<5\.5) & (z\['y'\]\>=1\.0)\] [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=6) Another way to combine filters is to operate on the object with successive methods such as z\['x'\].notnull() to eliminate NaN values in the x column, the .fillna(0) to replace NaN with zero in the y column, and .reset\_index(drop=True) to reset the DataFrame index. result \= z\[z\['x'\].notnull()\].fillna(0).reset\_index(drop\=True) [\[\$\[Get Code\]\]](https://apmonitor.com/pds/index.php/Main/CleanseData?action=sourceblock&num=7) ``` x y 0 1.0 2.0 1 4.0 0.0 2 5.0 6.0 ``` #### Further Reading - Brownlee, J., [How to Remove Outliers for Machine Learning](https://machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/), April 2018. ![](https://apmonitor.com/pds/uploads/Main/data_cleansing.png) * #### ✅ Knowledge Check 1\. What is the primary purpose of data cleansing? A. To add bad data into machine learning models. Incorrect. Data cleansing aims to remove bad data, not add it. B. To identify and remove bad data. Correct. Data cleansing is about removing bad data such as outliers, missing entries, and corrupted information. C. To improve the efficiency of data storage. Incorrect. While data cleansing can reduce storage needs by removing unnecessary data, its primary purpose is to improve data quality for analysis. D. To make data visualization more attractive. Incorrect. While clean data can lead to clearer visualizations, the main goal of data cleansing is to improve data quality for analysis. 2\. How can NaN values be removed from a Numpy array? A. By using the dropna() method. Incorrect. The dropna() method is associated with Pandas, not Numpy. B. By replacing NaN values with the mean of the array. Incorrect. While this method is used to replace NaN values, it doesn't remove them. C. By using the command: z=z\[~iz\] where iz is the selection of NaN rows. Correct. This method uses bitwise not operator (~) and a selection to remove rows containing NaN values from a Numpy array. D. By using the fillna() method with the parameter 0. Incorrect. The fillna() method is associated with Pandas, and it replaces NaN values rather than removing them. Course on ![](https://apmonitor.com/pds/uploads/Main/github.png) [GitHub](https://github.com/APMonitor/pds) - [Overview](https://apmonitor.com/pds/index.php/Main/HomePage) - [Syllabus](https://apmonitor.com/pds/index.php/Main/CourseSyllabus) - [Schedule](https://apmonitor.com/pds/index.php/Main/CourseSchedule) - [4-Day Course](https://apmonitor.com/pds/index.php/Main/ShortCourse) - [Course Project](https://apmonitor.com/pds/index.php/Main/CourseProject) - [Startup Incubator](https://apmonitor.com/pds/index.php/Main/StartupIncubator) - [Resources](https://apmonitor.com/pds/index.php/Main/LearningResources) - [Install Python](https://apmonitor.com/pds/index.php/Main/InstallPython) - [Install Packages](https://apmonitor.com/pds/index.php/Main/InstallPythonPackages) - [AI Ethics](https://apmonitor.com/pds/index.php/Main/AIEthics) Exams - [Midterm Exam](https://apmonitor.com/pds/index.php/Main/CourseExam1) Data Engineering - [Overview](https://apmonitor.com/pds/index.php/Main/DataPreparation) - [1️⃣ Gather Data](https://apmonitor.com/pds/index.php/Main/GatherData) - [2️⃣ Statistics](https://apmonitor.com/pds/index.php/Main/StatisticsMath) - [3️⃣ Visualize](https://apmonitor.com/pds/index.php/Main/VisualizeData) - [4️⃣ Cleanse](https://apmonitor.com/pds/index.php/Main/CleanseData) - [5️⃣ Features](https://apmonitor.com/pds/index.php/Main/FeatureEngineering) - [6️⃣ Balance](https://apmonitor.com/pds/index.php/Main/ImbalancedData) - [7️⃣ Scale](https://apmonitor.com/pds/index.php/Main/ScaleData) - [8️⃣ Split](https://apmonitor.com/pds/index.php/Main/SplitData) - [9️⃣ Deploy](https://apmonitor.com/pds/index.php/Main/DeployMachineLearning) - [🔟 Apps](https://apmonitor.com/pds/index.php/Main/AppDevelopment) Agentic Engineering - [1️⃣ Workplan](https://apmonitor.com/pds/index.php/Main/AgenticWorkplan) - [2️⃣ Coding](https://apmonitor.com/pds/index.php/Main/AgenticCoding) - [3️⃣ Coordination](https://apmonitor.com/pds/index.php/Main/AgenticCoordination) - [4️⃣ Visualization](https://apmonitor.com/pds/index.php/Main/AgenticVisualization) - [5️⃣ Communication](https://apmonitor.com/pds/index.php/Main/AgenticReports) Classification - [Overview](https://apmonitor.com/pds/index.php/Main/ClassificationOverview) - [Hyperparameters](https://apmonitor.com/pds/index.php/Main/HyperparameterOptimization) - [Cybersecurity](https://apmonitor.com/pds/index.php/Main/CyberSecurity) Supervised Learning - [AdaBoost](https://apmonitor.com/pds/index.php/Main/AdaBoost) - [Decision Tree](https://apmonitor.com/pds/index.php/Main/DecisionTree) - [k-Nearest Neighbors](https://apmonitor.com/pds/index.php/Main/KNearestNeighbors) - [Logistic Regression](https://apmonitor.com/pds/index.php/Main/LogisticRegression) - [Naïve Bayes](https://apmonitor.com/pds/index.php/Main/NaiveBayes) - [Neural Network Classifier](https://apmonitor.com/pds/index.php/Main/DeepLearningNeuralNetwork) - [Random Forest](https://apmonitor.com/pds/index.php/Main/RandomForest) - [Stochastic Gradient Descent](https://apmonitor.com/pds/index.php/Main/StochasticGradientDescent) - [Support Vector Classifier](https://apmonitor.com/pds/index.php/Main/SupportVectorClassifier) - [XGBoost Classifier](https://apmonitor.com/pds/index.php/Main/XGBoostClassifier) Unsupervised Learning** - [Gaussian Mixture Model](https://apmonitor.com/pds/index.php/Main/GaussianMixtureModel) - [K-Means Clustering](https://apmonitor.com/pds/index.php/Main/KMeansClustering) - [Spectral Clustering](https://apmonitor.com/pds/index.php/Main/SpectralClustering) Regression - [Overview](https://apmonitor.com/pds/index.php/Main/RegressionOverview) - [Linear Regression](https://apmonitor.com/pds/index.php/Main/LinearRegression) - [k-Nearest Neighbors](https://apmonitor.com/pds/index.php/Main/KNearestNeighborsRegression) - [Support Vector Regressor](https://apmonitor.com/pds/index.php/Main/SupportVectorRegressor) - [Gaussian Processes](https://apmonitor.com/pds/index.php/Main/GaussianProcessRegression) - [Neural Network Regressor](https://apmonitor.com/pds/index.php/Main/MultilayerPerceptronNeuralNetwork) - [XGBoost Regressor](https://apmonitor.com/pds/index.php/Main/XGBoostRegressor) Time-Series - [ARX Model](https://apmonitor.com/pds/index.php/Main/ARXTimeSeries) - [LSTM Network](https://apmonitor.com/pds/index.php/Main/LongShortTermMemory) Computer Vision - [Introduction](https://apmonitor.com/pds/index.php/Main/ComputerVisionIntro) - [Cascade Classifier](https://apmonitor.com/pds/index.php/Main/CascadeClassifier) - [Deep Learning](https://apmonitor.com/pds/index.php/Main/VisionDeepLearning) Applications [3D Print](https://apmonitor.com/pds/index.php/Main/AdditiveManufacturing) 📈📊 [Automation LSTM](https://apmonitor.com/pds/index.php/Main/LSTMAutomation) ⏱️ [Automotive](https://apmonitor.com/pds/index.php/Main/AutomotiveMonitoring) 📈📊 [Battery Life](https://apmonitor.com/pds/index.php/Main/BatteryLife)⏱️📈 [Bit Classification](https://apmonitor.com/pds/index.php/Main/BitClassification) 👁️📊 [Concrete Strength](https://apmonitor.com/pds/index.php/Main/CementStrength) 📈📊 [Chemical Properties](https://apmonitor.com/pds/index.php/Main/ThermophysicalProperties) 📈 [Draw Classification](https://apmonitor.com/pds/index.php/Main/DrawClassification) 📊 [Facial Recognition](https://apmonitor.com/pds/index.php/Main/FacialRecognition) 👁️📊 [Glass Type](https://apmonitor.com/pds/index.php/Main/GlassCharacterization)⏱️📈 [Hand Tracking](https://apmonitor.com/pds/index.php/Main/HandTracking) 👁️ [OT Cybersecurity](https://apmonitor.com/pds/index.php/Main/ActuatorMonitor) ⏱️📊 [Batteries](https://apmonitor.com/pds/index.php/Main/LithiumIonBatteries) 📊 [Polymers](https://apmonitor.com/pds/index.php/Main/PolymerMeltFlowRate) 📈 [Road Detection](https://apmonitor.com/pds/index.php/Main/RoadDetection) 👁️📊 [Safety](https://apmonitor.com/pds/index.php/Main/SafetySurveillance) 👁️ [Soils](https://apmonitor.com/pds/index.php/Main/SoilClassification) 👁️📊 [Sonar](https://apmonitor.com/pds/index.php/Main/SonarDetection) 📊 [Steel Plate](https://apmonitor.com/pds/index.php/Main/SteelPlateFaults) 📊 [Texture](https://apmonitor.com/pds/index.php/Main/TextureClassification) 👁️📊 [Wind Power](https://apmonitor.com/pds/index.php/Main/WindPower) ⏱️📈 📈=Regression 📊=Classification ⏱️=Time Series 👁️=Computer Vision 🎧=Audio [TCLab Project](https://apmonitor.github.io/data_science) - [Overview](https://apmonitor.com/pds/index.php/Main/TCLabIntro) - [TCLab Help](https://apmonitor.com/pds/notebooks/TCLab_Help.html) - [1\. Overview](https://apmonitor.com/pds/notebooks/01_overview.html) - [2\. Import Data](https://apmonitor.com/pds/notebooks/02_import_export.html) - [3\. Statistics](https://apmonitor.com/pds/notebooks/03_analyze.html) - [4\. Visualize](https://apmonitor.com/pds/notebooks/04_visualize.html) - [5\. Prepare Data](https://apmonitor.com/pds/notebooks/05_prepare_data.html) - [6\. Regression](https://apmonitor.com/pds/notebooks/06_regression.html) - [7\. Features](https://apmonitor.com/pds/notebooks/07_features.html) - [8\. Classification](https://apmonitor.com/pds/notebooks/08_classification.html) - [9\. Interpolation](https://apmonitor.com/pds/notebooks/09_interpolation.html) - [10\. Solve Equations](https://apmonitor.com/pds/notebooks/10_solve_equations.html) - [11\. Differential Equations](https://apmonitor.com/pds/notebooks/11_differential_equations.html) - [12\. Time Series](https://apmonitor.com/pds/notebooks/12_time_series.html) - [Final TCLab Project](https://apmonitor.com/pds/notebooks/XX_final.html) Related Courses - [🎓 Begin Python](https://apmonitor.com/che263/index.php/Main/CourseProjects) - [🎓 Begin Matlab](https://apmonitor.com/che263/index.php/Main/BeginMatlab) - [🎓 Begin Java](https://apmonitor.com/che263/index.php/Main/BeginJava) - [🎓 Engineering Computing](https://apmonitor.com/che263) - [🎓 Data Science](https://apmonitor.com/pds/index.php/Main/TCLabIntro) - [🎓 Data-Driven Engineering](https://apmonitor.com/dde) - [🎓 Machine Learning](https://apmonitor.com/pds) - [🎓 Control (MATLAB)](https://apmonitor.com/che436) - [🎓 Control (Python)](https://apmonitor.com/pdc) - [🎓 Optimization](https://apmonitor.com/me575) - [🎓 Dynamic Optimization](https://apmonitor.com/do) [Admin](https://apmonitor.com/pds/index.php/PmWiki/PmWiki) Page last modified on November 22, 2023, at 12:20 am 💬 Send Minimize
Readable Markdown	Measurements from sensors or from human input can contain bad data that negatively affects machine learning. This tutorial demonstrates how to identify and remove bad data. ![](https://apmonitor.com/pds/uploads/Main/analyze.png) Data cleansing is the process of removing bad data that may include outliers, missing entries, failed sensors, or other types of missing or corrupted information. Bad data can be detected with [summary statistics](https://apmonitor.com/pds/index.php/Main/StatisticsMath) and [data visualization](https://apmonitor.com/pds/index.php/Main/VisualizeData). An effective way to remove bad data is with filters that segregate based on conditions that remove outliers, replace bad values such as NaN (Not a Number), or remove the data row that contains an NaN value. *** #### Remove Bad Data with Numpy ![](https://apmonitor.com/pds/uploads/Main/python_numpy.png) NaN values are removed with numpy by identifying rows ix that contain NaN. Next, the rows are removed with z=z\[~iz\] where ~ is a bitwise not operator. import numpy as np z \= np.array(\[\[ 1, 2\], \[ np.nan, 3\], \[ 4, np.nan\], \[ 5, 6\]\]) iz \= np.any(np.isnan(z), axis\=1) print(~iz) z \= z\[~iz\] print(z) The result of the selection iz is True if NaN is found anywhere in the row of False if NaN is not found in that row. The ~ is the not operator to reverse True and False. ``` print(~iz) > [ True False False True] ``` Only the rows with True are kept in the final z result. ``` print(z) > [[1. 2.] > [5. 6.]] ``` *** #### Remove Bad Data with Pandas ![](https://apmonitor.com/pds/uploads/Main/python_pandas.png) [Pandas](https://pandas.pydata.org/) manipulates data tables called DataFrames. There are many functions to efficiently manipulate data and remove bad data. import numpy as np import pandas as pd z \= pd.DataFrame({'x':\[1,np.nan,4,5\],'y':\[2,3,np.nan,6\]}) print(z) This produces the same values as shown with the Numpy example. ``` x y 0 1.0 2.0 1 NaN 3.0 2 4.0 NaN 3 5.0 6.0 ``` There are two common ways to deal with NaN values: drop the rows or fill in values. The first is to remove the rows with dropna. Use inplace=True to avoid the extra assignment of result=z.dropna() to modify z directly. result \= z.dropna() ``` x y 0 1.0 2.0 3 5.0 6.0 ``` #### Replace Bad Data with Pandas If data is very limited then it may be better to keep the row and fill-in values with methods interpolate for time-series data or fillna to replace NaN with a default value. result \= z.fillna(z.mean()) A common replacement is the mean value for each column with z.mean(). In this case, the mean (average) of column x is 3.333 and column y is 3.667. ``` x y 0 1.000000 2.000000 1 3.333333 3.000000 2 4.000000 3.666667 3 5.000000 6.000000 ``` #### Filters with Pandas [Data visualization](https://apmonitor.com/pds/index.php/Main/VisualizeData) can help identify outliers, especially with box plots. Statistical information such as standard deviation can also help to identify outliers such as eliminating data points that are more than 5 standard deviations away from the mean. A conditional filter can be created to eliminate bad values. The conditional statement z\['y'\]\<5.5 creates a Logical array of True for z\['y'\]\<5.5 and False for z\['y'\]\>=5.5. result \= z\[z\['y'\]\<5\.5\] This filter eliminates the last row of the DataFrame and the NaN in that column. ``` x y 0 1.0 2.0 1 NaN 3.0 ``` Filters can be combined with the and bitwise operator & or the or bitwise operator \\|. result \= z\[(z\['y'\]\<5\.5) & (z\['y'\]\>=1\.0)\] Another way to combine filters is to operate on the object with successive methods such as z\['x'\].notnull() to eliminate NaN values in the x column, the .fillna(0) to replace NaN with zero in the y column, and .reset\_index(drop=True) to reset the DataFrame index. result \= z\[z\['x'\].notnull()\].fillna(0).reset\_index(drop\=True) ``` x y 0 1.0 2.0 1 4.0 0.0 2 5.0 6.0 ``` #### Further Reading - Brownlee, J., [How to Remove Outliers for Machine Learning](https://machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/), April 2018. ![](https://apmonitor.com/pds/uploads/Main/data_cleansing.png) * #### ✅ Knowledge Check 1\. What is the primary purpose of data cleansing? A. To add bad data into machine learning models. Incorrect. Data cleansing aims to remove bad data, not add it. B. To identify and remove bad data. Correct. Data cleansing is about removing bad data such as outliers, missing entries, and corrupted information. C. To improve the efficiency of data storage. Incorrect. While data cleansing can reduce storage needs by removing unnecessary data, its primary purpose is to improve data quality for analysis. D. To make data visualization more attractive. Incorrect. While clean data can lead to clearer visualizations, the main goal of data cleansing is to improve data quality for analysis. 2\. How can NaN values be removed from a Numpy array? A. By using the dropna() method. Incorrect. The dropna() method is associated with Pandas, not Numpy. B. By replacing NaN values with the mean of the array. Incorrect. While this method is used to replace NaN values, it doesn't remove them. C. By using the command: z=z\[~iz\] where iz is the selection of NaN rows. Correct. This method uses bitwise not operator (~) and a selection to remove rows containing NaN values from a Numpy array. D.** By using the fillna() method with the parameter 0. Incorrect. The fillna() method is associated with Pandas, and it replaces NaN values rather than removing them.
Shard	37 (laksa)
Root Hash	13934051142679362237
Unparsed URL	com,apmonitor!/pds/index.php/Main/CleanseData s443