🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 148 (from laksa067)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa148.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://contrib.scikit-learn.org/category_encoders/jamesstein.html\')), getAhrefsUnparsedNoserviceFromURL(\'https://contrib.scikit-learn.org/category_encoders/jamesstein.html\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html","crawl_time":1773493351,"first_indexed_time":1616109997,"http_code":200,"src_unparsed":"org,scikit-learn!contrib,\/category_encoders\/jamesstein.html s443","src_root_hash":"6052685795207125548","history_drop_reason":null,"meta_title":"James-Stein Encoder — Category Encoders 2.8.1 documentation","meta_descriptions":[],"attrs_boilerpipe_text":"James-Stein Encoder\n\nclass\ncategory_encoders.james_stein.\nJamesSteinEncoder\n(\nverbose\n=\n0\n,\ncols\n=\nNone\n,\ndrop_invariant\n=\nFalse\n,\nreturn_df\n=\nTrue\n,\nhandle_unknown\n=\n'value'\n,\nhandle_missing\n=\n'value'\n,\nmodel\n=\n'independent'\n,\nrandom_state\n=\nNone\n,\nrandomized\n=\nFalse\n,\nsigma\n=\n0.05\n)\n[source]\n\nJames-Stein estimator.\nSupported targets: binomial and continuous.\nFor polynomial target support, see PolynomialWrapper.\nFor feature value\ni\n, James-Stein estimator returns a weighted average of:\nThe mean target value for the observed feature value\ni\n.\nThe mean target value (regardless of the feature value).\nThis can be written as:\nJS_i\n=\n(\n1\n-\nB\n)\n*\nmean\n(\ny_i\n)\n+\nB\n*\nmean\n(\ny\n)\nThe question is, what should be the weight\nB\n?\nIf we put too much weight on the conditional mean value, we will overfit.\nIf we put too much weight on the global mean, we will underfit.\nThe canonical solution in machine learning is to perform cross-validation.\nHowever, Charles Stein came with a closed-form solution to the problem.\nThe intuition is: If the estimate of\nmean(y_i)\nis unreliable (\ny_i\nhas high variance),\nwe should put more weight on\nmean(y)\n. Stein put it into an equation as:\nB\n=\nvar\n(\ny_i\n)\n\/\n(\nvar\n(\ny_i\n)\n+\nvar\n(\ny\n))\nThe only remaining issue is that we do not know\nvar(y)\n, let alone\nvar(y_i)\n.\nHence, we have to estimate the variances. But how can we reliably estimate the\nvariances, when we already struggle with the estimation of the mean values?!\nThere are multiple solutions:\n1. If we have the same count of observations for each feature value\ni\nand all\ny_i\nare close to each other, we can pretend that all\nvar(y_i)\nare identical.\nThis is called a pooled model.\n2. If the observation counts are not equal, it makes sense to replace the variances\nwith squared standard errors, which penalize small observation counts:\nSE\n^\n2\n=\nvar\n(\ny\n)\n\/\ncount\n(\ny\n)\nThis is called an independent model.\nJames-Stein estimator has, however, one practical limitation - it was defined\nonly for normal distributions. If you want to apply it for binary classification,\nwhich allows only values {0, 1}, it is better to first convert the mean target value\nfrom the bound interval <0,1> into an unbounded interval by replacing mean(y)\nwith log-odds ratio:\nlog\n-\nodds_ratio_i\n=\nlog\n(\nmean\n(\ny_i\n)\n\/\nmean\n(\ny_not_i\n))\nThis is called binary model. The estimation of parameters of this model is, however,\ntricky and sometimes it fails fatally. In these situations, it is better to use beta\nmodel, which generally delivers slightly worse accuracy than binary model but does\nnot suffer from fatal failures.\nParameters\n:\nverbose: int\ninteger indicating verbosity of the output. 0 for none.\ncols: list\na list of columns to encode, if None, all string columns will be encoded.\ndrop_invariant: bool\nboolean for whether or not to drop encoded columns with 0 variance.\nreturn_df: bool\nboolean for whether to return a pandas DataFrame from transform\n(otherwise it will be a numpy array).\nhandle_missing: str\noptions are ‘return_nan’, ‘error’ and ‘value’, defaults to ‘value’,\nwhich returns the prior probability.\nhandle_unknown: str\noptions are ‘return_nan’, ‘error’ and ‘value’, defaults to ‘value’,\nwhich returns the prior probability.\nmodel: str\noptions are ‘pooled’, ‘beta’, ‘binary’ and ‘independent’, defaults to ‘independent’.\nrandomized: bool,\nadds normal (Gaussian) distribution noise into training data in order to decrease\noverfitting (testing data are untouched).\nsigma: float\nstandard deviation (spread or “width”) of the normal distribution.\nMethods\nfit\n(X[, y])\nFits the encoder according to X and y.\nfit_transform\n(X[, y])\nFit and transform using the target information.\nget_feature_names\n()\nDeprecated method to get feature names.\nget_feature_names_in\n()\nGet the names of all input columns present when fitting.\nget_feature_names_out\n([input_features])\nGet the names of all transformed \/ added columns.\nget_metadata_routing\n()\nGet metadata routing of this object.\nget_params\n([deep])\nGet parameters for this estimator.\nset_output\n(*[, transform])\nSet output container.\nset_params\n(**params)\nSet the parameters of this estimator.\nset_transform_request\n(*[, override_return_df])\nConfigure whether metadata should be requested to be passed to the\ntransform\nmethod.\ntransform\n(X[, y, override_return_df])\nPerform the transformation to new categorical data.\nReferences\n[\n1\n]\nParametric empirical Bayes inference: Theory and applications, equations 1.19 & 1.20,\nfrom\nhttps:\/\/www.jstor.org\/stable\/2287098\n[\n2\n]\nEmpirical Bayes for multiple sample sizes, from\nhttp:\/\/chris-said.io\/2017\/05\/03\/empirical-bayes-for-multiple-sample-sizes\/\n[\n3\n]\nShrinkage Estimation of Log-odds Ratios for Comparing Mobility Tables, from\nhttps:\/\/journals.sagepub.com\/doi\/abs\/10.1177\/0081175015570097\n[\n4\n]\nStein’s paradox and group rationality, from\nhttp:\/\/www.philos.rug.nl\/~romeyn\/presentation\/\n2017_romeijn_\n-_Paris_Stein.pdf\n[\n5\n]\nStein’s Paradox in Statistics, from\nhttp:\/\/statweb.stanford.edu\/~ckirby\/brad\/other\/Article1977.pdf\nfit\n(\nX\n:\nndarray\n|\nDataFrame\n|\nlist\n|\ngeneric\n|\ncsr_matrix\n,\ny\n:\nlist\n|\nSeries\n|\nndarray\n|\ntuple\n|\nDataFrame\n|\nNone\n=\nNone\n,\n**\nkwargs\n)\n\nFits the encoder according to X and y.\nParameters\n:\nX\narray-like, shape = [n_samples, n_features]\nTraining vectors, where n_samples is the number of samples\nand n_features is the number of features.\ny\narray-like, shape = [n_samples]\nTarget values.\nReturns\n:\nself\nencoder\nReturns self.\nfit_transform\n(\nX\n:\nndarray\n|\nDataFrame\n|\nlist\n|\ngeneric\n|\ncsr_matrix\n,\ny\n:\nlist\n|\nSeries\n|\nndarray\n|\ntuple\n|\nDataFrame\n|\nNone\n=\nNone\n,\n**\nfit_params\n)\n\nFit and transform using the target information.\nThis also uses the target for transforming, not only for training.\nget_feature_names\n(\n)\n→\nndarray\n\nDeprecated method to get feature names. Use\nget_feature_names_out\ninstead.\nget_feature_names_in\n(\n)\n→\nndarray\n\nGet the names of all input columns present when fitting.\nThese columns are necessary for the transform step.\nget_feature_names_out\n(\ninput_features\n=\nNone\n)\n→\nndarray\n\nGet the names of all transformed \/ added columns.\nNote that in sklearn the get_feature_names_out function takes the feature_names_in\nas an argument and determines the output feature names using the input.\nA fit is usually not necessary and if so a NotFittedError is raised.\nWe just require a fit all the time and return the fitted output columns.\nReturns\n:\nfeature_names: np.ndarray\nA numpy array with all feature names transformed or added.\nNote: potentially dropped features (because the feature is constant\/invariant)\nare not included!\nget_metadata_routing\n(\n)\n\nGet metadata routing of this object.\nPlease check\nUser Guide\non how the routing\nmechanism works.\nReturns\n:\nrouting\nMetadataRequest\nA\nMetadataRequest\nencapsulating\nrouting information.\nget_params\n(\ndeep\n=\nTrue\n)\n\nGet parameters for this estimator.\nParameters\n:\ndeep\nbool, default=True\nIf True, will return the parameters for this estimator and\ncontained subobjects that are estimators.\nReturns\n:\nparams\ndict\nParameter names mapped to their values.\nset_output\n(\n*\n,\ntransform\n=\nNone\n)\n\nSet output container.\nSee\nsphx_glr_auto_examples_miscellaneous_plot_set_output.py\nfor an example on how to use the API.\nParameters\n:\ntransform\n{“default”, “pandas”, “polars”}, default=None\nConfigure output of\ntransform\nand\nfit_transform\n.\n“default”\n: Default output format of a transformer\n“pandas”\n: DataFrame output\n“polars”\n: Polars output\nNone\n: Transform configuration is unchanged\nAdded in version 1.4:\n“polars”\noption was added.\nReturns\n:\nself\nestimator instance\nEstimator instance.\nset_params\n(\n**\nparams\n)\n\nSet the parameters of this estimator.\nThe method works on simple estimators as well as on nested objects\n(such as\nPipeline\n). The latter have\nparameters of the form\n<component>__<parameter>\nso that it’s\npossible to update each component of a nested object.\nParameters\n:\n**params\ndict\nEstimator parameters.\nReturns\n:\nself\nestimator instance\nEstimator instance.\nset_transform_request\n(\n*\n,\noverride_return_df\n:\nbool\n|\nNone\n|\nstr\n=\n'$UNCHANGED$'\n)\n→\nJamesSteinEncoder\n\nConfigure whether metadata should be requested to be passed to the\ntransform\nmethod.\nNote that this method is only relevant when this estimator is used as a\nsub-estimator within a\nmeta-estimator\nand metadata routing is enabled\nwith\nenable_metadata_routing=True\n(see\nsklearn.set_config()\n).\nPlease check the\nUser Guide\non how the routing\nmechanism works.\nThe options for each parameter are:\nTrue\n: metadata is requested, and passed to\ntransform\nif provided. The request is ignored if metadata is not provided.\nFalse\n: metadata is not requested and the meta-estimator will not pass it to\ntransform\n.\nNone\n: metadata is not requested, and the meta-estimator will raise an error if the user provides it.\nstr\n: metadata should be passed to the meta-estimator with this given alias instead of the original name.\nThe default (\nsklearn.utils.metadata_routing.UNCHANGED\n) retains the\nexisting request. This allows you to change the request for some\nparameters and not others.\nAdded in version 1.3.\nParameters\n:\noverride_return_df\nstr, True, False, or None,                     default=sklearn.utils.metadata_routing.UNCHANGED\nMetadata routing for\noverride_return_df\nparameter in\ntransform\n.\nReturns\n:\nself\nobject\nThe updated object.\ntransform\n(\nX\n:\nndarray\n|\nDataFrame\n|\nlist\n|\ngeneric\n|\ncsr_matrix\n,\ny\n:\nlist\n|\nSeries\n|\nndarray\n|\ntuple\n|\nDataFrame\n|\nNone\n=\nNone\n,\noverride_return_df\n:\nbool\n=\nFalse\n)\n\nPerform the transformation to new categorical data.\nSome encoders behave differently on whether y is given or not.\nThis is mainly due to regularisation in order to avoid overfitting.\nOn training data transform should be called with y, on test data without.\nParameters\n:\nX\narray-like, shape = [n_samples, n_features]\ny\narray-like, shape = [n_samples] or None\noverride_return_df\nbool\noverride self.return_df to force to return a data frame\nReturns\n:\np\narray or DataFrame, shape = [n_samples, n_features_out]\nTransformed values with encoding applied.","attrs_markdown":"[Category Encoders](https:\/\/contrib.scikit-learn.org\/category_encoders\/index.html)\n\n- [Backward Difference Coding](https:\/\/contrib.scikit-learn.org\/category_encoders\/backward_difference.html)\n- [BaseN](https:\/\/contrib.scikit-learn.org\/category_encoders\/basen.html)\n- [Binary](https:\/\/contrib.scikit-learn.org\/category_encoders\/binary.html)\n- [CatBoost Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/catboost.html)\n- [Count Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/count.html)\n- [Generalized Linear Mixed Model Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/glmm.html)\n- [Gray](https:\/\/contrib.scikit-learn.org\/category_encoders\/gray.html)\n- [Hashing](https:\/\/contrib.scikit-learn.org\/category_encoders\/hashing.html)\n- [Helmert Coding](https:\/\/contrib.scikit-learn.org\/category_encoders\/helmert.html)\n- [James-Stein Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html)\n  - [`JamesSteinEncoder`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder)\n    - [`JamesSteinEncoder.fit()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit)\n    - [`JamesSteinEncoder.fit_transform()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit_transform)\n    - [`JamesSteinEncoder.get_feature_names()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names)\n    - [`JamesSteinEncoder.get_feature_names_in()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in)\n    - [`JamesSteinEncoder.get_feature_names_out()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out)\n    - [`JamesSteinEncoder.get_metadata_routing()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing)\n    - [`JamesSteinEncoder.get_params()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_params)\n    - [`JamesSteinEncoder.set_output()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_output)\n    - [`JamesSteinEncoder.set_params()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_params)\n    - [`JamesSteinEncoder.set_transform_request()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_transform_request)\n    - [`JamesSteinEncoder.transform()`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.transform)\n- [Leave One Out](https:\/\/contrib.scikit-learn.org\/category_encoders\/leaveoneout.html)\n- [M-estimate](https:\/\/contrib.scikit-learn.org\/category_encoders\/mestimate.html)\n- [One Hot](https:\/\/contrib.scikit-learn.org\/category_encoders\/onehot.html)\n- [Ordinal](https:\/\/contrib.scikit-learn.org\/category_encoders\/ordinal.html)\n- [Polynomial Coding](https:\/\/contrib.scikit-learn.org\/category_encoders\/polynomial.html)\n- [Quantile Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/quantile.html)\n- [RankHotEncoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/rankhot.html)\n- [Sum Coding](https:\/\/contrib.scikit-learn.org\/category_encoders\/sum.html)\n- [Summary Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/summary.html)\n- [Target Encoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/targetencoder.html)\n- [Weight of Evidence](https:\/\/contrib.scikit-learn.org\/category_encoders\/woe.html)\n- [Wrappers](https:\/\/contrib.scikit-learn.org\/category_encoders\/wrapper.html)\n\n[Category Encoders](https:\/\/contrib.scikit-learn.org\/category_encoders\/index.html)\n\n- James-Stein Encoder\n- [View page source](https:\/\/contrib.scikit-learn.org\/category_encoders\/_sources\/jamesstein.rst.txt)\n***\n\n# James-Stein Encoder[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#james-stein-encoder \"Link to this heading\")\n*class* category\\_encoders.james\\_stein.JamesSteinEncoder(*verbose\\=0*, *cols\\=None*, *drop\\_invariant\\=False*, *return\\_df\\=True*, *handle\\_unknown\\='value'*, *handle\\_missing\\='value'*, *model\\='independent'*, *random\\_state\\=None*, *randomized\\=False*, *sigma\\=0\\.05*)[\\[source\\]](https:\/\/contrib.scikit-learn.org\/category_encoders\/_modules\/category_encoders\/james_stein.html#JamesSteinEncoder)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder \"Link to this definition\")\n\nJames-Stein estimator.\n\nSupported targets: binomial and continuous. For polynomial target support, see PolynomialWrapper.\n\nFor feature value i, James-Stein estimator returns a weighted average of:\n\n> 1. The mean target value for the observed feature value i.\n> 2. The mean target value (regardless of the feature value).\n\nThis can be written as:\n```\nJS_i = (1 - B) * mean(y_i) + B * mean(y)\n```\nThe question is, what should be the weight B? If we put too much weight on the conditional mean value, we will overfit. If we put too much weight on the global mean, we will underfit. The canonical solution in machine learning is to perform cross-validation. However, Charles Stein came with a closed-form solution to the problem. The intuition is: If the estimate of mean(y\\_i) is unreliable (y\\_i has high variance), we should put more weight on mean(y). Stein put it into an equation as:\n```\nB = var(y_i) \/ (var(y_i) + var(y))\n```\nThe only remaining issue is that we do not know var(y), let alone var(y\\_i). Hence, we have to estimate the variances. But how can we reliably estimate the variances, when we already struggle with the estimation of the mean values?! There are multiple solutions:\n\n> 1\\. If we have the same count of observations for each feature value i and all y\\_i are close to each other, we can pretend that all var(y\\_i) are identical. This is called a pooled model. 2. If the observation counts are not equal, it makes sense to replace the variances with squared standard errors, which penalize small observation counts:\n> ```\n> SE^2 = var(y)\/count(y)\n> ```\n> This is called an independent model.\n\nJames-Stein estimator has, however, one practical limitation - it was defined only for normal distributions. If you want to apply it for binary classification, which allows only values {0, 1}, it is better to first convert the mean target value from the bound interval \\<0,1\\> into an unbounded interval by replacing mean(y) with log-odds ratio:\n```\nlog-odds_ratio_i = log(mean(y_i)\/mean(y_not_i))\n```\nThis is called binary model. The estimation of parameters of this model is, however, tricky and sometimes it fails fatally. In these situations, it is better to use beta model, which generally delivers slightly worse accuracy than binary model but does not suffer from fatal failures.\n\nParameters:\n\n**verbose: int**\n\ninteger indicating verbosity of the output. 0 for none.\n\n**cols: list**\n\na list of columns to encode, if None, all string columns will be encoded.\n\n**drop\\_invariant: bool**\n\nboolean for whether or not to drop encoded columns with 0 variance.\n\n**return\\_df: bool**\n\nboolean for whether to return a pandas DataFrame from transform (otherwise it will be a numpy array).\n\n**handle\\_missing: str**\n\noptions are ‘return\\_nan’, ‘error’ and ‘value’, defaults to ‘value’, which returns the prior probability.\n\n**handle\\_unknown: str**\n\noptions are ‘return\\_nan’, ‘error’ and ‘value’, defaults to ‘value’, which returns the prior probability.\n\n**model: str**\n\noptions are ‘pooled’, ‘beta’, ‘binary’ and ‘independent’, defaults to ‘independent’.\n\n**randomized: bool,**\n\nadds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched).\n\n**sigma: float**\n\nstandard deviation (spread or “width”) of the normal distribution.\n\nMethods\n\n|   |   |\n|---|---|\n| [`fit`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit \"category_encoders.james_stein.JamesSteinEncoder.fit\")(X\\[, y\\]) | Fits the encoder according to X and y. |\n| [`fit_transform`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit_transform \"category_encoders.james_stein.JamesSteinEncoder.fit_transform\")(X\\[, y\\]) | Fit and transform using the target information. |\n| [`get_feature_names`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names \"category_encoders.james_stein.JamesSteinEncoder.get_feature_names\")() | Deprecated method to get feature names. |\n| [`get_feature_names_in`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in \"category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in\")() | Get the names of all input columns present when fitting. |\n| [`get_feature_names_out`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out \"category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out\")(\\[input\\_features\\]) | Get the names of all transformed \/ added columns. |\n| [`get_metadata_routing`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing \"category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing\")() | Get metadata routing of this object. |\n| [`get_params`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_params \"category_encoders.james_stein.JamesSteinEncoder.get_params\")(\\[deep\\]) | Get parameters for this estimator. |\n| [`set_output`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_output \"category_encoders.james_stein.JamesSteinEncoder.set_output\")(\\*\\[, transform\\]) | Set output container. |\n| [`set_params`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_params \"category_encoders.james_stein.JamesSteinEncoder.set_params\")(\\*\\*params) | Set the parameters of this estimator. |\n| [`set_transform_request`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_transform_request \"category_encoders.james_stein.JamesSteinEncoder.set_transform_request\")(\\*\\[, override\\_return\\_df\\]) | Configure whether metadata should be requested to be passed to the `transform` method. |\n| [`transform`](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.transform \"category_encoders.james_stein.JamesSteinEncoder.transform\")(X\\[, y, override\\_return\\_df\\]) | Perform the transformation to new categorical data. |\n\nReferences\n\n\\[1\\]\n\nParametric empirical Bayes inference: Theory and applications, equations 1.19 & 1.20,\n\nfrom <https:\/\/www.jstor.org\/stable\/2287098>\n\n\\[2\\]\n\nEmpirical Bayes for multiple sample sizes, from\n\n<http:\/\/chris-said.io\/2017\/05\/03\/empirical-bayes-for-multiple-sample-sizes\/>\n\n\\[3\\]\n\nShrinkage Estimation of Log-odds Ratios for Comparing Mobility Tables, from\n\n<https:\/\/journals.sagepub.com\/doi\/abs\/10.1177\/0081175015570097>\n\n\\[4\\]\n\nStein’s paradox and group rationality, from\n\n<http:\/\/www.philos.rug.nl\/~romeyn\/presentation\/>[2017\\_romeijn\\_](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#id6)\\-\\_Paris\\_Stein.pdf\n\n\\[5\\]\n\nStein’s Paradox in Statistics, from\n\n<http:\/\/statweb.stanford.edu\/~ckirby\/brad\/other\/Article1977.pdf>\n\nfit(*X: ndarray \\| DataFrame \\| list \\| generic \\| csr\\_matrix*, *y: list \\| Series \\| ndarray \\| tuple \\| DataFrame \\| None \\= None*, *\\*\\*kwargs*)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit \"Link to this definition\")\n\nFits the encoder according to X and y.\n\nParameters:\n\n**X**array-like, shape = \\[n\\_samples, n\\_features\\]\n\nTraining vectors, where n\\_samples is the number of samples and n\\_features is the number of features.\n\n**y**array-like, shape = \\[n\\_samples\\]\n\nTarget values.\n\nReturns:\n\n**self**encoder\n\nReturns self.\n\nfit\\_transform(*X: ndarray \\| DataFrame \\| list \\| generic \\| csr\\_matrix*, *y: list \\| Series \\| ndarray \\| tuple \\| DataFrame \\| None \\= None*, *\\*\\*fit\\_params*)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit_transform \"Link to this definition\")\n\nFit and transform using the target information.\n\nThis also uses the target for transforming, not only for training.\n\nget\\_feature\\_names() → ndarray[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names \"Link to this definition\")\n\nDeprecated method to get feature names. Use get\\_feature\\_names\\_out instead.\n\nget\\_feature\\_names\\_in() → ndarray[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in \"Link to this definition\")\n\nGet the names of all input columns present when fitting.\n\nThese columns are necessary for the transform step.\n\nget\\_feature\\_names\\_out(*input\\_features\\=None*) → ndarray[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out \"Link to this definition\")\n\nGet the names of all transformed \/ added columns.\n\nNote that in sklearn the get\\_feature\\_names\\_out function takes the feature\\_names\\_in as an argument and determines the output feature names using the input. A fit is usually not necessary and if so a NotFittedError is raised. We just require a fit all the time and return the fitted output columns.\n\nReturns:\n\nfeature\\_names: np.ndarray\n\nA numpy array with all feature names transformed or added. Note: potentially dropped features (because the feature is constant\/invariant) are not included\\!\n\nget\\_metadata\\_routing()[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing \"Link to this definition\")\n\nGet metadata routing of this object.\n\nPlease check User Guide on how the routing mechanism works.\n\nReturns:\n\n**routing**MetadataRequest\n\nA `MetadataRequest` encapsulating routing information.\n\nget\\_params(*deep\\=True*)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_params \"Link to this definition\")\n\nGet parameters for this estimator.\n\nParameters:\n\n**deep**bool, default=True\n\nIf True, will return the parameters for this estimator and contained subobjects that are estimators.\n\nReturns:\n\n**params**dict\n\nParameter names mapped to their values.\n\nset\\_output(*\\**, *transform\\=None*)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_output \"Link to this definition\")\n\nSet output container.\n\nSee sphx\\_glr\\_auto\\_examples\\_miscellaneous\\_plot\\_set\\_output.py for an example on how to use the API.\n\nParameters:\n\n**transform**{“default”, “pandas”, “polars”}, default=None\n\nConfigure output of transform and fit\\_transform.\n\n- “default”: Default output format of a transformer\n- “pandas”: DataFrame output\n- “polars”: Polars output\n- None: Transform configuration is unchanged\n\nAdded in version 1.4: “polars” option was added.\n\nReturns:\n\n**self**estimator instance\n\nEstimator instance.\n\nset\\_params(*\\*\\*params*)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_params \"Link to this definition\")\n\nSet the parameters of this estimator.\n\nThe method works on simple estimators as well as on nested objects (such as `Pipeline`). The latter have parameters of the form `<component>__<parameter>` so that it’s possible to update each component of a nested object.\n\nParameters:\n\n**\\*\\*params**dict\n\nEstimator parameters.\n\nReturns:\n\n**self**estimator instance\n\nEstimator instance.\n\nset\\_transform\\_request(*\\**, *override\\_return\\_df: bool \\| None \\| str \\= '\\$UNCHANGED\\$'*) → [JamesSteinEncoder](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder \"category_encoders.james_stein.JamesSteinEncoder\")[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_transform_request \"Link to this definition\")\n\nConfigure whether metadata should be requested to be passed to the `transform` method.\n\nNote that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with `enable_metadata_routing=True` (see `sklearn.set_config()`). Please check the User Guide on how the routing mechanism works.\n\nThe options for each parameter are:\n\n- `True`: metadata is requested, and passed to `transform` if provided. The request is ignored if metadata is not provided.\n- `False`: metadata is not requested and the meta-estimator will not pass it to `transform`.\n- `None`: metadata is not requested, and the meta-estimator will raise an error if the user provides it.\n- `str`: metadata should be passed to the meta-estimator with this given alias instead of the original name.\n\nThe default (`sklearn.utils.metadata_routing.UNCHANGED`) retains the existing request. This allows you to change the request for some parameters and not others.\n\nAdded in version 1.3.\n\nParameters:\n\n**override\\_return\\_df**str, True, False, or None, default=sklearn.utils.metadata\\_routing.UNCHANGED\n\nMetadata routing for `override_return_df` parameter in `transform`.\n\nReturns:\n\n**self**object\n\nThe updated object.\n\ntransform(*X: ndarray \\| DataFrame \\| list \\| generic \\| csr\\_matrix*, *y: list \\| Series \\| ndarray \\| tuple \\| DataFrame \\| None \\= None*, *override\\_return\\_df: bool \\= False*)[](https:\/\/contrib.scikit-learn.org\/category_encoders\/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.transform \"Link to this definition\")\n\nPerform the transformation to new categorical data.\n\nSome encoders behave differently on whether y is given or not. This is mainly due to regularisation in order to avoid overfitting. On training data transform should be called with y, on test data without.\n\nParameters:\n\n**X**array-like, shape = \\[n\\_samples, n\\_features\\]\n\n**y**array-like, shape = \\[n\\_samples\\] or None\n\n**override\\_return\\_df**bool\n\noverride self.return\\_df to force to return a data frame\n\nReturns:\n\n**p**array or DataFrame, shape = \\[n\\_samples, n\\_features\\_out\\]\n\nTransformed values with encoding applied.\n\n[Previous](https:\/\/contrib.scikit-learn.org\/category_encoders\/helmert.html \"Helmert Coding\") [Next](https:\/\/contrib.scikit-learn.org\/category_encoders\/leaveoneout.html \"Leave One Out\")\n***\n© Copyright 2024, Paul Westenthanner, Will McGinnis.\n\nBuilt with [Sphinx](https:\/\/www.sphinx-doc.org\/) using a [theme](https:\/\/github.com\/readthedocs\/sphinx_rtd_theme) provided by [Read the Docs](https:\/\/readthedocs.org\/).","attrs_readable_markdown":null,"meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

1 month ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	1.2 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://contrib.scikit-learn.org/category_encoders/jamesstein.html
Last Crawled	2026-03-14 13:02:31 (1 month ago)
First Indexed	2021-03-18 23:26:37 (5 years ago)
HTTP Status Code	200
Meta Title	James-Stein Encoder — Category Encoders 2.8.1 documentation
Meta Description	null
Meta Canonical	null
Boilerpipe Text	James-Stein Encoder  class category_encoders.james_stein. JamesSteinEncoder ( verbose = 0 , cols = None , drop_invariant = False , return_df = True , handle_unknown = 'value' , handle_missing = 'value' , model = 'independent' , random_state = None , randomized = False , sigma = 0.05 ) [source]  James-Stein estimator. Supported targets: binomial and continuous. For polynomial target support, see PolynomialWrapper. For feature value i , James-Stein estimator returns a weighted average of: The mean target value for the observed feature value i . The mean target value (regardless of the feature value). This can be written as: JS_i = ( 1 - B ) * mean ( y_i ) + B * mean ( y ) The question is, what should be the weight B ? If we put too much weight on the conditional mean value, we will overfit. If we put too much weight on the global mean, we will underfit. The canonical solution in machine learning is to perform cross-validation. However, Charles Stein came with a closed-form solution to the problem. The intuition is: If the estimate of mean(y_i) is unreliable ( y_i has high variance), we should put more weight on mean(y) . Stein put it into an equation as: B = var ( y_i ) / ( var ( y_i ) + var ( y )) The only remaining issue is that we do not know var(y) , let alone var(y_i) . Hence, we have to estimate the variances. But how can we reliably estimate the variances, when we already struggle with the estimation of the mean values?! There are multiple solutions: 1. If we have the same count of observations for each feature value i and all y_i are close to each other, we can pretend that all var(y_i) are identical. This is called a pooled model. 2. If the observation counts are not equal, it makes sense to replace the variances with squared standard errors, which penalize small observation counts: SE ^ 2 = var ( y ) / count ( y ) This is called an independent model. James-Stein estimator has, however, one practical limitation - it was defined only for normal distributions. If you want to apply it for binary classification, which allows only values {0, 1}, it is better to first convert the mean target value from the bound interval <0,1> into an unbounded interval by replacing mean(y) with log-odds ratio: log - odds_ratio_i = log ( mean ( y_i ) / mean ( y_not_i )) This is called binary model. The estimation of parameters of this model is, however, tricky and sometimes it fails fatally. In these situations, it is better to use beta model, which generally delivers slightly worse accuracy than binary model but does not suffer from fatal failures. Parameters : verbose: int integer indicating verbosity of the output. 0 for none. cols: list a list of columns to encode, if None, all string columns will be encoded. drop_invariant: bool boolean for whether or not to drop encoded columns with 0 variance. return_df: bool boolean for whether to return a pandas DataFrame from transform (otherwise it will be a numpy array). handle_missing: str options are ‘return_nan’, ‘error’ and ‘value’, defaults to ‘value’, which returns the prior probability. handle_unknown: str options are ‘return_nan’, ‘error’ and ‘value’, defaults to ‘value’, which returns the prior probability. model: str options are ‘pooled’, ‘beta’, ‘binary’ and ‘independent’, defaults to ‘independent’. randomized: bool, adds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched). sigma: float standard deviation (spread or “width”) of the normal distribution. Methods fit (X[, y]) Fits the encoder according to X and y. fit_transform (X[, y]) Fit and transform using the target information. get_feature_names () Deprecated method to get feature names. get_feature_names_in () Get the names of all input columns present when fitting. get_feature_names_out ([input_features]) Get the names of all transformed / added columns. get_metadata_routing () Get metadata routing of this object. get_params ([deep]) Get parameters for this estimator. set_output ([, transform]) Set output container. set_params (params) Set the parameters of this estimator. set_transform_request ([, override_return_df]) Configure whether metadata should be requested to be passed to the transform method. transform (X[, y, override_return_df]) Perform the transformation to new categorical data. References [ 1 ] Parametric empirical Bayes inference: Theory and applications, equations 1.19 & 1.20, from https://www.jstor.org/stable/2287098 [ 2 ] Empirical Bayes for multiple sample sizes, from http://chris-said.io/2017/05/03/empirical-bayes-for-multiple-sample-sizes/ [ 3 ] Shrinkage Estimation of Log-odds Ratios for Comparing Mobility Tables, from https://journals.sagepub.com/doi/abs/10.1177/0081175015570097 [ 4 ] Stein’s paradox and group rationality, from http://www.philos.rug.nl/~romeyn/presentation/ 2017_romeijn_ -_Paris_Stein.pdf [ 5 ] Stein’s Paradox in Statistics, from http://statweb.stanford.edu/~ckirby/brad/other/Article1977.pdf fit ( X : ndarray \| DataFrame \| list \| generic \| csr_matrix , y : list \| Series \| ndarray \| tuple \| DataFrame \| None = None , kwargs )  Fits the encoder according to X and y. Parameters : X array-like, shape = [n_samples, n_features] Training vectors, where n_samples is the number of samples and n_features is the number of features. y array-like, shape = [n_samples] Target values. Returns : self encoder Returns self. fit_transform ( X : ndarray \| DataFrame \| list \| generic \| csr_matrix , y : list \| Series \| ndarray \| tuple \| DataFrame \| None = None , fit_params )  Fit and transform using the target information. This also uses the target for transforming, not only for training. get_feature_names ( ) → ndarray  Deprecated method to get feature names. Use get_feature_names_out instead. get_feature_names_in ( ) → ndarray  Get the names of all input columns present when fitting. These columns are necessary for the transform step. get_feature_names_out ( input_features = None ) → ndarray  Get the names of all transformed / added columns. Note that in sklearn the get_feature_names_out function takes the feature_names_in as an argument and determines the output feature names using the input. A fit is usually not necessary and if so a NotFittedError is raised. We just require a fit all the time and return the fitted output columns. Returns : feature_names: np.ndarray A numpy array with all feature names transformed or added. Note: potentially dropped features (because the feature is constant/invariant) are not included! get_metadata_routing ( )  Get metadata routing of this object. Please check User Guide on how the routing mechanism works. Returns : routing MetadataRequest A MetadataRequest encapsulating routing information. get_params ( deep = True )  Get parameters for this estimator. Parameters : deep bool, default=True If True, will return the parameters for this estimator and contained subobjects that are estimators. Returns : params dict Parameter names mapped to their values. set_output ( * , transform = None )  Set output container. See sphx_glr_auto_examples_miscellaneous_plot_set_output.py for an example on how to use the API. Parameters : transform {“default”, “pandas”, “polars”}, default=None Configure output of transform and fit_transform . “default” : Default output format of a transformer “pandas” : DataFrame output “polars” : Polars output None : Transform configuration is unchanged Added in version 1.4: “polars” option was added. Returns : self estimator instance Estimator instance. set_params ( params )  Set the parameters of this estimator. The method works on simple estimators as well as on nested objects (such as Pipeline ). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object. Parameters : params dict Estimator parameters. Returns : self estimator instance Estimator instance. set_transform_request ( * , override_return_df : bool \| None \| str = '$UNCHANGED$' ) → JamesSteinEncoder  Configure whether metadata should be requested to be passed to the transform method. Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config() ). Please check the User Guide on how the routing mechanism works. The options for each parameter are: True : metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided. False : metadata is not requested and the meta-estimator will not pass it to transform . None : metadata is not requested, and the meta-estimator will raise an error if the user provides it. str : metadata should be passed to the meta-estimator with this given alias instead of the original name. The default ( sklearn.utils.metadata_routing.UNCHANGED ) retains the existing request. This allows you to change the request for some parameters and not others. Added in version 1.3. Parameters : override_return_df str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED Metadata routing for override_return_df parameter in transform . Returns : self object The updated object. transform ( X : ndarray \| DataFrame \| list \| generic \| csr_matrix , y : list \| Series \| ndarray \| tuple \| DataFrame \| None = None , override_return_df : bool = False )  Perform the transformation to new categorical data. Some encoders behave differently on whether y is given or not. This is mainly due to regularisation in order to avoid overfitting. On training data transform should be called with y, on test data without. Parameters : X array-like, shape = [n_samples, n_features] y array-like, shape = [n_samples] or None override_return_df bool override self.return_df to force to return a data frame Returns : p array or DataFrame, shape = [n_samples, n_features_out] Transformed values with encoding applied.
Markdown	[Category Encoders](https://contrib.scikit-learn.org/category_encoders/index.html) - [Backward Difference Coding](https://contrib.scikit-learn.org/category_encoders/backward_difference.html) - [BaseN](https://contrib.scikit-learn.org/category_encoders/basen.html) - [Binary](https://contrib.scikit-learn.org/category_encoders/binary.html) - [CatBoost Encoder](https://contrib.scikit-learn.org/category_encoders/catboost.html) - [Count Encoder](https://contrib.scikit-learn.org/category_encoders/count.html) - [Generalized Linear Mixed Model Encoder](https://contrib.scikit-learn.org/category_encoders/glmm.html) - [Gray](https://contrib.scikit-learn.org/category_encoders/gray.html) - [Hashing](https://contrib.scikit-learn.org/category_encoders/hashing.html) - [Helmert Coding](https://contrib.scikit-learn.org/category_encoders/helmert.html) - [James-Stein Encoder](https://contrib.scikit-learn.org/category_encoders/jamesstein.html) - [`JamesSteinEncoder`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder) - [`JamesSteinEncoder.fit()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit) - [`JamesSteinEncoder.fit_transform()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit_transform) - [`JamesSteinEncoder.get_feature_names()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names) - [`JamesSteinEncoder.get_feature_names_in()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in) - [`JamesSteinEncoder.get_feature_names_out()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out) - [`JamesSteinEncoder.get_metadata_routing()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing) - [`JamesSteinEncoder.get_params()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_params) - [`JamesSteinEncoder.set_output()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_output) - [`JamesSteinEncoder.set_params()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_params) - [`JamesSteinEncoder.set_transform_request()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_transform_request) - [`JamesSteinEncoder.transform()`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.transform) - [Leave One Out](https://contrib.scikit-learn.org/category_encoders/leaveoneout.html) - [M-estimate](https://contrib.scikit-learn.org/category_encoders/mestimate.html) - [One Hot](https://contrib.scikit-learn.org/category_encoders/onehot.html) - [Ordinal](https://contrib.scikit-learn.org/category_encoders/ordinal.html) - [Polynomial Coding](https://contrib.scikit-learn.org/category_encoders/polynomial.html) - [Quantile Encoder](https://contrib.scikit-learn.org/category_encoders/quantile.html) - [RankHotEncoder](https://contrib.scikit-learn.org/category_encoders/rankhot.html) - [Sum Coding](https://contrib.scikit-learn.org/category_encoders/sum.html) - [Summary Encoder](https://contrib.scikit-learn.org/category_encoders/summary.html) - [Target Encoder](https://contrib.scikit-learn.org/category_encoders/targetencoder.html) - [Weight of Evidence](https://contrib.scikit-learn.org/category_encoders/woe.html) - [Wrappers](https://contrib.scikit-learn.org/category_encoders/wrapper.html) [Category Encoders](https://contrib.scikit-learn.org/category_encoders/index.html) - James-Stein Encoder - [View page source](https://contrib.scikit-learn.org/category_encoders/_sources/jamesstein.rst.txt) *** # James-Stein Encoder[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#james-stein-encoder "Link to this heading") class category\_encoders.james\_stein.JamesSteinEncoder(verbose\=0, cols\=None, drop\_invariant\=False, return\_df\=True, handle\_unknown\='value', handle\_missing\='value', model\='independent', random\_state\=None, randomized\=False, sigma\=0\.05)[\[source\]](https://contrib.scikit-learn.org/category_encoders/_modules/category_encoders/james_stein.html#JamesSteinEncoder)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder "Link to this definition") James-Stein estimator. Supported targets: binomial and continuous. For polynomial target support, see PolynomialWrapper. For feature value i, James-Stein estimator returns a weighted average of: > 1. The mean target value for the observed feature value i. > 2. The mean target value (regardless of the feature value). This can be written as: ``` JS_i = (1 - B) * mean(y_i) + B * mean(y) ``` The question is, what should be the weight B? If we put too much weight on the conditional mean value, we will overfit. If we put too much weight on the global mean, we will underfit. The canonical solution in machine learning is to perform cross-validation. However, Charles Stein came with a closed-form solution to the problem. The intuition is: If the estimate of mean(y\_i) is unreliable (y\_i has high variance), we should put more weight on mean(y). Stein put it into an equation as: ``` B = var(y_i) / (var(y_i) + var(y)) ``` The only remaining issue is that we do not know var(y), let alone var(y\_i). Hence, we have to estimate the variances. But how can we reliably estimate the variances, when we already struggle with the estimation of the mean values?! There are multiple solutions: > 1\. If we have the same count of observations for each feature value i and all y\_i are close to each other, we can pretend that all var(y\_i) are identical. This is called a pooled model. 2. If the observation counts are not equal, it makes sense to replace the variances with squared standard errors, which penalize small observation counts: > ``` > SE^2 = var(y)/count(y) > ``` > This is called an independent model. James-Stein estimator has, however, one practical limitation - it was defined only for normal distributions. If you want to apply it for binary classification, which allows only values {0, 1}, it is better to first convert the mean target value from the bound interval \<0,1\> into an unbounded interval by replacing mean(y) with log-odds ratio: ``` log-odds_ratio_i = log(mean(y_i)/mean(y_not_i)) ``` This is called binary model. The estimation of parameters of this model is, however, tricky and sometimes it fails fatally. In these situations, it is better to use beta model, which generally delivers slightly worse accuracy than binary model but does not suffer from fatal failures. Parameters: verbose: int integer indicating verbosity of the output. 0 for none. cols: list a list of columns to encode, if None, all string columns will be encoded. drop\_invariant: bool boolean for whether or not to drop encoded columns with 0 variance. return\_df: bool boolean for whether to return a pandas DataFrame from transform (otherwise it will be a numpy array). handle\_missing: str options are ‘return\_nan’, ‘error’ and ‘value’, defaults to ‘value’, which returns the prior probability. handle\_unknown: str options are ‘return\_nan’, ‘error’ and ‘value’, defaults to ‘value’, which returns the prior probability. model: str options are ‘pooled’, ‘beta’, ‘binary’ and ‘independent’, defaults to ‘independent’. randomized: bool, adds normal (Gaussian) distribution noise into training data in order to decrease overfitting (testing data are untouched). sigma: float standard deviation (spread or “width”) of the normal distribution. Methods \| \| \| \|---\|---\| \| [`fit`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit "category_encoders.james_stein.JamesSteinEncoder.fit")(X\[, y\]) \| Fits the encoder according to X and y. \| \| [`fit_transform`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit_transform "category_encoders.james_stein.JamesSteinEncoder.fit_transform")(X\[, y\]) \| Fit and transform using the target information. \| \| [`get_feature_names`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names "category_encoders.james_stein.JamesSteinEncoder.get_feature_names")() \| Deprecated method to get feature names. \| \| [`get_feature_names_in`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in "category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in")() \| Get the names of all input columns present when fitting. \| \| [`get_feature_names_out`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out "category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out")(\[input\_features\]) \| Get the names of all transformed / added columns. \| \| [`get_metadata_routing`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing "category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing")() \| Get metadata routing of this object. \| \| [`get_params`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_params "category_encoders.james_stein.JamesSteinEncoder.get_params")(\[deep\]) \| Get parameters for this estimator. \| \| [`set_output`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_output "category_encoders.james_stein.JamesSteinEncoder.set_output")(\\[, transform\]) \| Set output container. \| \| [`set_params`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_params "category_encoders.james_stein.JamesSteinEncoder.set_params")(\\params) \| Set the parameters of this estimator. \| \| [`set_transform_request`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_transform_request "category_encoders.james_stein.JamesSteinEncoder.set_transform_request")(\\[, override\_return\_df\]) \| Configure whether metadata should be requested to be passed to the `transform` method. \| \| [`transform`](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.transform "category_encoders.james_stein.JamesSteinEncoder.transform")(X\[, y, override\_return\_df\]) \| Perform the transformation to new categorical data. \| References \[1\] Parametric empirical Bayes inference: Theory and applications, equations 1.19 & 1.20, from <https://www.jstor.org/stable/2287098> \[2\] Empirical Bayes for multiple sample sizes, from <http://chris-said.io/2017/05/03/empirical-bayes-for-multiple-sample-sizes/> \[3\] Shrinkage Estimation of Log-odds Ratios for Comparing Mobility Tables, from <https://journals.sagepub.com/doi/abs/10.1177/0081175015570097> \[4\] Stein’s paradox and group rationality, from <http://www.philos.rug.nl/~romeyn/presentation/>[2017\_romeijn\_](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#id6)\-\_Paris\_Stein.pdf \[5\] Stein’s Paradox in Statistics, from <http://statweb.stanford.edu/~ckirby/brad/other/Article1977.pdf> fit(X: ndarray \\| DataFrame \\| list \\| generic \\| csr\_matrix, y: list \\| Series \\| ndarray \\| tuple \\| DataFrame \\| None \= None, \\kwargs)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit "Link to this definition") Fits the encoder according to X and y. Parameters: Xarray-like, shape = \[n\_samples, n\_features\] Training vectors, where n\_samples is the number of samples and n\_features is the number of features. yarray-like, shape = \[n\_samples\] Target values. Returns: selfencoder Returns self. fit\_transform(X: ndarray \\| DataFrame \\| list \\| generic \\| csr\_matrix, y: list \\| Series \\| ndarray \\| tuple \\| DataFrame \\| None \= None, \\fit\_params)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.fit_transform "Link to this definition") Fit and transform using the target information. This also uses the target for transforming, not only for training. get\_feature\_names() → ndarray[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names "Link to this definition") Deprecated method to get feature names. Use get\_feature\_names\_out instead. get\_feature\_names\_in() → ndarray[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_in "Link to this definition") Get the names of all input columns present when fitting. These columns are necessary for the transform step. get\_feature\_names\_out(input\_features\=None) → ndarray[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_feature_names_out "Link to this definition") Get the names of all transformed / added columns. Note that in sklearn the get\_feature\_names\_out function takes the feature\_names\_in as an argument and determines the output feature names using the input. A fit is usually not necessary and if so a NotFittedError is raised. We just require a fit all the time and return the fitted output columns. Returns: feature\_names: np.ndarray A numpy array with all feature names transformed or added. Note: potentially dropped features (because the feature is constant/invariant) are not included\! get\_metadata\_routing()[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_metadata_routing "Link to this definition") Get metadata routing of this object. Please check User Guide on how the routing mechanism works. Returns: routingMetadataRequest A `MetadataRequest` encapsulating routing information. get\_params(deep\=True)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.get_params "Link to this definition") Get parameters for this estimator. Parameters: deepbool, default=True If True, will return the parameters for this estimator and contained subobjects that are estimators. Returns: paramsdict Parameter names mapped to their values. set\_output(\, transform\=None)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_output "Link to this definition") Set output container. See sphx\_glr\_auto\_examples\_miscellaneous\_plot\_set\_output.py for an example on how to use the API. Parameters: transform{“default”, “pandas”, “polars”}, default=None Configure output of transform and fit\_transform. - “default”: Default output format of a transformer - “pandas”: DataFrame output - “polars”: Polars output - None: Transform configuration is unchanged Added in version 1.4: “polars” option was added. Returns: selfestimator instance Estimator instance. set\_params(\\params)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_params "Link to this definition") Set the parameters of this estimator. The method works on simple estimators as well as on nested objects (such as `Pipeline`). The latter have parameters of the form `<component>__<parameter>` so that it’s possible to update each component of a nested object. Parameters: \\paramsdict Estimator parameters. Returns: selfestimator instance Estimator instance. set\_transform\_request(\*, override\_return\_df: bool \\| None \\| str \= '\$UNCHANGED\$') → [JamesSteinEncoder](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder "category_encoders.james_stein.JamesSteinEncoder")[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.set_transform_request "Link to this definition") Configure whether metadata should be requested to be passed to the `transform` method. Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with `enable_metadata_routing=True` (see `sklearn.set_config()`). Please check the User Guide on how the routing mechanism works. The options for each parameter are: - `True`: metadata is requested, and passed to `transform` if provided. The request is ignored if metadata is not provided. - `False`: metadata is not requested and the meta-estimator will not pass it to `transform`. - `None`: metadata is not requested, and the meta-estimator will raise an error if the user provides it. - `str`: metadata should be passed to the meta-estimator with this given alias instead of the original name. The default (`sklearn.utils.metadata_routing.UNCHANGED`) retains the existing request. This allows you to change the request for some parameters and not others. Added in version 1.3. Parameters: override\_return\_dfstr, True, False, or None, default=sklearn.utils.metadata\_routing.UNCHANGED Metadata routing for `override_return_df` parameter in `transform`. Returns: selfobject The updated object. transform(X: ndarray \\| DataFrame \\| list \\| generic \\| csr\_matrix, y: list \\| Series \\| ndarray \\| tuple \\| DataFrame \\| None \= None, override\_return\_df: bool \= False)[](https://contrib.scikit-learn.org/category_encoders/jamesstein.html#category_encoders.james_stein.JamesSteinEncoder.transform "Link to this definition") Perform the transformation to new categorical data. Some encoders behave differently on whether y is given or not. This is mainly due to regularisation in order to avoid overfitting. On training data transform should be called with y, on test data without. Parameters: Xarray-like, shape = \[n\_samples, n\_features\] yarray-like, shape = \[n\_samples\] or None override\_return\_dfbool override self.return\_df to force to return a data frame Returns: parray or DataFrame, shape = \[n\_samples, n\_features\_out\] Transformed values with encoding applied. [Previous](https://contrib.scikit-learn.org/category_encoders/helmert.html "Helmert Coding") [Next](https://contrib.scikit-learn.org/category_encoders/leaveoneout.html "Leave One Out") ** © Copyright 2024, Paul Westenthanner, Will McGinnis. Built with [Sphinx](https://www.sphinx-doc.org/) using a [theme](https://github.com/readthedocs/sphinx_rtd_theme) provided by [Read the Docs](https://readthedocs.org/).
Readable Markdown	null
Shard	148 (laksa)
Root Hash	6052685795207125548
Unparsed URL	org,scikit-learn!contrib,/category_encoders/jamesstein.html s443