mlresearch.metrics
.BetaRecall¶
- class mlresearch.metrics.BetaRecall(scorer_synth=None, beta=0.05, n_neighbors=5, metric='euclidean', n_jobs=None)[source]¶
Checks whether the synthetic data is diverse enough to cover the variability of real data, i.e., a model should be able to generate a wide variety of good samples.
This is an implementation of the metric proposed in [1].
Warning
This metric is not listed in the
get_scorer_names
function since it is following an unconventional structure.- Parameters:
- scorer_synthfunction
Method used to map a dataset into a score, or a 1-dimensional projection of itself. The mapping should be modelled over the synthetic dataset.
- betafloat, default=0.05
Percentile used to determine the radius of the euclidean ball.
- n_neighborsint, default=5
Number of neighbors to use by default for computing the radius for each sample in X_real for scoring. Ignored if scorer_synth is not None.
- metricstr or callable, default=’euclidean’
Metric to use for distance computation. Default is “euclidean”, which results in the standard Euclidean distance. See the documentation of scipy.spatial.distance and the metrics listed in
distance_metrics
for valid metric values.If metric is a callable function, it takes two arrays representing 1D vectors as inputs and must return one value indicating the distance between those vectors. This works for Scipy’s metrics, but is less efficient than passing the metric name as a string.
Ignored if scorer_synth is not None.
- n_jobsint, default=None
The number of parallel jobs to run for neighbors search.
None
means 1 unless in ajoblib.parallel_backend
context.-1
means using all processors. Ignored if scorer_synth is not None.
- Attributes:
- center_float
Value of the center of the euclidean ball.
References
[1]Alaa, A., Van Breugel, B., Saveliev, E. S., & van der Schaar, M. (2022, June). How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning (pp. 290-306). PMLR.
- fit(X_synth)[source]¶
Compute statistics necessary to calculate $beta$-recall.
- Parameters:
- X_syntharray-like or pd.DataFrame, shape (n_samples, n_features)
The synthetic dataset used to fit self.scorer_synth.
- Returns:
- selfobject
Returns an instance of the class.
- get_metadata_routing()¶
Get metadata routing of this object.
Please check User Guide on how the routing mechanism works.
- Returns:
- routingMetadataRequest
A
MetadataRequest
encapsulating routing information.
- score(X_real)[source]¶
Returns 1 if a sample resides in the $beta$-support of the synthetic distribution, 0 otherwise.
- Parameters:
- X_realarray-like or pd.DataFrame, shape (n_samples, n_features)
Input data over which $beta$-recall will be calculated.
- Returns:
- scoresnp.ndarray, shape (n_samples,)
$beta$-recall scores.
- set_fit_request(*, X_synth: bool | None | str = '$UNCHANGED$') BetaRecall ¶
Request metadata passed to the
fit
method.Note that this method is only relevant if
enable_metadata_routing=True
(seesklearn.set_config()
). Please see User Guide on how the routing mechanism works.The options for each parameter are:
True
: metadata is requested, and passed tofit
if provided. The request is ignored if metadata is not provided.False
: metadata is not requested and the meta-estimator will not pass it tofit
.None
: metadata is not requested, and the meta-estimator will raise an error if the user provides it.str
: metadata should be passed to the meta-estimator with this given alias instead of the original name.
The default (
sklearn.utils.metadata_routing.UNCHANGED
) retains the existing request. This allows you to change the request for some parameters and not others.Added in version 1.3.
Note
This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a
Pipeline
. Otherwise it has no effect.- Parameters:
- X_synthstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED
Metadata routing for
X_synth
parameter infit
.
- Returns:
- selfobject
The updated object.
- set_score_request(*, X_real: bool | None | str = '$UNCHANGED$') BetaRecall ¶
Request metadata passed to the
score
method.Note that this method is only relevant if
enable_metadata_routing=True
(seesklearn.set_config()
). Please see User Guide on how the routing mechanism works.The options for each parameter are:
True
: metadata is requested, and passed toscore
if provided. The request is ignored if metadata is not provided.False
: metadata is not requested and the meta-estimator will not pass it toscore
.None
: metadata is not requested, and the meta-estimator will raise an error if the user provides it.str
: metadata should be passed to the meta-estimator with this given alias instead of the original name.
The default (
sklearn.utils.metadata_routing.UNCHANGED
) retains the existing request. This allows you to change the request for some parameters and not others.Added in version 1.3.
Note
This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a
Pipeline
. Otherwise it has no effect.- Parameters:
- X_realstr, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED
Metadata routing for
X_real
parameter inscore
.
- Returns:
- selfobject
The updated object.