adapt.feature_based.ADDA

class adapt.feature_based.ADDA(encoder=None, task=None, discriminator=None, Xt=None, pretrain=True, tol=0.001, copy=True, verbose=1, random_state=None, **params)

ADDA: Adversarial Discriminative Domain Adaptation

ADDA is a feature-based domain adaptation method.

The purpose of ADDA is to build a new feature representation in which source and target data could not be distinguished by any discriminator network. This feature representation is built with two encoder networks:

• a source encoder trained to provide good features in order to learn the task on the source domain. The task is learned through a task network trained with the source encoder.

• a target encoder trained to fool a discriminator network which tries to classify source and target data in the encoded space. The target encoder and the discriminator are trained in an adversarial fashion in the same way as GAN.

The parameters of the four networks are optimized in a two stage algorithm where source encoder and task networks are first fitted according to the following optimization problem:

\[\min_{\phi_S, F} \mathcal{L}_{task}(F(\phi_S(X_S)), y_S)\]

In the second stage, target encoder and discriminator networks are fitted according to:

\[\begin{split}\min_{\phi_T} & \; - \log(D(\phi_T(X_T)))) \\ \min_{D} & \; - \log(D(\phi_S(X_S))) - \log(1 - D(\phi_T(X_T)))\end{split}\]

Where:

• \((X_S, y_S), (X_T)\) are respectively the labeled source data and the unlabeled target data.

• \(\phi_S, \phi_T, F, D\) are respectively the source encoder, the target encoder, the task and the discriminator networks.

The method has been originally introduced for unsupervised classification DA but it could be widen to other task in supervised DA straightforwardly.

Overview of the ADDA approach (source: [1])

Parameters

encodertensorflow Model (default=None)

Encoder netwok. If None, a shallow network with 10 neurons and ReLU activation is used as encoder network.

tasktensorflow Model (default=None)

Task netwok. If None, a two layers network with 10 neurons per layer and ReLU activation is used as task network.

discriminatortensorflow Model (default=None)

Discriminator netwok. If None, a two layers network with 10 neurons per layer and ReLU activation is used as discriminator network. Note that the output shape of the discriminator should be (None, 1) and a sigmoid activation should be used.

Xtnumpy array (default=None)

Target input data.

pretrainbool (default=True)

Weither to perform pretraining of the encoder_src_ and task_ networks on source data or not. separated compile and fit arguments for the pretraining can be given by using the prefix pretrain__ as pretrain__epochs=10 or pretrain__learning_rate=0.1 for instance. If no pretrain arguments are given, the training arguments are used by default

tolfloat (default=0.001)

Tolerance on the loss for early stopping of pretraining.

copyboolean (default=True)

Whether to make a copy of estimator or not.

verboseint (default=1)

Verbosity level.

random_stateint (default=None)

Seed of random generator.

paramskey, value arguments

Arguments given at the different level of the adapt object. It can be, for instance, compile or fit parameters of the estimator or kernel parameters etc… Accepted parameters can be found by calling the method _get_legal_params(params).

Yields

optimizerstr or instance of tf.keras.optimizers (default=”rmsprop”)

Optimizer for the task. It should be an instance of tf.keras.optimizers as: tf.keras.optimizers.SGD(0.001) or tf.keras.optimizers.Adam(lr=0.001, beta_1=0.5). A string can also be given as "adam". Default optimizer is rmsprop.

lossstr or instance of tf.keras.losses (default=”mse”)

Loss for the task. It should be an instance of tf.keras.losses as: tf.keras.losses.MeanSquaredError() or tf.keras.losses.CategoricalCrossentropy(). A string can also be given as "mse" or categorical_crossentropy. Default loss is mse.

metricslist of str or list of tf.keras.metrics.Metric instance

List of metrics to be evaluated by the model during training and testing. Typically you will use metrics=['accuracy'].

optimizer_encstr or instance of tf.keras.optimizers

If the Adapt Model has an encoder attribute, a specific optimizer for the encoder network can be given. Typically, this parameter can be used to give a smaller learning rate to the encoder. If not specified, optimizer_enc=optimizer.

optimizer_discstr or instance of tf.keras.optimizers

If the Adapt Model has a discriminator attribute, a specific optimizer for the discriminator network can be given. If not specified, optimizer_disc=optimizer.

kwargskey, value arguments

Any arguments of the fit method from the Tensorflow Model can be given, as epochs and batch_size. Specific arguments from optimizer can also be given as learning_rate or beta_1 for Adam. This allows to perform GridSearchCV from scikit-learn on these arguments.

See also

DANN

DeepCORAL

References

1

[1] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. “Adversarial discriminative domain adaptation”. In CVPR, 2017.

Examples

>>> from adapt.utils import make_classification_da
>>> from adapt.feature_based import ADDA
>>> Xs, ys, Xt, yt = make_classification_da()
>>> model = ADDA(Xt=Xt, metrics=["acc"], random_state=0)
>>> model.fit(Xs, ys, epochs=100, verbose=0)
>>> model.score(Xt, yt)
1/1 [==============================] - 0s 153ms/step - loss: 0.0960 - acc: 0.9300
0.09596743434667587

Attributes

encoder_tensorflow Model

encoder network.

task_tensorflow Model

task network.

discriminator_tensorflow Model

discriminator network.

encoder_src_tensorflow Model

Source encoder network

history_dict

history of the losses and metrics across the epochs. If yt is given in fit method, target metrics and losses are recorded too.

Methods

__init__([encoder, task, discriminator, Xt, ...])
compile([optimizer, loss, metrics, ...])	Configures the model for training.
fit(X[, y, Xt, yt, domains])	Fit Model.
get_params([deep])	Get parameters for this estimator.
predict(x[, batch_size, verbose, steps, ...])	Generates output predictions for the input samples.
predict_disc(X[, domain])	Return predictions of the discriminator on the encoded features.
predict_task(X[, domain])	Return predictions of the task on the encoded features.
pretrain_step(data)
score(X, y[, sample_weight])	Return the evaluation of the model on X, y.
set_params(**params)	Set the parameters of this estimator.
transform(X[, domain])	Return the encoded features of X.
unsupervised_score(Xs, Xt)	Return unsupervised score.

__init__(encoder=None, task=None, discriminator=None, Xt=None, pretrain=True, tol=0.001, copy=True, verbose=1, random_state=None, **params)

compile(optimizer=None, loss=None, metrics=None, loss_weights=None, weighted_metrics=None, run_eagerly=None, steps_per_execution=None, **kwargs)

Configures the model for training.

Parameters

optimizer: str or `tf.keras.optimizer` instance

Optimizer

loss: str or `tf.keras.losses.Loss` instance

Loss function. A loss function is any callable with the signature loss = fn(y_true, y_pred), where y_true are the ground truth values, and y_pred are the model’s predictions. y_true should have shape (batch_size, d0, .. dN) (except in the case of sparse loss functions such as sparse categorical crossentropy which expects integer arrays of shape (batch_size, d0, .. dN-1)). y_pred should have shape (batch_size, d0, .. dN). The loss function should return a float tensor. If a custom Loss instance is used and reduction is set to None, return value has shape (batch_size, d0, .. dN-1) i.e. per-sample or per-timestep loss values; otherwise, it is a scalar. If the model has multiple outputs, you can use a different loss on each output by passing a dictionary or a list of losses. The loss value that will be minimized by the model will then be the sum of all individual losses, unless loss_weights is specified.

metrics: list of str or list of `tf.keras.metrics.Metric` instance

List of metrics to be evaluated by the model during training and testing. Typically you will use metrics=[‘accuracy’]. A function is any callable with the signature result = fn(y_true, y_pred). To specify different metrics for different outputs of a multi-output model, you could also pass a dictionary, such as metrics={‘output_a’: ‘accuracy’, ‘output_b’: [‘accuracy’, ‘mse’]}. You can also pass a list to specify a metric or a list of metrics for each output, such as metrics=[[‘accuracy’], [‘accuracy’, ‘mse’]] or metrics=[‘accuracy’, [‘accuracy’, ‘mse’]]. When you pass the strings ‘accuracy’ or ‘acc’, we convert this to one of tf.keras.metrics.BinaryAccuracy, tf.keras.metrics.CategoricalAccuracy, tf.keras.metrics.SparseCategoricalAccuracy based on the loss function used and the model output shape. We do a similar conversion for the strings ‘crossentropy’ and ‘ce’ as well.

loss_weights: List or dict of floats

Scalars to weight the loss contributions of different model outputs. The loss value that will be minimized by the model will then be the weighted sum of all individual losses, weighted by the loss_weights coefficients. If a list, it is expected to have a 1:1 mapping to the model’s outputs. If a dict, it is expected to map output names (strings) to scalar coefficients.

weighted_metrics: list of metrics

List of metrics to be evaluated and weighted by sample_weight or class_weight during training and testing.

run_eagerly: bool (default=False)

If True, this Model’s logic will not be wrapped in a tf.function. Recommended to leave this as None unless your Model cannot be run inside a tf.function. run_eagerly=True is not supported when using tf.distribute.experimental.ParameterServerStrategy.

steps_per_execution: int (default=1)

The number of batches to run during each tf.function call. Running multiple batches inside a single tf.function call can greatly improve performance on TPUs or small models with a large Python overhead. At most, one full epoch will be run each execution. If a number larger than the size of the epoch is passed, the execution will be truncated to the size of the epoch. Note that if steps_per_execution is set to N, Callback.on_batch_begin and Callback.on_batch_end methods will only be called every N batches (i.e. before/after each tf.function execution).

**kwargs: key, value arguments

Arguments supported for backwards compatibility only.

Returns

None: None

fit(X, y=None, Xt=None, yt=None, domains=None, **fit_params)

Fit Model. Note that fit does not reset the model but extend the training.

Notice also that the compile method will be called if the model has not been compiled yet.

Parameters

Xarray or Tensor

Source input data.

yarray or Tensor (default=None)

Source output data.

Xtarray (default=None)

Target input data. If None, the Xt argument given in init is used.

ytarray (default=None)

Target input data. Only needed for supervised and semi-supervised Adapt model. If None, the yt argument given in init is used.

domainsarray (default=None)

Vector giving the domain for each source data. Can be used for multisource purpose.

fit_paramskey, value arguments

Arguments given to the fit method of the model (epochs, batch_size, callbacks…).

Returns

selfreturns an instance of self

get_params(deep=True)

Get parameters for this estimator.

Parameters

deepbool, default=True

Not used, here for scikit-learn compatibility.

Returns

paramsdict

Parameter names mapped to their values.

predict(x, batch_size=None, verbose=0, steps=None, callbacks=None, max_queue_size=10, workers=1, use_multiprocessing=False)

Generates output predictions for the input samples.

Computation is done in batches. This method is designed for performance in large scale inputs. For small amount of inputs that fit in one batch, directly using __call__() is recommended for faster execution, e.g., model(x), or model(x, training=False) if you have layers such as tf.keras.layers.BatchNormalization that behaves differently during inference. Also, note the fact that test loss is not affected by regularization layers like noise and dropout.

Parameters

x: array

Input samples.

batch_size: int (default=`None`)

Number of samples per batch. If unspecified, batch_size will default to 32. Do not specify the batch_size if your data is in the form of dataset, generators, or keras.utils.Sequence instances (since they generate batches).

verbose: int (default=0)

Verbosity mode, 0 or 1.

steps: int (default=None)

Total number of steps (batches of samples) before declaring the prediction round finished. Ignored with the default value of None. If x is a tf.data dataset and steps is None, predict() will run until the input dataset is exhausted.

callbacks: List of `keras.callbacks.Callback` instances.

List of callbacks to apply during prediction. See [callbacks](/api_docs/python/tf/keras/callbacks).

max_queue_size: int (default=10)

Used for generator or keras.utils.Sequence input only. Maximum size for the generator queue. If unspecified, max_queue_size will default to 10.

workers: int (default=1)

Used for generator or keras.utils.Sequence input only. Maximum number of processes to spin up when using process-based threading. If unspecified, workers will default to 1.

use_multiprocessing: bool (default=False)

Used for generator or keras.utils.Sequence input only. If True, use process-based threading. If unspecified, use_multiprocessing will default to False. Note that because this implementation relies on multiprocessing, you should not pass non-picklable arguments to the generator as they can’t be passed easily to children processes.

Returns

y_predarray

Numpy array(s) of predictions.

predict_disc(X, domain='tgt')

Return predictions of the discriminator on the encoded features.

Parameters

Xarray

input data

domain: str (default=”tgt”)

If domain is "tgt" or "target", the target encoder is used. If domain is "src" or "source", the source encoder is used.

Returns

y_discarray

predictions of discriminator network

predict_task(X, domain='tgt')

Return predictions of the task on the encoded features.

Parameters

Xarray

input data

domain: str (default=”tgt”)

If domain is "tgt" or "target", the target encoder is used. If domain is "src" or "source", the source encoder is used.

Returns

y_taskarray

predictions of task network

pretrain_step(data)

score(X, y, sample_weight=None)

Return the evaluation of the model on X, y.

Call evaluate on tensorflow Model.

Parameters

Xarray

input data

yarray

output data

sample_weightarray (default=None)

Sample weights

Returns

scorefloat

Score.

set_params(**params)

Set the parameters of this estimator.

Parameters

**paramsdict

Estimator parameters.

Returns

selfestimator instance

Estimator instance.

transform(X, domain='tgt')

Return the encoded features of X.

Parameters

Xarray

input data

domain: str (default=”tgt”)

If domain is "tgt" or "target", the target encoder is used. If domain is "src" or "source", the source encoder is used.

Returns

X_encarray

predictions of encoder network

unsupervised_score(Xs, Xt)

Return unsupervised score.

The normalized discrepancy distance is computed between the reweighted/transformed source input data and the target input data.

Parameters

Xsarray

Source input data.

Xtarray

Source input data.

Returns

scorefloat

Unsupervised score.

Examples

Two Moons

Rotation