Loss, optimizer, metrics in the Keras

The next step after building the model schema with Keras is to perform the compile operation. At compile time, it is often necessary to specify three parameters

• Loss
• Optimizer
• Metrics

There are two types of options for these three parameters:

• Using strings
• Use identifiers, such as functions under the Keras.losses,keras.optimizers,metrics package

For example:

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)model.compile(loss='categorical_crossentropy',              optimizer=sgd,              metrics=['accuracy'])

Because strings can sometimes be used and identifiers can sometimes be used, it is very important to know how to do it behind the scenes. The following is a study of the acquisition of Optimizer,loss,metrics three objects respectively.

Optimizer

A model can have only one optimizer, and only one optimizer can be specified when the compilation is performed.
In keras.optimizers.py, there is a get function that gets an instance of the optimizer based on the optimizer parameter that the user passes in:

def get(identifier):    # 如果后端是tensorflow并且使用的是tensorflow自带的优化器实例，可以直接使用tensorflow原生的优化器     if K.backend() == 'tensorflow':        # Wrap TF optimizer instances        if isinstance(identifier, tf.train.Optimizer):            return TFOptimizer(identifier)    # 如果以json串的形式定义optimizer并进行参数配置    if isinstance(identifier, dict):        return deserialize(identifier)    elif isinstance(identifier, six.string_types):        # 如果以字符串形式指定optimizer，那么使用优化器的默认配置参数        config = {'class_name': str(identifier), 'config': {}}        return deserialize(config)    if isinstance(identifier, Optimizer):        # 如果使用keras封装的Optimizer的实例        return identifier    else:        raise ValueError('Could not interpret optimizer identifier: ' +                         str(identifier))

The function of Deserilize (config) is to make an instance of optimizer deserialization.

Loss

The Keras.losses function also has a get (identifier) method. One of the following points to note:

If identifier is a callable function name, which is a custom loss function, the return value of this loss function is a tensor. This makes it easy to implement a custom loss function. In addition to using STR and dict types of identifier, we can also directly use the loss function below the keras.losses package.

def get(identifier):    if identifier is None:        return None    if isinstance(identifier, six.string_types):        identifier = str(identifier)        return deserialize(identifier)    if isinstance(identifier, dict):        return deserialize(identifier)    elif callable(identifier):        return identifier    else:        raise ValueError('Could not interpret '                         'loss function identifier:', identifier)

Metrics

In the Model.compile () function, both optimizer and loss are singular, and only metrics is a plural form. Because a model can specify only one optimizer and loss, it may indicate multiple metrics. Metrics is also the most complex of the three processing logic.

In the Keras most central place keras.engine.train.py has the following functions to deal with metrics. This function actually does two things:

• Find the specific metric corresponding function according to the input metric
• Calculate metric tensor

There are two steps to finding the metric correspondence function:

• using a string to indicate accuracy and cross-entropy
• using functions in keras.metrics.py

def handle_metrics (Metrics, weights=none): Metric_name_prefix = ' weighted_ ' If weights are not None else "for Metri            C in metrics: # If metrics is the most common kind: accuracy, cross-entropy if metric in (' Accuracy ', ' acc ', ' crossentropy ', ' CE '): # Custom handling of Accuracy/crossentropy # (because of class mode duality) Output_shape = K.i Nt_shape (Self.outputs[i]) # If the output dimension is 1 or the loss function is a two categorical loss function, then the description is a two classification problem and should use the crossover entropy if of the two classified accuracy and two classifications (Output_ SHAPE[-1] = = 1 or self.loss_functions[i] = = losses.binary_crossentropy): # case:binary Accu Racy/crossentropy if metric in (' Accuracy ', ' acc '): METRIC_FN = Metrics_module.binary_ac Curacy elif metric in (' crossentropy ', ' CE '): METRIC_FN = metrics_module.binary_crossent            Ropy # If the loss function is sparse_categorical_crossentropy, then the target y_input is not one-hot, so we need to use sparse's multi-class quasi-de-rate and sparse multi-class cross-entropy Elif Self.loss_functioNs[i] = = losses.sparse_categorical_crossentropy: # case:categorical Accuracy/crossentropy # With sparse targets if metric in (' Accuracy ', ' acc '): METRIC_FN = Metrics_module.sparse _categorical_accuracy elif metric in (' crossentropy ', ' CE '): METRIC_FN = Metrics_module. Sparse_categorical_crossentropy Else: # case:categorical Accuracy/crossentropy I F Metric in (' Accuracy ', ' acc '): METRIC_FN = Metrics_module.categorical_accuracy elif Me Tric in (' crossentropy ', ' CE '): METRIC_FN = metrics_module.categorical_crossentropy if Metri                    C in (' Accuracy ', ' acc '): suffix = ' acc ' elif metric in (' crossentropy ', ' CE '): suffix = ' ce ' WEIGHTED_METRIC_FN = weighted_masked_objective (metric_fn) metric_name = Metric_    Name_prefix + suffix    Else: # If the input metric is not a string, then call the Metrics module to get METRIC_FN = Metrics_module.get (Metric) Weig HTED_METRIC_FN = Weighted_masked_objective (METRIC_FN) # Get metric name As String if Hasattr (metric_            FN, ' name '): Metric_name = metric_fn.name Else:metric_name = metric_fn.__name__ Metric_name = Metric_name_prefix + metric_name with K.name_scope (metric_name): Metric_result                                                = Weighted_metric_fn (Y_true, y_pred, Weights=weights, Mask=masks[i] # Append to Self.metrics_names, Self.metric_tensors, # Self.statefu        L_metric_names If Len (self.output_names) > 1:metric_name = self.output_names[i] + ' _ ' + metric_name            # Dedupe Name j = 1 Base_metric_name = Metric_name while metric_name in Self.metrics_names: Metric_name = bAse_metric_name + ' _ ' + str (j) J + = 1 self.metrics_names.append (metric_name) self.metrics_tensors        . Append (Metric_result) # Keep track of the state updates created by # stateful metrics (i.e. metrics layers).            If Isinstance (METRIC_FN, Layer) and Metric_fn.stateful:self.stateful_metric_names.append (Metric_name) Self.stateful_metric_functions.append (METRIC_FN) self.metrics_updates + = Metric_fn.updates

No matter how you use metric, you will eventually become the function below the metrics package. When specifying accuracy and crossentropy in string form, Keras is very smart about which function to use under the metrics package. Because those metric functions under the metrics package have different usage scenarios, for example:

• Some deal with the One-hot form of y_input (the category of data), and some deal with non-one-hot forms of Y_input
• Some deal with the metric of the two classification problem, and some deal with the problem of multi-classification metric

When using the string "accuracy" and "crossentropy" to indicate metric, Keras determines which metric function should be used based on the loss function and the shape of the output layer. In any case, direct use of the Metrics function name below is always not an error.

There is also a get (identifier) function in the keras.metrics.py file to get the metric function.

def get(identifier):    if isinstance(identifier, dict):        config = {'class_name': str(identifier), 'config': {}}        return deserialize(config)    elif isinstance(identifier, six.string_types):        return deserialize(str(identifier))    elif callable(identifier):        return identifier    else:        raise ValueError('Could not interpret '                         'metric function identifier:', identifier)

If identifier is a string or dictionary, a metric function is deserialized according to identifier.
If identifier itself is a function name, then it returns the function name directly. This approach provides a great convenience for custom metric.

The design philosophy in Keras is perfect.

Loss, optimizer, metrics in the Keras