Argthtjtr

The original implement is correct which only includes the loss1 and loss2, so I think my input data is correct.

But the training loss is always 'nan' after I added the loss3 named 'sw_loss' which aims at minimize the L2 norm between the rows of the 'features'. The 'features' are the output of the last layer of the network.

Actually, the trainging loss became 'nan' in the second batch of the first epoch, and the loss of first batch is about 2.2.

Here are the main codes:

features, _ = mnist_net(images) 



centers = func.construct_center(features, FLAGS.num_classes)

loss1 = func.dce_loss(features, labels, centers, FLAGS.temp)

loss2 = func.pl_loss(features, labels, centers)

loss3 = func.sw_loss(features, similarity_weight_batch)  #loss3 is defined in the following 

loss = loss1 + FLAGS.weight_pl * loss2 + FLAGS.weight_sw * loss3

eval_correct = func.evaluation(features, labels, centers)

train_op = func.training(loss, lr)



init = tf.global_variables_initializer()



# initialize the variables

sess = tf.Session()

sess.run(init)

#compute_centers(sess, add_op, count_op, average_op, images, labels, train_x, train_y)



# run the computation graph (train and test process)

epoch = 1

loss_before = np.inf

score_before = 0.0

stopping = 0

index = list(range(train_num))

np.random.shuffle(index)

batch_size = FLAGS.batch_size

batch_num = train_num//batch_size if train_num % batch_size==0 else train_num//batch_size+1

train_start= time.time()

while stopping<FLAGS.stop:

    time1 = time.time()

    loss_now = 0.0

    score_now = 0.0



    for i in range(batch_num):

        batch_x = train_x[index[i*batch_size:(i+1)*batch_size]]

        batch_y = train_y[index[i*batch_size:(i+1)*batch_size]]

        batch_index = np.asarray( index[i*batch_size:(i+1)*batch_size])

        weight_batch = np.zeros(shape=(batch_index.shape[0],batch_index.shape[0]))

        for j in range(batch_index.shape[0]):

            for k in range(batch_index.shape[0]):

                weight_batch[j,k] = similarity_weight[[batch_index[j,]],[batch_index[k,]]]

        result = sess.run([train_op, loss, eval_correct], feed_dict={images:batch_x,                

            labels:batch_y, lr:FLAGS.learning_rate, similarity_weight_batch:weight_batch})

        loss_now += result[1]

        score_now += result[2][1]

    score_now /= train_num

The sw_loss is defined in the function file as follows:.

def sw_loss(features, similarity_weight_batch):  #'similarity_weight_batch' is the coefficients，which is between(0,1]. 

    sw_loss_total = 0.0

    sqdiff = tf.squared_difference(features[:, tf.newaxis], features)

    feature_matrix = tf.sqrt(tf.reduce_sum(sqdiff, axis=-1)) #calculate the L2 norm between the rows of the features

    sw_loss_total = tf.multiply(similarity_weight_batch,feature_matrix)                                                                                                                                     

    return tf.reduce_mean(sw_loss_total)

The printed logs are as follows, the training loss is 'nan' in all epoch:

epoch 1: training: loss --> nan, acc --> 15.514%

time for this epoch: 0.074 minutes

epoch 2:  training: loss --> nan, acc --> 15.514%

time for this epoch: 0.024 minutes

epoch 3: training: loss --> nan, acc --> 15.514%

time for this epoch: 0.073 minutes

epoch 4: training: loss --> nan, acc --> 15.514%

time for this epoch: 0.033 minutes

epoch 5: training: loss --> nan, acc --> 15.514%

time for this epoch: 0.021 minutes

...