title: Cannot have access to all data when enumerating over the dataloader

body:
I defined a custom Dataset and a custom Dataloader, and I want to access all the batches using for i,batch in enumerate(loader). But this for loop gives me different number of batches in every epoch, and all of them are far smaller then the actual number of batches (which equals to number_of_samples/batch_size).
Here is how I define my dataset and dataloader:

class UsptoDataset(Dataset):
    def __init__(self, csv_file):
        df = pd.read_csv(csv_file)
        self.rea_trees = df['reactants_trees'].to_numpy()
        self.syn_trees = df['synthons_trees'].to_numpy()
        self.syn_smiles = df['synthons'].to_numpy()
        self.product_smiles = df['product'].to_numpy()

    def __len__(self):
        return len(self.rea_trees)

    def __getitem__(self, item):
        rea_tree = self.rea_trees[item]
        syn_tree = self.syn_trees[item]
        syn_smile = self.syn_smiles[item]
        pro_smile = self.product_smiles[item]
        # omit the snippet used to process the data here, which gives us the variables used in the return statement.
        return {'input_words': input_words,
                'input_chars': input_chars,
                'syn_tree_indices': syn_tree_indices,
                'syn_rule_nl_left': syn_rule_nl_left,
                'syn_rule_nl_right': syn_rule_nl_right,
                'rea_tree_indices': rea_tree_indices,
                'rea_rule_nl_left': rea_rule_nl_left,
                'rea_rule_nl_right': rea_rule_nl_right,
                'class_mask': class_mask,
                'query_paths': query_paths,
                'labels': labels,
                'parent_matrix': parent_matrix,
                'syn_parent_matrix': syn_parent_matrix,
                'path_lens': path_lens,
                'syn_path_lens': syn_path_lens}

    @staticmethod
    def collate_fn(batch):
        input_words = torch.tensor(np.stack([_['input_words'] for _ in batch], axis=0), dtype=torch.long)
        input_chars = torch.tensor(np.stack([_['input_chars'] for _ in batch], axis=0), dtype=torch.long)
        syn_tree_indices = torch.tensor(np.stack([_['syn_tree_indices'] for _ in batch], axis=0), dtype=torch.long)
        syn_rule_nl_left = torch.tensor(np.stack([_['syn_rule_nl_left'] for _ in batch], axis=0), dtype=torch.long)
        syn_rule_nl_right = torch.tensor(np.stack([_['syn_rule_nl_right'] for _ in batch], axis=0), dtype=torch.long)
        rea_tree_indices = torch.tensor(np.stack([_['rea_tree_indices'] for _ in batch], axis=0), dtype=torch.long)
        rea_rule_nl_left = torch.tensor(np.stack([_['rea_rule_nl_left'] for _ in batch], axis=0), dtype=torch.long)
        rea_rule_nl_right = torch.tensor(np.stack([_['rea_rule_nl_right'] for _ in batch], axis=0), dtype=torch.long)
        class_mask = torch.tensor(np.stack([_['class_mask'] for _ in batch], axis=0), dtype=torch.float32)
        query_paths = torch.tensor(np.stack([_['query_paths'] for _ in batch], axis=0), dtype=torch.long)
        labels = torch.tensor(np.stack([_['labels'] for _ in batch], axis=0), dtype=torch.long)
        parent_matrix = torch.tensor(np.stack([_['parent_matrix'] for _ in batch], axis=0), dtype=torch.float)
        syn_parent_matrix = torch.tensor(np.stack([_['syn_parent_matrix'] for _ in batch], axis=0), dtype=torch.float)
        path_lens = torch.tensor(np.stack([_['path_lens'] for _ in batch], axis=0), dtype=torch.long)
        syn_path_lens = torch.tensor(np.stack([_['syn_path_lens'] for _ in batch], axis=0), dtype=torch.long)

        return_dict = {'input_words': input_words,
                       'input_chars': input_chars,
                       'syn_tree_indices': syn_tree_indices,
                       'syn_rule_nl_left': syn_rule_nl_left,
                       'syn_rule_nl_right': syn_rule_nl_right,
                       'rea_tree_indices': rea_tree_indices,
                       'rea_rule_nl_left': rea_rule_nl_left,
                       'rea_rule_nl_right': rea_rule_nl_right,
                       'class_mask': class_mask,
                       'query_paths': query_paths,
                       'labels': labels,
                       'parent_matrix': parent_matrix,
                       'syn_parent_matrix': syn_parent_matrix,
                       'path_lens': path_lens,
                       'syn_path_lens': syn_path_lens}
        return return_dict

train_dataset=UsptoDataset("train_trees.csv")

train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True, num_workers=1, collate_fn=UsptoDataset.collate_fn)

And when I use the dataloader as follows, it gives me different number of batches every epoch:
epoch_steps = len(train_loader)
for e in range(epochs):
    for j, batch_data in enumerate(train_loader):
        step = e * epoch_steps + j

The log shows that the first epoch only has 5 batches, the second epoch has 3 batches, and the third epoch has 5 batches, and so on.
 1 Config:
  2 Namespace(batch_size_per_gpu=4, epochs=400, eval_every_epoch=1, hidden_size=128, keep=10, log_every_step=1, lr=0.001, new_model=False, save_dir='saved_model/', workers=1)
  3 2021-01-06 15:33:17,909 - __main__ - WARNING - Checkpoints not found in dir saved_model/, creating a new model.
  4 2021-01-06 15:33:18,340 - __main__ - INFO - Step: 0, Loss: 5.4213, Rule acc: 0.1388
  5 2021-01-06 15:33:18,686 - __main__ - INFO - Step: 1, Loss: 4.884, Rule acc: 0.542
  6 2021-01-06 15:33:18,941 - __main__ - INFO - Step: 2, Loss: 4.6205, Rule acc: 0.6122
  7 2021-01-06 15:33:19,174 - __main__ - INFO - Step: 3, Loss: 4.4442, Rule acc: 0.61
  8 2021-01-06 15:33:19,424 - __main__ - INFO - Step: 4, Loss: 4.3033, Rule acc: 0.6211
  9 2021-01-06 15:33:20,684 - __main__ - INFO - Dev Loss: 3.5034, Dev Sample Acc: 0.0, Dev Rule Acc: 0.5970844200679234, in epoch 0
 10 2021-01-06 15:33:22,203 - __main__ - INFO - Test Loss: 3.4878, Test Sample Acc: 0.0, Test Rule Acc: 0.6470248053471247
 11 2021-01-06 15:33:22,394 - __main__ - INFO - Found better dev sample accuracy 0.0 in epoch 0
 12 2021-01-06 15:33:22,803 - __main__ - INFO - Step: 10002, Loss: 3.6232, Rule acc: 0.6555
 13 2021-01-06 15:33:23,046 - __main__ - INFO - Step: 10003, Loss: 3.53, Rule acc: 0.6442
 14 2021-01-06 15:33:23,286 - __main__ - INFO - Step: 10004, Loss: 3.4907, Rule acc: 0.6498
 15 2021-01-06 15:33:24,617 - __main__ - INFO - Dev Loss: 3.3081, Dev Sample Acc: 0.0, Dev Rule Acc: 0.5980878387178693, in epoch 1
 16 2021-01-06 15:33:26,215 - __main__ - INFO - Test Loss: 3.2859, Test Sample Acc: 0.0, Test Rule Acc: 0.6466992994149526
 17 2021-01-06 15:33:26,857 - __main__ - INFO - Step: 20004, Loss: 3.3965, Rule acc: 0.6493
 18 2021-01-06 15:33:27,093 - __main__ - INFO - Step: 20005, Loss: 3.3797, Rule acc: 0.6314
 19 2021-01-06 15:33:27,353 - __main__ - INFO - Step: 20006, Loss: 3.3959, Rule acc: 0.5727
 20 2021-01-06 15:33:27,609 - __main__ - INFO - Step: 20007, Loss: 3.3632, Rule acc: 0.6279
 21 2021-01-06 15:33:27,837 - __main__ - INFO - Step: 20008, Loss: 3.3331, Rule acc: 0.6158
 22 2021-01-06 15:33:29,122 - __main__ - INFO - Dev Loss: 3.0911, Dev Sample Acc: 0.0, Dev Rule Acc: 0.6016287207603455, in epoch 2
 23 2021-01-06 15:33:30,689 - __main__ - INFO - Test Loss: 3.0651, Test Sample Acc: 0.0, Test Rule Acc: 0.6531393428643545
 24 2021-01-06 15:33:32,143 - __main__ - INFO - Dev Loss: 3.0911, Dev Sample Acc: 0.0, Dev Rule Acc: 0.6016287207603455, in epoch 3
 25 2021-01-06 15:33:33,765 - __main__ - INFO - Test Loss: 3.0651, Test Sample Acc: 0.0, Test Rule Acc: 0.6531393428643545
 26 2021-01-06 15:33:34,359 - __main__ - INFO - Step: 40008, Loss: 3.108, Rule acc: 0.6816
 27 2021-01-06 15:33:34,604 - __main__ - INFO - Step: 40009, Loss: 3.0756, Rule acc: 0.6732
 28 2021-01-06 15:33:35,823 - __main__ - INFO - Dev Loss: 3.0419, Dev Sample Acc: 0.0, Dev Rule Acc: 0.613776079245976, in epoch 4

FYI, the value of len(train_loader.dataset), batch_size and len(train_loader) are 40008, 4 and 10002 respectively, which are exactly what I expected. So it is so confusing that using enumerate gives me only several batches such as 3 or 5 (10002 is expected).


title: How can I express this custom loss function in tensorflow?

body:
I've got a loss function that fulfills my needs, but is only in PyTorch. I need to implement it into my TensorFlow code, but while most of it can trivially be "translated" I am stuck with a particular line:
y_hat[:, torch.arange(N), torch.arange(N)] = torch.finfo(y_hat.dtype).max  # to be "1" after sigmoid

You can see the whole code in following and it is indeed pretty straight forward except for that line:
def get_loss(y_hat, y):
 # No loss on diagonal
 B, N, _ = y_hat.shape
 y_hat[:, torch.arange(N), torch.arange(N)] = torch.finfo(y_hat.dtype).max  # to be "1" after sigmoid

 # calc loss
 loss = F.binary_cross_entropy_with_logits(y_hat, y)  # cross entropy

 y_hat = torch.sigmoid(y_hat)
 tp = (y_hat * y).sum(dim=(1, 2))
 fn = ((1. - y_hat) * y).sum(dim=(1, 2))
 fp = (y_hat * (1. - y)).sum(dim=(1, 2))
 loss = loss - ((2 * tp) / (2 * tp + fp + fn + 1e-10)).sum()  # fscore

return loss

So far I came up with following:
def get_loss(y_hat, y):
 loss = tf.keras.losses.BinaryCrossentropy()(y_hat,y)  # cross entropy (but no logits)

 y_hat = tf.math.sigmoid(y_hat)

 tp = tf.math.reduce_sum(tf.multiply(y_hat, y),[1,2])
 fn = tf.math.reduce_sum((y - tf.multiply(y_hat, y)),[1,2])
 fp = tf.math.reduce_sum((y_hat -tf.multiply(y_hat,y)),[1,2])
 loss = loss - ((2 * tp) / tf.math.reduce_sum((2 * tp + fp + fn + 1e-10)))  # fscore

return loss

so my questions boil down to:

What does torch.finfo() do and how to express it in TensorFlow?
Does y_hat.dtype just return the data type?


title: Problem with adding smiles on photos with convolutional autoencoder

body:
I have a dataset with images and another dataset as it's description:

There are a lot of pictures: people with and without sunglasses, smiles and other attributes. What I want to do is be able to add smiles to photos where people are not smiling.
I've started like this:
smile_ids = attrs['Smiling'].sort_values(ascending=False).iloc[100:125].index.values
smile_data = data[smile_ids]

no_smile_ids = attrs['Smiling'].sort_values(ascending=True).head(5).index.values
no_smile_data = data[no_smile_ids]

eyeglasses_ids = attrs['Eyeglasses'].sort_values(ascending=False).head(25).index.values
eyeglasses_data = data[eyeglasses_ids]

sunglasses_ids = attrs['Sunglasses'].sort_values(ascending=False).head(5).index.values
sunglasses_data = data[sunglasses_ids]

When I print them their are fine:
plot_gallery(smile_data, IMAGE_H, IMAGE_W, n_row=5, n_col=5, with_title=True, titles=smile_ids)

Plot gallery looks like this:
def plot_gallery(images, h, w, n_row=3, n_col=6, with_title=False, titles=[]):
plt.figure(figsize=(1.5 * n_col, 1.7 * n_row))
plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)
for i in range(n_row * n_col):
    plt.subplot(n_row, n_col, i + 1)
    try:
        plt.imshow(images[i].reshape((h, w, 3)), cmap=plt.cm.gray, vmin=-1, vmax=1, interpolation='nearest')
        if with_title:
            plt.title(titles[i])
        plt.xticks(())
        plt.yticks(())
    except:
        pass

Then I do:
def to_latent(pic):
with torch.no_grad():
    inputs = torch.FloatTensor(pic.reshape(-1, 45*45*3))
    inputs = inputs.to('cpu')
    autoencoder.eval()
    output = autoencoder.encode(inputs)
    return output

def from_latent(vec):
with torch.no_grad():
    inputs = vec.to('cpu')
    autoencoder.eval()
    output = autoencoder.decode(inputs)
    return output

After that:
smile_latent = to_latent(smile_data).mean(axis=0)
no_smile_latent = to_latent(no_smile_data).mean(axis=0)
sunglasses_latent = to_latent(sunglasses_data).mean(axis=0)

smile_vec = smile_latent-no_smile_latent
sunglasses_vec = sunglasses_latent - smile_latent

And finally:
def add_smile(ids):
for id in ids:
    pic = data[id:id+1]
    latent_vec = to_latent(pic)
    latent_vec[0] += smile_vec
    pic_output = from_latent(latent_vec)
    pic_output = pic_output.view(-1,45,45,3).cpu()
    plot_gallery([pic,pic_output], IMAGE_H, IMAGE_W, n_row=1, n_col=2)

def add_sunglasses(ids):
for id in ids:
    pic = data[id:id+1]
    latent_vec = to_latent(pic)
    latent_vec[0] += sunglasses_vec
    pic_output = from_latent(latent_vec)
    pic_output = pic_output.view(-1,45,45,3).cpu()
    plot_gallery([pic,pic_output], IMAGE_H, IMAGE_W, n_row=1, n_col=2)

But when I execute this line I don't get any faces:
add_smile(no_smile_ids)

The output:

Could someone please explain where is my mistake or why it can happen? Thanks for any help.
Added: checking the shape of pic_output:


title: How to run inference of 2 deep learning models simultaneously on video?

body:
I want to get inference of 2 models.
First model(Runs at 20fps, Pytorch), Second one is a heavier model(Inference time 1 sec, Tensorflow) on webcam feed.
The first model would be running on every frame, The other model is not required on every frame, Something like 1 in every 50 frames.
I tried to use multiprocessing, But I am stuck on how to return outputs of function. The input to both the models is the same.
First model processes the frame and returns the processed frame, The second model processes and returns the string.
String needs to be displayed along with the processed frame, And it would be updated after every 50 frames.
I have written a pseudo code below, .start() function does not return the processed output, Need to replace that.
def first_model(frame):
    #Process frame here
    return processed_frame

def second_model(frame):
    #Process frame here
    return string_output

cap = cv2.VideoCapture(0)
i = 0
second_output = "Random text" #Output of second model is a string
while(True):
   _,frame = cv2.read()
   p = multiprocessing.pool(args=(frame,),target=first_model)
   first_output = p.start() #This is not correct
   if(i%50 == 0):
         q = multiprocessing.pool(args=(frame,),target=second_model)
         second_output = q.start() #Again, This is not allowed
   cv2.putText(first_output,second_output,region) #Put second output on every frame, on some predefined region
   cv2.imshow(first_output)
   i = i + 1


title: Pytorch and optimizing custom loss for polynomial regression

body:
I have I dataset on which I am trying to train a regression model. The pattern of the values in the dataset are more or less always the same and can be described from the following image:
Sample1

Sample 2

on the left is the raw values in the space and in the right after I've sorted them out (following a polynomial pattern). Now someone watching the raw valued graphs would consider that the sparse points on the top are outliers, however there are not and are quite important for the study that I want to apply. So they should be taken account. For regressing the values I've tried mainly 4 different loss functions i.e. MAE, MSE, RMSE and a custom one which is a combination of MAE and MSE on the low and high error values:
class CustomLoss(torch.nn.Module):
    def __init__(self):
        super(CustomLoss, self).__init__()

    def forward(self, predicted, true):
        errors = torch.abs(predicted - true)
        mask = errors > 250
        return torch.mean((0.5 * mask * (errors ** 2)) + ~mask * errors)

The output for each can be seen below:
MAE

MSE

RMSE

MAE+MSE

However, as you can see all of the fail to address the high frequency values. A similar output I am also getting for sample 2. Thus, I would like to ask whether there is a way to address such kind of value with a custom loss or something.
Thanks in advance.


title: How to minimise loss with constrain in PyTorch?

body:
How to put constrain while minimising loss?
I am trying to minimise the mse loss with constrain loss but constrain was increasing instead of decreasing.then i tried to only minimise constrain then it throw following error.
class Edge_Detector(nn.Module):
  def __init__(self,kernel_size,padding):
    torch.manual_seed(1)
    super(Edge_Detector,self).__init__()
    self.sobelx=nn.Conv2d(1,1,kernel_size=kernel_size,padding=padding,bias=False)
    self.relu=nn.ReLU()
    self.sobely=nn.Conv2d(1,1,kernel_size=kernel_size,padding=padding,bias=False)

  def forward(self,x):
    x1=self.sobelx(x)
    x2=self.sobely(x)
    x=self.relu(x1+x2)
    return x

  def loss(self,x,y):
    x=x.view(x.size(0),-1)
    y=y.view(y.size(0),-1).float()
    sobelx=self.sobelx.weight.data.squeeze().squeeze()
    sobely=self.sobely.weight.data.squeeze().squeeze()
    loss_mse=nn.MSELoss()(x,y)
    loss_constrain=torch.matmul(sobelx,sobely.transpose(0,1)).trace()
    #print('mse_loss :  ',loss_mse)
    #print('constrain_loss  : ',loss_constrain)
    #total_loss=loss_mse+loss_constrain
    return loss_constrain

#Error Message:
RuntimeError                              Traceback (most recent call last)
<ipython-input-67-28b5b5719682> in <module>()
----> 1 learn.fit_one_cycle(15, 5e-2)               #training for 4 epochs with lr=1e-3

13 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn


title: Load the dataset in Pytorch

body:
I cannot upload my dataset into Pytorch.
I am using the following code:
import pandas as pd
import torch
from torch.utils.data import Dataset
from skimage import io
import torchvision.transforms as transforms

class CatsAndDogsDataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform
    def __len__(self):
        return len(self.annotations) #24 images
    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotatoins.iloc[index, 0])
        image = io.imread(img_path)
        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

        if self.transform:
            image = self.transform(image)

        return (image, y_label)

dataset = CatsAndDogsDataset(csv_file = 'C:/***/Dataset/PetImages/Shuffled', root_dir = 'C:/***/Dataset/PetImages/Shuffled',
                     transform = transforms.ToTensor())

and I keep receiving the error in form of: PermissionError: [Errno 13] Permission denied: 'C:/***/Dataset/PetImages/Shuffled'
Here I have two questions:

When I have been trying to upload a similar dataset without defining the class and simply using cv2.imread function, there was no problems with permission. So why does this error pops up?

How can I upload the dataset using this method?

Thank you in advance!
Regards


title: Random erasing at multiple random regions of image

body:
I want to implement a custom random erasing function.
This function would take an input image and a percentage to mask, but would then mask between 1 and 4 random rectangles whose total area adds up to the mask percentage.
For example, say my image is 100100 pixels, and my mask percent is 15% so I randomly choose to create 3 rectangles with random shapes such that their combined area sums up to 100100*0.15 pixels.
so far i managed to write the code that decides upon the width and height and amount of rectangles, but i struggle with the part that makes sure they don't mask the same spot.
img_c, img_h, img_w = img.shape[-3], img.shape[-2], img.shape[-1]
        area = img_h * img_w

        for _ in range(10):
            block_num = torch.randint(1,4,(1,)).item()
            block_sizes = torch.rand((block_num))
            block_sizes = torch.round((block_sizes / block_sizes.sum()) * (area * mask_percent))
            h = torch.round((torch.rand(block_num)+0.5) * block_sizes.sqrt())
            w = torch.round(block_sizes / h)

            xs = []
            ys = []
            if not (any(h < img_h) and any(w < img_w)):
                continue
            term = True
            while term:
                xs = [torch.randint(0, img_h - h_ + 1, size=(1, )).item() for h_ in h]
                ys = [torch.randint(0, img_w - w_ + 1, size=(1, )).item() for w_ in w]
                for iter,x in enumerate(xs):
                    if (x+h[iter]-xs)<0

#here i get all confused. should have a loop that goes over each point and checks that the location + axial size
#doesn't go over another point. it's confusing because should i also take care  vice versa? maybe someone knows of a ready made solution?

            return i, j, h, w, v

        # Return original image
        return 0, 0, img_h, img_w, img

the while loop is released once the random location generator generates locations that corresppond to the terms.
edit_____________
my latest attempt seems to work, but always exits the loop unsolved! is it just not a very likely set of parameters?
img = torch.rand(1,160,1024)
img_c, img_h, img_w = img.shape[-3], img.shape[-2], img.shape[-1]
        area = img_h * img_w

        for _ in range(100):
            block_num = torch.randint(1,3,(1,)).item()
            block_sizes = torch.rand((block_num))
            block_sizes = torch.round((block_sizes / block_sizes.sum()) * (area * 0.15))
            h = torch.round((torch.rand(block_num)+0.5) * block_sizes.sqrt()).to(dtype=torch.long)
            w = torch.round(block_sizes / h).to(dtype=torch.long)

            if (h > img_h).any() or (w > img_w).any():
                continue
            overlap1 = torch.zeros(img.shape)
            xs = [torch.randint(0, img_h - h_.item() + 1, size=(1, )).item() for h_ in h]
            ys = [torch.randint(0, img_w - w_.item() + 1, size=(1, )).item() for w_ in w]
            for iter,(x,y) in enumerate(zip(xs,ys)):
                overlap1[0,x:x+h[iter],y:y+w[iter]] += 1
            if (overlap1>1).any():
                continue


title: MSELoss gives mismatching batch dimensions with batch size>1

body:
Overview of the relevant code:

def train_net(net,
              device,
              epochs=5,
              batch_size=10,
              lr=0.1,
              val_percent=0.1,
              save_cp=True,
              img_scale=0.5,
              n_channels=5,
              n_classes=1):

    dataset = BasicDataset(dir_img, dir_gt, img_scale, n_channels)
    n_val = int(len(dataset) * val_percent)
    n_train = len(dataset) - n_val
    train, val = random_split(dataset, [n_train, n_val])
    train_loader = DataLoader(train, batch_size=batch_size, shuffle=True, num_workers=8, pin_memory=True)
    val_loader = DataLoader(val, batch_size=batch_size, shuffle=False, num_workers=8, pin_memory=True)

    writer = SummaryWriter(comment=f'LR_{lr}_BS_{batch_size}_SCALE_{img_scale}')    #Add folder location
    global_step = 0

    optimizer = optim.Adam(net.parameters(), lr=lr)

    criterion = nn.MSELoss()

    for epoch in range(epochs):
        net.train()

        epoch_loss = 0
        with tqdm(total=n_train, desc=f'Epoch {epoch + 1}/{epochs}', unit='img') as pbar:
            for batch in train_loader:
                imgs = batch['image']
                true_masks = batch['mask']
                assert imgs.shape[1] == net.n_channels, \
                    f'Network has been defined with {net.n_channels} input channels, ' \
                    f'but loaded images have {imgs.shape[1]} channels. Please check that ' \
                    'the images are loaded correctly.'

                imgs = imgs.to(device=device, dtype=torch.float32)
                mask_type = torch.float32 if net.n_classes == 1 else torch.long
                true_masks = true_masks.to(device=device, dtype=mask_type)

                masks_pred = net(imgs)

                loss = criterion(masks_pred, true_masks)

                epoch_loss += loss.item()
                writer.add_scalar('Loss/train', loss.item(), global_step)

                pbar.set_postfix(**{'loss (batch)': loss.item()})

                optimizer.zero_grad()
                loss.backward()
                nn.utils.clip_grad_value_(net.parameters(), 0.1)
                optimizer.step()

                pbar.update(imgs.shape[0])
                global_step += 1

When I choose a batch size of 1 the loss function doesn't give a warning. When I change it to 10, it complains that:
:\workdir\Programs\Anaconda3\envs\XXXX\lib\site-packages\torch\nn\modules\loss.py:446: UserWarning: Using a target size **(torch.Size([1, 1, 89, 99]))** that is different to
the input size **(torch.Size([10, 1, 89, 99]))**. This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

                                                                                C:\workdir\Programs\Anaconda3\envs\XXXX\lib\site-packages\torch\nn\modules\loss.py:446: UserWarning: Using a target size **(torch.Size([1, 1, 89, 99]))** that is different to
the input size **(torch.Size([4, 1, 89, 99]))**. This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

However, when I debug, the shape of the input and target are both [10,1,89,99]. I think it's something about the training size and validation size...
Training size:   579
Validation size: 64

Does anybody know how to tackle this issue, so I can also work with larger batch sizes, larger than 1?


title: Why are embeddings necessary in Pytorch?

body:
I (think) I understand the basic principle behind embeddings: they're a shortcut to quickly perform the operation (one hot encoded vector) * matrix without actually performing that operation (by utilizing the fact that this operation is equivalent to indexing into the matrix) while still maintaining the same gradient as if that operation was actually performed.
I know in the following example:
e = Embedding(3, 2)
n = e(torch.LongTensor([0, 2]))

n will be a tensor of shape 2, 2.
But, we could also do:
p = nn.Parameter(torch.zeros([3, 2]).normal_(0, 0.01))
p[tensor([0, 2])]

and get the same result without an embedding.
This in and of itself wouldn't be confusing since in the first example n has a grad_fn called EmbeddingBackward whereas in the 2nd example p has a grad_fn called IndexBackward, which is what we expect since we know embeddings simulate a different derivative.
The confusing part is in chapter 8 of the fastbook they use embeddings to compute movie recommendations. But then, they do it without embeddings in basically the same manner & the model still works. I would expect the version without embeddings to fail because the derivative would be incorrect.
Version with embeddings:
class DotProductBias(Module):
    def __init__(self, n_users, n_movies, n_factors, y_range=(0,5.5)):
        self.user_factors = Embedding(n_users, n_factors)
        self.user_bias = Embedding(n_users, 1)
        self.movie_factors = Embedding(n_movies, n_factors)
        self.movie_bias = Embedding(n_movies, 1)
        self.y_range = y_range

    def forward(self, x):
        users = self.user_factors(x[:,0])
        movies = self.movie_factors(x[:,1])
        res = (users * movies).sum(dim=1, keepdim=True)
        res += self.user_bias(x[:,0]) + self.movie_bias(x[:,1])
        return sigmoid_range(res, *self.y_range)

Version without:
def create_params(size):
    return nn.Parameter(torch.zeros(*size).normal_(0, 0.01))

class DotProductBias(Module):
    def __init__(self, n_users, n_movies, n_factors, y_range=(0,5.5)):
        self.user_factors = create_params([n_users, n_factors])
        self.user_bias = create_params([n_users])
        self.movie_factors = create_params([n_movies, n_factors])
        self.movie_bias = create_params([n_movies])
        self.y_range = y_range

    def forward(self, x):
        users = self.user_factors[x[:,0]]
        movies = self.movie_factors[x[:,1]]
        res = (users*movies).sum(dim=1)
        res += self.user_bias[x[:,0]] + self.movie_bias[x[:,1]]
        return sigmoid_range(res, *self.y_range)

Does anyone know what is going on?


title: how to download dataset in colab using torchvision iam getting this error

body:
dataset = MNIST(root='data/', download=True)
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-2-9432298eb00b> in <module>()
----> 1 dataset = MNIST(root='data/', download=True, transform=ToTensor())

11 frames
/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden


title: Error while executing CrossEntropyLoss() in PyTorch

body:
My dataset contains images of shape [3,28,28]. I have written the following code:
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(nn.Conv2d(3, 28, kernel_size=5, stride=1, padding=2),nn.ReLU(),nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(nn.Conv2d(28, 56, kernel_size=5, stride=1, padding=2),nn.ReLU(),nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(7 * 7 * 56, 1000)
        self.fc2 = nn.Linear(1000, 10)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

model = ConvNet()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
total_step = len(loader_train)

for e in range(num_epochs):
    print("Epoch ", e+1,": ")
    for i, (images, labels) in enumerate(loader_train):
        optimizer.zero_grad()
        actual_out = model(images)
        loss = criterion(actual_out, labels)
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.3f}' .format(e+1, num_epochs, i+1, total_step, loss.item()))

However, I'm getting the following error:
AttributeError                            Traceback (most recent call last)
 in
8         actual_out = model(images)
9
---> 10         loss = criterion(actual_out, labels)
11         loss.backward()
AttributeError: 'tuple' object has no attribute 'size'
I converted labels into a tensor by the following method:
target_out = torch.empty(batch_size,dtype=torch.long).random_(labels)
loss = criterion(actual_out, target_out)

But that generates:
TypeError                                 Traceback (most recent call last)
 in
---> 11         target_out = torch.empty(batch_size,dtype=torch.long).random_(labels)
12         loss = criterion(actual_out, target_out)
TypeError: random_() received an invalid combination of arguments - got (tuple), but expected one of:

(*, torch.Generator generator)
(int from, int to, *, torch.Generator generator)
(int to, *, torch.Generator generator)


title: How to get confusion matrix in pytorch?

body:
There are 15 classes. I have two tensors, one has the target classes and other has the predicted classes.
How can I get a confusion matrix?


title: Appling sliding window to torch.tensor and adjusting tensor initial size

body:
Looking for a simpler way of torch.tensor modification. Probably there is a way to apply Unfold to initial tensor directly.
input:
tensor([[0., 1., 2.],
        [3., 4., 5.],
        [6., 7., 8.]])

output:
tensor([[0., 1., 3., 4.],
        [1., 2., 4., 5.],
        [3., 4., 6., 7.],
        [4., 5., 7., 8.]])

possible solution:
import torch

t = torch.linspace(0., 8., steps=9)

t1 = t.reshape(3,3) # starting point

t2 = torch.flatten(t1)

t3 = t2.reshape(1, 1, 1, -1) # unfold works with 4D only

unfold = torch.nn.Unfold(kernel_size=(1, 5), dilation=1)

t4 = unfold(t3)

indices = torch.tensor([0, 1, 3, 4]) # deleting 3d (or middle) row and 3d (middle) column

t5 = torch.index_select(torch.index_select(t4.squeeze(), 0, indices), 1, indices)

t5


title: Image classification model works with 32x32 images but not 64x64

body:
I'm trying to build a deep learning food classification algorithm using the food 101 dataset. I was able to successfully implement it by using the following model which works when all my images are sized with dimensions 32x32. However, I realised some of the images were almost incomprehensible so I increased the size to 64x64 for all images. However, when I run my code with these larger image sizes it no longer works.
I believe the error is to do with how I've defined the model. I'm new to the area of deep learning and would appreciate any help. If you need any further info pls comment below without taking down the post.
Model definition (uses convolutional layers and a residual block):
class SimpleResidualBlock(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)
        self.relu1 = nn.ReLU()
        self.conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)
        self.relu2 = nn.ReLU()

    def forward(self, x):
        out = self.conv1(x)
        out = self.relu1(out)
        out = self.conv2(out)
        return self.relu2(out) + x # ReLU can be applied before or after adding the input

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))

class ImageClassificationBase(nn.Module):
    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], last_lr: {:.5f}, train_loss: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}".format(
            epoch, result['lrs'][-1], result['train_loss'], result['val_loss'], result['val_acc']))

def conv_block(in_channels, out_channels, pool=False):
    layers = [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
              nn.BatchNorm2d(out_channels),
              nn.ReLU(inplace=True)]
    if pool: layers.append(nn.MaxPool2d(2))
    return nn.Sequential(*layers)

class ResNet9(ImageClassificationBase):
    def __init__(self, in_channels, num_classes):
        super().__init__()

        self.conv1 = conv_block(in_channels, 64)
        self.conv2 = conv_block(64, 128, pool=True)
        self.res1 = nn.Sequential(conv_block(128, 128), conv_block(128, 128))

        self.conv3 = conv_block(128, 256, pool=True)
        self.conv4 = conv_block(256, 512, pool=True)
        self.res2 = nn.Sequential(conv_block(512, 512), conv_block(512, 512))

        self.classifier = nn.Sequential(nn.MaxPool2d(4),
                                        nn.Flatten(),
                                        nn.Dropout(0.2),
                                        nn.Linear(512, num_classes))

    def forward(self, xb):
        out = self.conv1(xb)
        out = self.conv2(out)
        out = self.res1(out) + out
        out = self.conv3(out)
        out = self.conv4(out)
        out = self.res2(out) + out
        out = self.classifier(out)
        return out

Error I get when executing it:
RuntimeError: mat1 dim 1 must match mat2 dim 0


title: PyTorch: Dynamic Programming as Tensor Operation

body:
Is it possible to get the following loop with a Tensor operation?
a = torch.Tensor([1, 0, 0, 0])
b = torch.Tensor([1, 2, 3, 4])

for i in range(1, a.shape[0]):
    a[i] = b[i] + a[i-1]

print(a) # [1, 3, 6, 10]

The operation depends on the previous values in a and the values that are computed on the way (in a dynamic programming fashion).
Is it possible to get this type of sequential computation with a tensor operation?


title: "The NVIDIA driver on your system is too old (found version 9000).", but I do actaully have installed version 10.1

body:
I was trying to check CUDA information on my Ubuntu device and I run torch.cuda.current_device(), and it errs:
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.current_device()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 377, in current_device
    _lazy_init()
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 196, in _lazy_init
    _check_driver()
  File "/usr/local/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 110, in _check_driver
    of the CUDA driver.""".format(str(torch._C._cuda_getDriverVersion())))
AssertionError:
The NVIDIA driver on your system is too old (found version 9000).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
>>>

But I do have installed a higher version than 9.x:
>> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105

I wonder why this would happen and how I can solve this.


title: PyTorch indexing: select complement of indices

body:
Say I have a tensor and index:
x = torch.tensor([1,2,3,4,5])
idx = torch.tensor([0,2,4])

If I want to select all elements not in the index, I can manually define a Boolean mask like so:
mask = torch.ones_like(x)
mask[idx] = 0

x[mask]

is there a more elegant way of doing this?
i.e. a syntax where I can directly pass the indices as opposed to creating a mask e.g. something like:
x[~idx]


title: numpy equivalent code of torch tensor method

body:
I have these 2 tensors
box_a = torch.randn(1,4)
box_b = torch.randn(1,4)

and i have a code in pytorch
box_a[:, 2:].unsqueeze(1).expand(1, 1, 2)

but i want to convert the above code in numpy
for box_a and box_b i can do something like this
  box_a = numpy.random.randn(1,4)
  box_b = numpy.random.randn(1,4)

But what about this
box_a[:, 2:].unsqueeze(1).expand(1, 1, 2)


title: How to convert a pytorch Tensor to a numpy array without using detach()?

body:
I am trying to implement an algorithm and I am facing an error in the following block of code :
def get_action(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0)
        probs = self.policy_network.forward(Variable(state))
        highest_prob_action = np.random.choice(self.num_actions, p=np.squeeze(probs.numpy())) //This line
        log_prob = torch.log(probs.squeeze(0)[highest_prob_action])
        return highest_prob_action, log_prob

To pass in the probability distribution in np.random.choice, I used the probs tensor. I know I can use the detach function to make it work but I can't use that function because I have to submit this code on an autograder and apparently it returns an error when you use that function.
Is there a way I can do this without using the detach function?
Here's how the probs tensor looks like :
tensor([[0.2522, 0.2533, 0.2385, 0.2560]], grad_fn=<SoftmaxBackward>)

And I want the final output to be of the type (The values don't mean anything):
array([0., 1., 2., 3.], dtype=float32)

To reproduce the error, you can use:
import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.numpy()

print(tensor1)
print(type(tensor1))

What I tried : As suggested by GoodDeeds in the comments, I tried to use torch.multinomial as follows :
states.append(state)
            probs = self.policy.forward(Variable(torch.from_numpy(state).float().unsqueeze(0)))
            highest_prob_action = np.random.choice(torch.multinomial(probs, self.num_actions).squeeze(0))
#             highest_prob_action = np.random.choice(self.num_actions, p=np.squeeze(probs.numpy()))
            log_prob = torch.log(probs.squeeze(0)[highest_prob_action])

but after some iterations it returns the error:
    130             states.append(state)
    131             probs = self.policy.forward(Variable(torch.from_numpy(state).float().unsqueeze(0)))
--> 132             highest_prob_action = np.random.choice(torch.multinomial(probs, self.num_actions).squeeze(0))
    133 #             highest_prob_action = np.random.choice(self.num_actions, p=np.squeeze(probs.numpy()))
    134             log_prob = torch.log(probs.squeeze(0)[highest_prob_action])

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0


title: what does pytorch do for creating tensor from numpy

body:
I am interested in what torch have done when I call torch.from_numpy. As the name indicates, it seems that PyTorch creates a Tensor instance and allocates the memory for copying the content from numpy ndarray to itself. But how does PyTorch do the memcpy work and what else has PyTorch done in the background? It seems the implementation of tensor is in autograd. But I have no idea which part should I look for.
I have the question because I found it is really fast for constructing a tensor from numpy. And it is even fast than creating a tensor directly
a = np.random.randn(100,100)
%timeit torch.from_numpy(a)
759 ns ± 7.53 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
%timeit torch.randn(100,100)
61 µs ± 2.46 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit torch.zeros(100,100)
3.1 µs ± 136 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)


title: Calculate `dot` product of image and vector

body:
I would like to calculate something like dot product of vector and image with shapes:

(3)
(3,1080,1080)

and the output should be (1,1080,1080)
img = torch.rand((3,1080,1080))
R_coefficient=float(0.2126)
G_coefficient=float(0.7152)
B_coefficient=float(0.0722)
Y = img[0,...]*R_coefficient + img[1,...]*G_coefficient img[2,...]*B_coefficient

This code above gives me results, I look for, but I would like to use PyTorch utils like torch.dot, torch.matmul etc.
I have tried these
TensorImage(torch.dot(img[:,...],RGB_vector[:]))
TensorImage(torch.matmul(img,RGB_vector[:]))

These two options give me an errors connecting with shape, I have rejected them.
RGB_vector = torch.tensor([[[0.2126, 0.7152, 0.0722]]], dtype=img.dtype).permute(2,1,0)
    #     torch.t(RGB_vector)
        print(RGB_vector.shape)
        print(img.shape)
        return TensorImage(img[:,...]*RGB_vector)#.unsqueeze_(0)

this sample above works, but I am getting image with shape (3,1080,1080), but I need to get 1 instead of 3 at the first shape's dimension
Current working example
import torch
img = torch.rand((3,1080,1080))
RGB_vector = torch.tensor([[[0.2126, 0.7152, 0.0722]]], dtype=img.dtype).permute(2,1,0)
print(RGB_vector.shape)
print(img.shape)
TensorImage(img[:,...]*RGB_vector).unsqueeze_(0)

Greetings,
DA


title: Question about terminology used in LSTM inputs - seq_length vs context_size for sliding window approach

body:
I have time series of a sequence of n vectors that I need to feed to a LSTM with a sliding window approach.

In different resources that I read online, seq_length is often referred to as the window length [or the number of LSTM cells] and context_size is defined as the input size to LSTM at a timestep. (for eg. one input vector at a time step).

At any given point of time t, I want to pass points x_{t-m}, …, x_{t-1},…,x_{t} to LSTM followed by a dense layer and predict a categorical target attribute at every tilmestep.

If I want to follow the sliding window approach, do I need to explicitly split input data into windows of size m ?

For instance:

[x_{t-m}, …, x_{t-1},…,x_{t}], [x_{t-m+1}, …, x_{t},…,x_{t+1}], [x_{t-m+2}, …, x_{t+1}, x_{t+2}] etc.

Or, can I split the input data into non-overlapping chunks [x_{t-m}, …, x_{t-1},…,x_{t}], [x_{t+1}, …, x_{t+m-1},x_{t+m}] etc. and instead resize the context_size ?

     embeds = embeds.unfold(1, context_size, 1)  # Keeping the step size to be 1
     embeds = embeds.view(embeds.size(0), embeds.size(1), -1)

Is there a better way to implement sliding window approach for timeseries data ?


title: Curious on how to use some basic machine learning in a web application

body:
A co-worker and I had an idea to create a little web game where a user enters a chunk of data about themselves and then the application would write for them to sound like them in certain structures. (Trying to leave the idea a little vague.) We are both new to ML and thought this could be a fun first dive.

We have a decent bit of background with PHP, JavaScript (FE and Node), Ruby a little bit of other languages, and have had interest in learning Python for ML. Curious if you can run a cost efficient ML library for text well with a web app, being most servers lack GPUs?

Perhaps you have to pay for one of the cloud based systems, but wanted to find the best entry point for this idea without racking up too much cost. (So far I have been reading about running Pytorch or TensorFlow, but it sounds like you lose a lot of efficiency running with CPUs.)

Thank you!

(My other thought is doing it via an iOS app and trying Apple's ML setup.)


title: Print function does not print anything while working to build CNN Architecture

body:
I am learning image classification with CNN in Pytorch. While I started to build the CNN architecture I am getting 2 problem:

When I run my code I get the warning -warnings.warn("train_labels has been renamed targets")
I tried to print loss and accuracy using print function but it does not print anything.

Here is my code:

 def forward(self, x):
     out = self.layer1(x)
     out = self.layer2(out)
     out = out.reshape(out.size(0), -1)
     out = self.drop_out(out)
     out = self.fc1(out)
     out = self.fc2(out)
     return out
     from torch import optim
     model = ConvNet()
     criterion = nn.CrossEntropyLoss()
     optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
     total_step = len(train_loader)
     loss_list = []
     acc_list = []
     for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):

          outputs = model(images)
          loss = criterion(outputs, labels)
          loss_list.append(loss.item())
          optimizer.zero_grad()
          loss.backward()
          optimizer.step()

          total = labels.size(0)
          _, predicted = torch.max(outputs.data, 1)
          correct = (predicted == labels).sum().item()
          acc_list.append(correct / total)
          if (i + 1) % 100 == 0:
           print("Epoch [{}\{}], Step [{}\{}], Loss: {:.4f}, Accuracy: {:.2f}%"
                    .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                            correct / total) * 100)

        # Test the model
          model.eval()
          with torch.no_grad():
            correct = 0
            total = 0
            for images, labels in test_loader:
                outputs = model(images)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

            print("Test Accuracy of the model on the 10000 test images: {} %".format((correct / total) * 100))


title: How to apply operations after uniform_ form torch in Python?

body:
I have

x = torch.ones(n,3)
x[:,0].uniform_(-1.,1)
x[:5]

And I got

tensor([[-0.4899,  1.0000,  1.0000],
        [ 0.7395,  1.0000,  1.0000],
        [-0.3599,  1.0000,  1.0000],
        [ 0.0863,  1.0000,  1.0000],
        [ 0.8935,  1.0000,  1.0000]])

What I want to achieve is to assign random value to first and second row
and to make the first row squared

[0.25,0.5,1] (an example)

I have few questions:

how to apply the same uniform_ to only two rows?
Is there a square function, and how to apply it to only first row?
Can it be done in one operation?


title: Can I set the predicted values of a neural network?

body:
My question is more theoretical than practical but I can show some code as well. I have network that is mapping from random values in a domain uvw, to a xyz domain. I want a certain values of the uvw to go to other certain values in the xyz that I already know, since that is the idea of the function that I want to obtain and I want the network to over-fit.

My question is divided into two questions:

Can I set the predicted values I want into the network so it doesn't have to calculate those predicted values?
Does this is going to affect prediction of the other values?

This is my code, I want to show it so we have some notation that we can talk about.

 # The model is a simple fully connected network mapping a 3D parameter point to 3D
    phi = common.MLP(in_dim=3, out_dim=3).to(args.device)

    # Eps is 1/lambda and max_iters is the maximum number of Sinkhorn iterations to do
    emd_loss_fun = SinkhornLoss(eps=args.sinkhorn_eps, max_iters=args.max_sinkhorn_iters,
                                stop_thresh=1e-3, return_transport_matrix=True) # TODO add r-1 function to the weights

    mse_loss_fun = torch.nn.MSELoss()

    # Adam optimizer at first
    optimizer = torch.optim.Rprop(phi.parameters(), lr=args.learning_rate)

    fit_start_time = time.time()

    for epoch in range(args.num_epochs):
        optimizer.zero_grad()

        # Do the forward pass of the neural net, evaluating the function at the parametric points
        y = phi(t)

        # Compute the Sinkhorn divergence between the reconstruction*(using the francis library) and the target
        # NOTE: The Sinkhorn function expects a batch of b point sets (i.e. tensors of shape [b, n, 3])
        # since we only have 1, we unsqueeze so x and y have dimension [1, n, 3]
        with torch.no_grad():
            _, M = emd_loss_fun(phi(t[num:]).unsqueeze(0), x[num:].unsqueeze(0))
            _, Q = emd_loss_fun(phi(t[0:num]).unsqueeze(0), x[0:num].unsqueeze(0))
            P[0,num:,num:] = M[0]
            P[0,0:num,0:num] = Q[0]
            #print(y[Q.squeeze().max(0)[1], :])

        # Project the transport matrix onto the space of permutation matrices and compute the L-2 loss
        # between the permuted points
        loss =  mse_loss_fun(y[P.squeeze().max(0)[1], :], x)
        # loss = mse_loss_fun(P.squeeze() @ y,  x)  # Use the transport matrix directly

        # Take an optimizer step
        loss.backward()
        optimizer.step()

        print("Epoch %d, loss = %f" % (epoch, loss.item()))


title: What's the difference between the two methods of using sub-model in pytorch?

body:
method 1:

class subnet1():
    def __init__()

class subnet2():
    def __init__()

class father_net():
    def __init__():
        self.n1 = subnet1()
        self.n2 = subnet2()
    def forward():
        x = self.n1()
        x = self.n2(x)

method 2:

class father_net():
    def subnet1():
    def subnet2():
    def forward():
        x = self.subnet1()
        x = self.subnet2()

It seems nothing different when I train father_net.

However I still want to get some detail explanations. Any idea?


title: Output of a neural network

body:
I test my network with the mnist dataset. Therefore the output of the model has shape 10.

How do I reshape the output?
For example, if the output is label 3, is then the output [0 0 0 1 0 0 0 0 0 0], or [0 0 0 3 0 0 0 0 0 0] or completely different?

The thing is I don't want to use the dataloader. I use this method:

from mlxtend.data import loadlocal_mnist
X, y = loadlocal_mnist(
            images_path='/home/wai043/data/mnist/train-images-idx3-ubyte',
            labels_path='/home/wai043/data/mnist/train-labels-idx1-ubyte')


title: converting 1 x 1 x 33 tensor to 1 x 1 x 34 while maintaining values and leaving the 34th index 0

body: I have a tensor of 0 and and one 1 in a 1 by 1 by 33 tensor and I want all the indexes that are 1 to stay one, but be in a 1 by 1 by 34 tensor. What's the standard practice for this? Apparently using "reshape is bad" for this.


title: challenging special numpy operation

body:
I have a NumPy array that is full of indices of numbers input. I want to check if certain indices indices are in it. Say that the i'th row of input, input[i] has entries j_1<...<j_n that their values belong to indices. I would like to switch the value of input[i,j_n] with a random value from indices. How can I do it elegantly and efficiently?

Example:

input = [[1,2,3],[4,5,6],[7,8,9]]

indices = [2,4,6]

input --> [[1,4,3],[4,5,4],[7,8,9]]


title: how target[y == l] = label work is labeling make_blobs dataset in pytorch tutorial?

body:
I am learning PyTorch with this blog: [PyTorch Introduction to Neural Network] (https://medium.com/biaslyai/pytorch-introduction-to-neural-network-feedforward-neural-network-model-e7231cff47cb)
 I found this piece of code confusing:

from sklearn.datasets import make_blobs
def blob_label(y, label, loc): # assign labels
    target = numpy.copy(y)
    for l in loc:
        target[y == l] = label
    return target
x_train, y_train = make_blobs(n_samples=40, n_features=2, cluster_std=1.5, shuffle=True)
x_train = torch.FloatTensor(x_train)
y_train = torch.FloatTensor(blob_label(y_train, 0, [0]))
y_train = torch.FloatTensor(blob_label(y_train, 1, [1,2,3]))
x_test, y_test = make_blobs(n_samples=10, n_features=2, cluster_std=1.5, shuffle=True)
x_test = torch.FloatTensor(x_test)
y_test = torch.FloatTensor(blob_label(y_test, 0, [0]))
y_test = torch.FloatTensor(blob_label(y_test, 1, [1,2,3]))

how exactly is target[y == l] = label working in here?
how this line of code assigns 1s and 0s to the data?
Thanks!


title: PyTorch transferring index to 1-0

body:
How to quickly set the elements in the given index list to 1 and others to 0?

For example,I have an ID pool like:
torch.arange(10),
for a given input index tensor([1,5,7,9,2]) wanna return tensor([0,1,1,0,0,1,0,1,0,1])


title: Wrong Number of Init Arguments for Tanh in Pytorch

body:
For a homework assignment, I am implementing a simple neural network in Python using Pytorch. Here is my network class:

class Net(torch.nn.Module):
    def __init__(self, layer_dims, activation="sigmoid"):
        super(Net, self).__init__()
        layers = []
        if activation == 'sigmoid':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.Sigmoid(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.Sigmoid(layer_dims[i - 1]))
            layers.append(nn.Sigmoid(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.Sigmoid())
        elif activation == 'relu':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.ReLu(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.ReLU(layer_dims[i - 1]))
            layers.append(nn.ReLu(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.ReLu())
        elif activation == 'tanh':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.Tanh(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.Tanh(layer_dims[i - 1]))
            layers.append(nn.Tanh(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.Tanh())
        elif activation == 'identity':
            for i in range(1, len(layer_dims) - 1):
                layers.append(nn.Identity(layer_dims[i - 1], layer_dims[i]))
                layers.append(nn.Identity(layer_dims[i - 1]))
            layers.append(nn.Identity(layer_dims[-2], layer_dims[-1]))
            layers.append(nn.Identity())

        self.out = nn.Sequential(*layers)

    def forward(self, input):
        return self.out(input)

def train(data, labels, n, l, activation='sigmoid'):
    if activation not in ['sigmoid','identity','tanh','relu']:
        return
    net = Net([l for i in range(0,n)], activation)
    optim = torch.optim.Adam(net.parameters())
    for i in range(0,5):
        ypred = net.forward(torch.Tensor(data))
        ypred.backward()
        optim.step()
        optim.zero_grad()
    ypred = net.forward(torch.Tensor(data))
    return (net, torch.nn.CrossEntropyLoss(ypred, labels), net.parameters(), ypred)

When testing this, I have been trying to run the following code segment:

for i in range(3,5):
    for num in [10,30,50]:
        print(train(data.get('X_trn'), data.get('y_trn'), i, num, activation='tanh'))

Which is erroring out with a TypeError, saying that init() takes 1 positional argument when 3 is given.

<ipython-input-30-376b6c739a71> in __init__(self, layer_dims, activation)
     18         elif activation == 'tanh':
     19             for i in range(1, len(layer_dims) - 1):
---> 20                 layers.append(nn.Tanh(layer_dims[i - 1], layer_dims[i]))
     21                 layers.append(nn.Tanh(layer_dims[i - 1]))
     22             layers.append(nn.Tanh(layer_dims[-2], layer_dims[-1]))

TypeError: __init__() takes 1 positional argument but 3 were given

This error has appeared when I switch the activation function as well. I am unsure what the issue is, because as far as I know, when you create a layer you need to give the input and output dimensions, which is what I have. Any help sorting this out would be appreciated.


title: How do i add a dimension to a numpy array and copy the dimension from another numpy array

body:
I have a numpy array with the shape (128, 8)
I want to add an extra dimension so it has the shape (128, 168, 8)
And add the content of a 168 dimension from another array that has the shape (128, 168, 8).
I can always permute the positions of the dimensions if I can somehow add it.
Is this possible somehow? I have seen the append and concatenation methods but to no luck.


title: constraint of weights summation to be constant for every row

body: I want to train a net that has weights that sum to a constant for a specific axis. Is there a way to impose such a condition while training using Keras or Pytorch? Any pointer is appreciated.


title: how to set time_step in LSTM in pytorch

body:
I'm dealing with string classification, and I want to use LSTM, but I have some questions about setting time_step. My data set looks like:

Data ----- Label

DKWL----0
FCHN----0
KDQP----0
IHGS----1
....

Then I convert it into binary code:

00011101000001000111-----1
.....
shape:(N,20)

Now when I try to train LSTM, following is how I build LSTM:

class RNN(nn.Module):
    def __init__(self):
        super(RNN, self).__init__()

        self.rnn = nn.LSTM(
            input_size=20,     # Each sample has 20 features
            hidden_size=64,
            num_layers=1,
            batch_first=True,
        )

        self.out = nn.Linear(64, 2) # I set output 2, because there are two classes 0 & 1

    def forward(self, x):
        r_out, (h_n, h_c) = self.rnn(x, None)
        out = self.out(r_out[:, -1, :])
        return out

Then I try to train it with following codes:

for epoch in range(EPOCH):
    for step, (x, b_y) in enumerate(train_loader):   # gives batch data
        b_x = x.view(-1, ?, 20)   # at this step I need to reshape x to
                                  # (batch, time_step, input_size), but how I set this
                                  # time_step? 0 or 1?
        output = rnn(b_x)
        loss = loss_func(output, b_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()


title: Pytorch NN Training issue: Loss of NN does not decrase

body:
I want to classify random Instagram images as "image has a dog" or "image has not a dog".
To train my NN to classify dogs I want to use the Stanford Dogs Dataset, so I have about 20.000 training images of different dogs with different breeds.

But while training my NN the loss does not decrease, I checked that with different learning rates and with or without dropout layers.

Can anyone give tips or does anyone see bugs in the following code?:

import torch
import torchvision
from torchvision import transforms
from PIL import Image
from os import listdir
import os
import random
import torch.optim as optim
from torch.autograd import Variable
import torch.nn.functional as F
import torch.nn as nn

TRAINDATAPATH = 'C:/Users/.../Desktop/train/'
TESTDATAPATH = 'C:/Users/.../Desktop/#apfel/'

"""normalize = transforms.Normalize(
   mean=[0.485, 0.456, 0.406],
   std=[0.229, 0.224, 0.225]
)"""
normalize = transforms.Normalize(
   mean=[0.5, 0.5, 0.5],
   std=[0.5, 0.5, 0.5]
)
transforms = transforms.Compose([transforms.Resize(256),
                                 transforms.CenterCrop(256),
                                 transforms.ToTensor(),
                                 normalize])

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
train_data_list = []
target_list = []
train_data = []

batch_size = 1

files = listdir(TRAINDATAPATH)

for i in range(len(listdir(TRAINDATAPATH))):
    try:
        f = random.choice(files)
        files.remove(f)
        img = Image.open(TRAINDATAPATH + f)
        img_tensor = transforms(img) # (3,256,256)
        train_data_list.append(img_tensor)
        isObj = 1 if 'obj' in f else 0
        isNotObj = 0 if 'obj' in f else 1
        target = [isObj, isNotObj]

        target_list.append(target)
        if len(train_data_list) >= 1:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []
            target_list = []
            print('Loaded batch ', int(len(train_data)/batch_size), 'of ', int(len(listdir(TRAINDATAPATH))/batch_size))
            print('Percentage Done: ', 100*int(len(train_data)/batch_size)/int(len(listdir(TRAINDATAPATH))/batch_size), '%')
    except Exception:
        print("Error occured but ignored")
        print(str(Exception))
        continue

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 12, kernel_size=5)
        self.conv3 = nn.Conv2d(12, 18, kernel_size=5)
        self.conv4 = nn.Conv2d(18, 24, kernel_size=5)
        self.fc1 = nn.Linear(3456, 1000)
        self.fc2 = nn.Linear(1000, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = self.conv3(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = self.conv4(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = x.view(-1,3456)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)

        return torch.sigmoid(x)

model = Netz()
model.to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))

optimizer = optim.Adadelta(model.parameters(), lr=10)
def train(epoch):
    global model

    model.train()
    batch_idx = 0
    for data, target in train_data:
        batch_idx += 1
        data = data.to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))
        target = torch.Tensor(target).to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()

        output = model(data)

        criterion = F.binary_cross_entropy
        loss = criterion(output, target)
        loss.backward()

        optimizer.step()
        print('Train Epoch: '+ str(epoch) + '\tLoss: ' + str(loss.data.item()) )

def test():
    global model

    model.eval()
    files = listdir(TESTDATAPATH)
    f = random.choice(files)
    img = Image.open(TESTDATAPATH + f)
    img_eval_tensor = transforms(img)
    img_eval_tensor.unsqueeze_(0)
    data = Variable(img_eval_tensor.to(torch.device("cuda" if torch.cuda.is_available() else "cpu")) )
    out = model(data)
    string_prediction = str(out.data.max(0, keepdim=True)[1])
    print(string_prediction[9:10])

for epoch in range(1,4):
    train(epoch)
i = 100
while i > 0:
    test()
    i -= 1

In the TRAINDATAPATH are thousands of Dog images with the filename "obj_XXX.jpg" and some other images WITHOUT dogs with other filenames don't including "obj".

In the TESTDATAPATH are just random images, some with dogs, some without.
The NN classifies them all as "not including dogs" or "0" which is incorrect.

Thanks for every help!


title: Pytorch with bulk data

body:
My code is as

def add_outlinks(arr, source):
    for dest in arr:
        if int(dest) in _local_dict:
            _local_dict[dest].in_links.append(int(source))

    rpc.init_rpc(my_name, rank=rank, world_size=size,rpc_backend_options=rpc.ProcessGroupRpcBackendOptions(num_send_recv_threads=16,rpc_timeout=datetime.timedelta(seconds=10000)))  # initial_rpc

    #CALL rpc TO OTHER RANKS
    if rank==0:
        print("add-link...")
    try:
        array_rpc = list(range(0, size))
        count=0
        for it in _local_dict:
            count = count+1
            arr_send = []
            for i in range(0, size):
                arr_send.append([])
            u = _local_dict[it]
            source = u.vertexId

            for i in u.links:
                arr_send[int(i) % size].append(int(i))
            for i in array_rpc:
                my_target = "worker" + str(i)
                if len(arr_send[i])>0:
                    rpc.rpc_async(my_target, add_outlinks, args=(arr_send[i],source))
    except:
       print("rank ",rank," run ",count,"/",len(_local_dict))
    rpc.api._wait_all_workers()
    print("shutdown.... rpc... ", rank)
    rpc.api._wait_all_workers()
    rpc.shutdown()

arr_send[i] will send to rank i
For elements in _local_dict, we can run parallel.

1.     After 10000s, how many objects are processed?

--> the outputs are as below. The worker0 is not see in the output. I try to print "count" value. But there is no output for "count" variable.

....
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:45462
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:44970
....

[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:19635
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:27553
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:44970
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:47501: Connection reset by peer
Traceback (most recent call last):
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "pagerank.py", line 380, in init_process
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:44931: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
...
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:378] writev [2001:700:4a01:10::38]:22942: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 240, in shutdown
    _wait_all_workers()
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:5848: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:50095: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:29331
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:57022: Connection reset by peer
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 165, in _wait_all_workers
    args=(sequence_id, self_worker_name,),
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:15236
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:2720
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:23214
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:38547
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:50607
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
...
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:12173
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 554, in rpc_sync
    return fut.wait()
RuntimeError: Encountered exception in ProcessGroupAgent::enqueueSend: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:57022: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:3715
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:2693
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure

During handling of the above exception, another exception occurred:

[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:9877
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:3715
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
Traceback (most recent call last):
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:378] writev [2001:700:4a01:10::38]:22942: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:36780: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:47501: Connection reset by peer
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:44931: Connection reset by peer

[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:3213: Connection reset by peer
Traceback (most recent call last):
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:13723
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "pagerank.py", line 380, in init_process
    print("shutdown.... rpc... ", rank)
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:9140
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 240, in shutdown
    _wait_all_workers()
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 165, in _wait_all_workers
    args=(sequence_id, self_worker_name,),
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/site-packages/torch/distributed/rpc/api.py", line 554, in rpc_sync
    return fut.wait()

RuntimeError: Encountered exception in ProcessGroupAgent::enqueueSend: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [2001:700:4a01:10::38]:9889: Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
  File "/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
KeyboardInterrupt
[E thread_pool.cpp:112] Exception in thread pool task: Application timeout caused pair closure
[E thread_pool.cpp:112] Exception in thread pool task: [/opt/conda/conda-bld/pytorch_1587428228634/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [2001:700:4a01:10::38]:34716

I changed the code to print count as

for it in _local_dict:
     if(rank==0:)
            count = count+1
           print(count)

--> all elements in _local_dict is run. However, the program stopped by timeout of RPC. This means that all rpc is called and waiting to finish.

2.     Is distributed autograd/optimizer also used in this case?

--> Not yet. It is at https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html ??

3.     Are any of those CPUs locate on the same machine? (so that shm can be helpful)
-->  I use 32 CPUs on this machine. (All of my CPU on the machine.)

I try with small data (~60 elements in _local_dict), and it worked.
Perhaps, there are problems with a lot of rpc call?

For real data, size of _local_dict is nearly 190000 elements - 32 workers. With each element, I called 32 times for rpc(…).

For each worker, we have to call 190000*32 times for rpc(…). We have 32 workers. Hence, totally, there are 190000*32*32 times for calling RPC.

Is there any problem with rpc-pytorch or dictionary datatype in python?

Thanks,


title: How do I write a "predict" function with python/pytorch for an image classifier

body:
I'm writing a "predict" function with python/pytorch for an image classifier, and I'm not sure how to resolve this error:
RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]
Here's a link to the entire .ipynb notebook file:
https://github.com/willdata0101/Udacity/blob/master/Image%20Classifier%20Project%20(1).ipynb
I need to return the top 5 probabilities and classes for a certain image. I'm not sure how. This is the code that I'm having trouble with:
def predict(image, model, topk=5):
''' Predict the class (or classes) of an image using a trained deep learning model.
'''
# TODO: Implement the code to predict the class from an image file

image_tensor = pil_loader(image)
image_tensor = test_transforms(image_tensor).float()
input = Variable(image_tensor)
input = input.to(device)
output = model.forward(input)
output = torch.exp(output)
probs, classes = output.topk(topk, dim=1)
return probs, classes

class_to_idx = {v:k for k,v in class_to_idx.iteritems()}

for idx in class_to_idx:
    classes.append(class_to_idx[idx])

image = test_dir + '/1/image_06743.jpg'
probs, classes = predict(image, model)


title: Why is PyTorch handling similar weight update assignments differently?

body:
The below code does work updating the weights using:
w -= lr * w.grad

But when updating the weights using
w = w - lr * w.grad

it throws me:

element 0 of tensors does not require grad and does not have a grad_fn

Why is that and shouldn't both assignments be equal?
import torch

X = torch.tensor([1, 2, 3, 4], dtype=torch.float32)
y = torch.tensor([2, 4, 6, 8], dtype=torch.float32)
w = torch.tensor(0.0, dtype=torch.float32, requires_grad=True)
epochs = 10
lr = 0.002

for epoch in range(1, epochs + 1):
    y_pred = w * X
    loss = ((y_pred - y)**2).mean()
    loss.backward()

    print(w.grad)
    with torch.no_grad():
        ### Option 1 - doesn't work
        w = w - lr * w.grad

        ### Option 2 - does work
        w -= lr * w.grad

    w.grad.zero_()


title: Visualize PyTorch GPU arrays directly without copying them from GPU to CPU and back

body:
I'd like to visualize a PyTorch array (for example an RGB image that is the output of a neural network) that is in GPU memory.
The array changes very often, I want to see it "live", with little latency and with high frame rate.
Usually I copy it from GPU memory to main memory and then visualize it (i.e. then the image usually goes back to GPU memory in order to be visualized, as far as I know). This copying back and forth creates unnecessary overhead.
Can I instead allow PyTorch to write directly to some graphics buffer? Or at least fetch the data from PyTorch to a graphics buffer (i.e. GPU to GPU, instead of GPU to main memory to GPU)?
An example application is the usage of neural networks for effects/filters for games, screensavers, interactive art installations, live previews in image and video editing, ...


title: convert stft tensorflow code to pytorch/pure python code

body:
I am trying to convert the following code (fromhttps://github.com/saiteja-talluri/Speech2Face/blob/master/preprocess/speaker.py) to PyTorch or pure python.

def find_spec(self, filename):
        print("-------------finding spectrogram for {0}----------------".format(filename))
        with tf.Session(graph=tf.Graph()) as sess:

            holder = tf.placeholder(tf.string, [])
            file = tf.io.read_file(holder)
            decoder = tf.contrib.ffmpeg.decode_audio(file, file_format = "wav", samples_per_second = self.sample_rate, channel_count = 1)
            stft = tf.signal.stft(tf.transpose(a=decoder), frame_length = self.window, frame_step = self.stride, fft_length = self.fft_length, window_fn = tf.signal.hann_window)
            amp = tf.squeeze(tf.abs(stft)) ** self.amp_norm
            phase = tf.squeeze(tf.math.angle(stft))
            stacked = tf.stack([amp, phase], 2)
            stft = sess.run(stacked, feed_dict = {holder : self.audios_path + filename + ".wav"})
            pickle.dump(stft, open(self.spect_path + filename  + ".pkl", "wb"))
            print("============STFT SHAPE IS {0}=============".format(stft.shape))

I tried to use the code from here (https://tbnsilveira.info/2019/02/23/finding-cyclic-patterns-a-tutorial-on-how-to-implement-stft-in-python/) to convert, but I could not reach to a runnable code.
I really appreciate any help. Thank you.


title: Style loss is always zero

body:
I am trying to use feature reconstruction and style reconstruction losses on my model. For this, I followed the example code on PyTorch website for “Neural Style Transfer”.
https://pytorch.org/tutorials/advanced/neural_style_tutorial.html
Although the feature loss is calculated without problem, the style loss is always zero. And I could not find the reason since everything looks fine in the implementation. The calculation methods are the same as the proposed mathematical methods for these loss functions. Besides, as you know, the style and feature losses are almost the same in terms of calculation except Gram matrix step in style loss and no problem in feature loss.
Could anyone help me with this situation?
class Feature_and_style_losses():
def __init__(self, ):
    self.vgg_model = models.vgg19(pretrained=True).features.cuda().eval()
    self.content_layers = ['conv_16']
    self.style_layers = ['conv_5']

def calculate_feature_and_style_losses(self, input_, target, feature_coefficient, style_coefficient):
    i = 0
    feature_losses = []
    style_losses = []
    for layer_ in self.vgg_model.children():
        if isinstance(layer_, nn.Conv2d):
            i += 1
            name = "conv_{}".format(i)

            if name in self.content_layers:
                features_input = self.vgg_model(input_).detach()
                features_target = self.vgg_model(target).detach()
                feature_losses.append(self.feature_loss(features_input, features_target))
            if name in self.style_layers:
                style_input = self.vgg_model(input_).detach()
                style_target = self.vgg_model(target).detach()
                style_losses.append(self.style_loss(style_input, style_target))

    feature_loss_value = (torch.mean(torch.from_numpy(np.array(feature_losses, dtype=np.float32)))) * feature_coefficient
    style_loss_value = (torch.mean(torch.from_numpy(np.array(style_losses, dtype=np.float32)))) * style_coefficient

    return feature_loss_value, style_loss_value

def feature_loss(self, input_, target):
    target = target.detach()
    feature_reconstruction_loss = F.mse_loss(input_, target)
    return feature_reconstruction_loss

def gram_matrix(self, input_):
    a, b, c, d = input_.size() #??? check size
    features = input_.view(a*b, c*d)
    #features_t = features.transpose(1, 2)
    #G = features.bmm(features_t) / (b*c*d)
    #print(features.shape)
    G = torch.mm(features, features.t())
    return G.div(a*b*c*d)
    return G

def style_loss(self, input_, target):
    G_input = self.gram_matrix(input_)
    G_target = self.gram_matrix(target).detach()
    #style_reconstruction_loss = self.feature_loss(G_input, G_target)
    style_reconstruction_loss = F.mse_loss(G_input, G_target)
    return style_reconstruction_loss

feature_loss_ = Feature_and_style_losses()
...
for e in range(epochs):
    for i, batch in enumerate(dataloader):
        ...
        real_C = Variable(batch["C"].type(Tensor))
        fake_C = independent_decoder(features_all)
        f_loss, s_loss = feature_loss_.calculate_feature_and_style_losses(fake_C, real_C, 1, 10)
        loss_G_3 = loss_GAN_3 + lambda_pixel * (loss_pixel_3_object + loss_pixel_3_scene) * 0.5 + f_loss + s_loss
        loss_G_3.backward(retain_graph=True)
        optimizer_independent_decoder.step()

Best.


title: Is it possible to train a neural netowrk using a loss function on unseen data (data different from input data)?

body:
Normally, a loss function may be defined as
L(y_hat, y) or L(f(X), y), where f is the neural network, X is the input data, y is the target.
Is it possible to implement (preferably in PyTorch) a loss function that depends not only on the input data X, but also on X' (X != X)?
For example, let's say I have a neural network f, input data (X,y) and X'. Can I construct a loss function such that

f(X) is as close as possible to y, and also
f(X') > f(X)?

The first part can be easily implemented (PyTorch: nn.MSELoss()), the second part seems to be way harder.
P.S: this question is a reformulation of Multiple regression while avoiding line intersections using neural nets, which was closed. In the original data, input data and photos with a theoretical example are available.


title: Pytorch model is trained with 99% validation accuracy but during prediction predicts only one class

body:
I started to build an end-to-end face recognition system from facenet-pytorch library and I'm having a problem with the prediction part. After the training process is done with a validation accuracy of 99% the model classifies the training data itself wrong and it is completely biased to one a class (Garry_Marshall). Whatever image you take, it will still give you only that class. The code (colab) link is given below and if any other information is needed do ask. The code in question is the part below the heading "Face Classification".
A side question: My training accuracy will be lower than my validation accuracy at the initial stages of training. How is that possible?
https://colab.research.google.com/drive/1z1ARMkXBariUOj0A-PNcrliup8kmGtpH?usp=sharing


title: I can't seem to install Pytorch through Anaconda

body:
I am very new to Python and I'm trying to install Pytorch through Anaconda. I go to Start > Anaconda3 (64-bit) > Anaconda Prompt (anaconda3). I open the prompt and type in the command that I got from the website https://pytorch.org/get-started/locally, the command being, conda install pytorch torchvision cudatoolkit=10.2 -c pytorch, but when I type it in it says
"'conda' is not recognized as an internal or external command,
operable program or batch file."
Can someone help me please?


title: How to use torch autograd on an autoencoder model?

body:
I'm trying to extract the jacobian matrix from an autoencoder using pytorch automatic differentiation package (autograd.functional.jacobian).
My code is as follow:
Deffining the Jacobian:
class MyJaco(JacobianReg):

def _real_jacob(self, y, model, x, create_graph=False, strict=True):
    jacob = torch.autograd.functional.jacobian(model, x.float(), create_graph=False,
    strict=True) # (func, inputs, create_graph=False, strict=False)
    sum_jaco = 0
    for k in range(batch_size):
        sum_jaco += jacob[k,:,k]
    return sum_jaco/batch_size

Parameters:
reg = MyJaco() # Jacobian regularization
lambda_JR = 0.01 # hyperparameter
args = easydict.EasyDict({
'epochs':1,########
'reg_param': 0.001
})

epochs = args['epochs']
reg_param = args['reg_param']
learning_rate = 1e-3
batch_size = 3

Define the autoencoder model:
class Autoencoder(nn.Module):
  def __init__(self):
    super(Autoencoder, self).__init__()

    # encoder
    self.enc1 = nn.Linear(in_features=59, out_features=50, bias= True)
    nn.init.zeros_(self.enc1.bias.data)

    # decoder
    self.dec5 = nn.Linear(in_features=50, out_features=59, bias= True)
    nn.init.zeros_(self.dec5.bias.data)

  def forward(self, x):
    # encoding
    x = F.relu(self.enc1(x))#, nn.init.eye_(self.enc1.weight.data))

    # decoding
    x = F.relu(self.dec5(x))#, nn.init.eye_(self.dec5.weight.data))

    return x

model3 = Autoencoder().to(device)
# the loss function
criterion = nn.MSELoss()
# the optimizer
optimizer = optim.Adam(model3.parameters(), lr=learning_rate)

# get the layers as a list
model_children = list(model3.children())

Define the training function:
def fit(model, dataloader, epoch):
    train_loss = []
    print('Training')
    model.train()
    running_loss = 0.0
    counter = 0
    for i, data in tqdm(enumerate(dataloader), total=int(len(df_train_0_x_rescaled)/dataloader.batch_size)):
        counter += 1
        img = data
        img = img.to(device)
        img.requires_grad = True
        img = img.view(img.size(0), -1)

        optimizer.zero_grad()
        outputs = model(img.float())
        loss_super = criterion(outputs, img)
        jacob = reg(outputs, model, img)
        loss = loss_super
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

    epoch_loss = running_loss / counter
    print(f"Train Loss: {epoch_loss:.3f}")
    return epoch_loss, jacob, outputs

Define the validation function:
def validate(model, dataloader, epoch):
    print('Validating')
    model.eval()
    running_loss = 0.0

    counter = 0
    with torch.no_grad():
        for i, data in tqdm(enumerate(dataloader), total=int(len(df_valid_0_x_rescaled)/dataloader.batch_size)):
            counter += 1
            img = data
            img = img.to(device)

            img = img.view(img.size(0), -1)
            outputs = model(img.float())
            loss = criterion(outputs, img)
            running_loss += loss.item()

    epoch_loss = running_loss / counter
    print(f"Val Loss: {epoch_loss:.3f}")
    return epoch_loss

Train and validate the autoencoder neural network:
train_loss3 = []
val_loss3 = []
start3 = time.time()
for epoch in range(epochs):
    print(f"Epoch {epoch+1} of {epochs}")
    train_epoch_loss3, jacob3, outputs = fit(model3, trainloader, epoch)
    val_epoch_loss3 = validate(model3, testloader, epoch)
    train_loss3.append(train_epoch_loss3)
    val_loss3.append(val_epoch_loss3)
end3 = time.time()

print(f"{(end3-start3)/60:.3} minutes")

I'm currently receiving as a result a 59x59 matrix, which would be expected, but some channels are equal to zero, and this is mathematically wrong.
I have a representation of the output I am getting in the following figure.
Heatmap from 59x59 matrix
Is there any way I can get the jacobian matrix for all input x output channels during training using this function? could someone indicate what i'm doing wrong?


title: How to use the func like torch.nn.functional.conv2d() in mxnet?

body:
I want to do some convolution calculation with input data and a kernel.
In torch, I can write a func:
import torch
def torch_conv_func(x, num_groups):
    batch_size, num_channels, height, width = x.size()
    conv_kernel = torch.ones(num_channels, num_channels, 1, 1)

    return torch.nn.functional.conv2d(x, conv_kernel)

It works well and now I need rebuild in MXnet,so I write this:

from mxnet import nd
from mxnet.gluon import nn

def mxnet_conv_func(x, num_groups):
    batch_size, num_channels, height, width = x.shape
    conv_kernel = nd.ones((num_channels, num_channels, 1, 1))

    return nd.Convolution(x, conv_kernel)

And I got the error
mxnet.base.MXNetError: Required parameter kernel of Shape(tuple) is not presented, in operator Convolution(name="")

How to fix it?


title: Eval method in PyTorch loading

body:
I have a weird experience today. I trained a model in PyTorch and then I tested that with test samples. In this step, I got good results (imagine X). Then I save the model and load the model to see that I can get the same results (X) with the saved model.
Unfortunately, after loading the model, the results are worse (not X). I tested these (please do not discuss them) to fix the problem:

I saved with different methods --> whole model, parameter dictionary
Seed the same randomization in training and testing
Switch from CPU to GPU and return
Using grad zero and so on

Finally, I could figure out the problem. The problem was that I should turn the evaluation off in the loading model step. It means that in the loading section, I do not need to use torch.eval(). I know that eval() effects on Dropout layers and so on in training and evaluation modes.
My question is that when I have no dropout layer in my model, why torch.eval() works for me in reverse?
I have BatchNormalization in my model, I couldn't find the effect of eval() method on BatchNorm and DropOut on the PyTorch website.
In general, my question is that I read that in the loading model section, I should use eval() mode. However, why in my case, it is the opposite?


title: How can I see summary statistics (e.g. number of samples; type of data) of HuggingFace datasets?

body:
I'm looking for suitable datasets to test a few new machine learning ideas. Is there any way to see summary statistics (e.g. number of samples; type of data) of HuggingFace datasets?
They provide descriptions here https://huggingface.co/datasets , but it's a bit hard to filter them.