title: How does Pytorch handles backpropagation in this case?

body:
class ExNet(nn.Module):
    def __init__(self):
        super(ExNet,self).__init__()
        self.fc1 = nn.Linear(1024,500)
        self.fc2 = nn.Linear(500,100)
        self.fc3 = nn.Linear(100,10)

    def forward(self,x,mask1, mask2):
        x = F.relu(self.fc1(x))
        x = x * mask
        x = F.relu(self.fc2(x))
        x = x * mask2
        x = F.softmax(self.fc3(x),dim=1)
        return x

I have some customized masks (depending on the layer) that I would like to apply to my network. According to the code above, does Pytorch consider the masked features during backpropagation? I mean will my gradients during backpropagation be multiplied using these custom masks?


title: Pytorch: convolve a sample with a specific filter

body:
Given a batch of samples, I would like to convolve each of them with different filters. I have implemented the idea with keras and the code works:
import keras.backend as K

def single_conv(tupl):
    inp, kernel = tupl
    outputs = K.conv1d(inp, kernel, padding='same')
    return outputs

# inputs and filters are given in some way
res = K.squeeze(K.map_fn(single_conv, (inputs, filters), dtype=K.floatx()), axis=1)

Is there any way to do this with pytorch?


title: how to get Single text prediction from the CustomisedBERT Classification + PyTorch NLP model with/without DataLoader

body:
I have used BERT with HuggingFace and PyTorch and used DataLoader, Serializer for Training & Evaluation. Below is the code for that:
! pip install transformers==3.5.1
from transformers import AutoModel, BertTokenizerFast

bert = AutoModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

def textToTensor(text,labels=None,paddingLength=30):

  tokens = tokenizer.batch_encode_plus(text.tolist(), max_length=paddingLength, padding='max_length', truncation=True)

  text_seq = torch.tensor(tokens['input_ids'])
  text_mask = torch.tensor(tokens['attention_mask'])

  text_y = None
  if isinstance(labels,np.ndarray): # if we do not have y values
    text_y = torch.tensor(labels.tolist())

  return text_seq, text_mask, text_y

text = test_df['text'].values

seq,mask,_ = textToTensor(text,paddingLength=35)
data = TensorDataset(seq,mask)
dataloader = DataLoader(data,batch_size=1)

for step,batch in enumerate(dataloader):
  batch = [t.to(device) for t in batch]
  sent_id, mask = batch

  with torch.no_grad():
    print(np.argmax(model(sent_id, mask).detach().cpu().numpy(),1))

It gives me a numpy array as a result and since the batch_size=1 and No Serializer is used in this one, I am getting results as single array number as class prediction.
I have two questions:
Are the results strictly according to the index of df['text']?
**How can I get the predictions for a single sentence like Hello make my prediction. I am a single sentence?
Can someone please help me making a single prediction?


I've 2 folders.One image in 1 folder and another in another folder. I have to compare two images and find the dissimilarity but the code is written random folder.
class InferenceSiameseNetworkDataset(Dataset):

    def __init__(self,imageFolderDataset,transform=None,should_invert=True):
        self.imageFolderDataset = imageFolderDataset
        self.transform = transform
        self.should_invert = should_invert

    def __getitem__(self,index):
        img0_tuple = random.choice(self.imageFolderDataset.imgs)
        img1_tuple = random.choice(self.imageFolderDataset.imgs)
        #we need to make sure approx 50% of images are in the same class
        should_get_same_class = random.randint(0,1)
        if should_get_same_class:
            while True:
                #keep looping till the same class image is found
                img1_tuple = random.choice(self.imageFolderDataset.imgs)
                if img0_tuple[1]==img1_tuple[1]:
                    break
        else:
            while True:
                #keep looping till a different class image is found

                img1_tuple = random.choice(self.imageFolderDataset.imgs)
                if img0_tuple[1] !=img1_tuple[1]:
                    break

        img0 = Image.open(img0_tuple[0])
        img1 = Image.open(img1_tuple[0])
        img0 = img0.convert("L")
        img1 = img1.convert("L")

        if self.should_invert:
            img0 = PIL.ImageOps.invert(img0)
            img1 = PIL.ImageOps.invert(img1)

        if self.transform is not None:
            img0 = self.transform(img0)
            img1 = self.transform(img1)

        return img0, img1 , torch.from_numpy(np.array([int(img1_tuple[1]!=img0_tuple[1])],dtype=np.float32))

    def __len__(self):
        return len(self.imageFolderDataset.imgs)

I took this code from GitHub and when I'm trying to compare the two images dissimilarity it's randomly choosing the images. The input folders are 2. one image should be in one folder and another image should be in another folder. When I'm trying to test it's testing on the same image sometimes I mean it's not checking for another image in another folder.
testing_dir1 = '/content/drive/My Drive/Signature Dissimilarity/Forged_Signature_Verification/processed_dataset/training1/'
folder_dataset_test = dset.ImageFolder(root=testing_dir1)
siamese_dataset = InferenceSiameseNetworkDataset(imageFolderDataset=folder_dataset_test,
                                        transform=transforms.Compose([transforms.Resize((100,100)),
                                                                      transforms.ToTensor()
                                                                      ])
                                       ,should_invert=False)

test_dataloader = DataLoader(siamese_dataset,num_workers=6,batch_size=1,shuffle=False)
dataiter = iter(test_dataloader)
x0,_,_ = next(dataiter)

for i in range(2):
  _,x1,label2 = next(dataiter)
  concatenated = torch.cat((x0,x1),0)

  output1,output2 = net(Variable(x0).cuda(),Variable(x1).cuda())
  euclidean_distance = F.pairwise_distance(output1, output2)
  imshow(torchvision.utils.make_grid(concatenated),'Dissimilarity: {:.2f}'.format(euclidean_distance.item()))
  dis = 'Dissimilarity: {:.2f}'.format(euclidean_distance.item())
  dis1 = dis
  dis1 = dis1.replace("Dissimilarity:", "").replace(" ", "")
  print(dis)
  if float(dis1) < 0.5:
    print("It's Same Signature")
  else:
    print("It's Forged Signature")


title: Efficient numpy broadcasting not found

body:
It may be an easy problem but I could not find any practical solution. My code has following code segment involving 3 nested for loops. The target is to create a specialized intensity matrix for my algorithms for both prediction and ground_truth image matrix as follows:
    for i in range (batch):
      for j in range (img_width):
        for k in range (img_height):
            tensor=prediction[i][j][:]-prediction[i][k][:]
            extracted_intensity_pred[i][j][k]=torch.norm(tensor, 2)
            tensor=ground truth[i][j][:]-ground_truth[i][k][:]
            extracted_intensity_ground_truth[i][j][k]=torch.norm(tensor, 2)

This nested for loop structure is slowing the execution intensively. Is there any broadcasting implementation(in numpy or pytorch tensor based) that may be used?


title: Pytorch modifying intermediate values during forward

body: If I have a model with several different layers, is there a way for me to modify the values in between those layers and see what the result would be if that modified value was passed through the rest of the network? I know you can use hooks to obtain the intermediate values during a forward pass but I would also like to modify them and send them through the rest of the network. How would I go about doing this?


title: How to use pytroch to load different size datasets?

body:
I have written a siamese network and want to train it.
The code I have written:
txt_path = 'home\xxx\train.txt'  #contains all different npy files.

class MyDataset(Dataset):

      def __init__(self, txt, transform=None):

            self.txt = txt
            self.transform = transform

      def __getitem__(self, index):

            line = linecache.getline(self.txt, random.randint(1, self.__len__()))
            line.strip('\n')
            emb0_list= line.split()
            should_get_same_class = random.randint(0,1)
            if should_get_same_class:
                  while True:
                        emb1_list = linecache.getline(self.txt, random.randint(1, self.__len__())).strip('\n').split()
                        if emb0_list[1]==emb1_list[1]:
                              break
            else:
                  emb1_list = linecache.getline(self.txt, random.randint(1,self.__len__())).strip('\n').split()

            emb0 = np.load(emb0_list[0])
            emb1 = np.load(emb1_list[0])
            emb0 = np.array(emb0,dtype=np.float32)
            emb1 = np.array(emb1,dtype=np.float32)
            emb0 = torch.from_numpy(emb0)
            emb1 = torch.from_numpy(emb1)

            if self.transform is not None:
                emb0 = self.transform(emb0)
                emb1 = self.transform(emb1)

            label = torch.from_numpy(np.array([int(emb1_list[1]!=emb0_list[1])],dtype=np.float32))
            return emb0, emb1, label

      def __len__(self):
            fh = open(self.txt, 'r')
            num = len(fh.readlines())
            fh.close()
            return num

Here, emb0's shape is [1,128], emb1's shape is [1,700]
train_data = MyDataset(txt = txt_path)
train_dataloader = DataLoader(dataset=train_data, shuffle=True, num_workers=0, batch_size = 20)
for epoch in range(0, 10):
      for i, data in enumerate(train_dataloader, 0):
            emb0, emb1, label = data
            emb0, emb1, label = Variable(emb0), Variable(emb1), Variable(label)

When I set batch_size=1, it works. However when I set batch_size>1, the error is：
stack expects each tensor to be equal size, but got [128] at entry 0 and [1, 700] at entry 1

As the embeddings are from text, so I can't use transform.resize(). I think collate_fn would be helpful. But I have read the document of Pytorch, I'm still confusing how to custom a collate_fn function for my datasets?


title: What is the most efficient way to load a large numpy `.npy` file as PyTorch dataset?

body:
I have a large numpy .npy file, with shape like (100000, 200, 200) for ConvNet. The first dimension is sample size and the last two dimensions can be regarded as the width and height of images.
It will be out of memory if I load the whole file into memory using np.load. I could use mmap_mode='r' to load it lazily. But if I want to preprocess the data, such as normalization, training/test set split, I still need to load all data into memory.
Do I need to preprocess the data and save it as another file, to load them lazily in PyTorch Dataset instance? Is there any "standard" way to load one single large data file for PyTorch? Thank you!
I know there were many similar questions here. But I am still confused after reading them.


title: Pytorch resnet with zero loss

body:
I am training a Pytorch Resnet model for image segmentation. I have two classes and I am training with RGB images and corresponding binary masks. While my accuracy values are very high (~0.99), my loss values for both training and validation are zeroes throughout all epochs. How can I fix this issue?
#we do the spatial transformations first, and afterwards do any color augmentations
img_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomVerticalFlip(),
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(size=(patch_size,patch_size),pad_if_needed=True), #these need to be in a reproducible order, first affine transforms and then color
    transforms.RandomResizedCrop(size=patch_size),
    transforms.RandomRotation(180),
    transforms.ColorJitter(brightness=0, contrast=0, saturation=0, hue=.5),
    transforms.RandomGrayscale(),
    transforms.ToTensor()
    ])

mask_transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.RandomVerticalFlip(),
    transforms.RandomHorizontalFlip(),
    transforms.RandomCrop(size=(patch_size,patch_size),pad_if_needed=True), #these need to be in a reproducible order, first affine transforms and then color
    transforms.RandomResizedCrop(size=patch_size,interpolation=PIL.Image.NEAREST),
    transforms.RandomRotation(180),
    ])

dataset={}
dataLoader={}
for phase in phases: #now for each of the phases, we're creating the dataloader
                     #interestingly, given the batch size, i've not seen any improvements from using a num_workers>0
    dataset[phase]=Dataset(f"/content/{dataname}_{phase}.pytable", img_transform=img_transform , mask_transform = mask_transform ,edge_weight=edge_weight)
    dataLoader[phase]=DataLoader(dataset[phase], batch_size=batch_size,
                                shuffle=True, num_workers=0, pin_memory=True)

optim = torch.optim.Adam(model.parameters())
nclasses = dataset["train"].numpixels.shape[1]
class_weight=dataset["train"].numpixels[1,0:2] #don't take ignored class into account here
class_weight = torch.from_numpy(1-class_weight/class_weight.sum()).type('torch.FloatTensor').to(device)

print(class_weight) #show final used weights, make sure that they're reasonable before continouing
criterion = nn.CrossEntropyLoss(weight = class_weight, ignore_index = ignore_index ,reduce=False) #reduce = False makes sure we get a 2D output instead of a 1D "summary" value

for epoch in range(num_epochs):
    #zero out epoch based performance variables
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    all_acc = {key: 0 for key in phases}
    all_loss = {key: torch.zeros(0).to(device) for key in phases}
    cmatrix = {key: np.zeros((2,2)) for key in phases}

    for phase in phases: #iterate through both training and validation states

        if phase == 'train':
            model.train()  # Set model to training mode
        else: #when in eval mode, we don't want parameters to be updated
            model.eval()   # Set model to evaluate mode
        running_loss = 0.0
        running_corrects = 0
        for ii , (X, y, y_weight) in enumerate(dataLoader[phase]): #for each of the batches
            optim.zero_grad()
            X = X.to(device)  # [Nbatch, 3, H, W]
            y_weight = y_weight.type('torch.FloatTensor').to(device)
            y = y.type('torch.LongTensor').to(device)  # [Nbatch, H, W] with class indices (0, 1)
            with torch.set_grad_enabled(phase == 'train'): #dynamically set gradient computation, in case of validation, this isn't needed
                                                            #disabling is good practice and improves inference time

                prediction = model_ft(X)  # [N, Nclass]
                y = y[:, 0,0]
                loss = criterion(prediction, y)
                print(loss)
                _, preds = torch.max(X, 1)
                preds = preds[:,0,0]
                if phase=="train": #in case we're in train mode, need to do back propogation

                    loss.mean().backward()
                    optim.step()

            running_loss += loss.data[0]
            running_corrects += torch.sum(preds == y)
        epoch_loss = running_loss / len(dataLoader[phase].dataset)
        epoch_acc = running_corrects.double() / len(dataLoader[phase].dataset)

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))

        # deep copy the model
        if phase == 'val' and epoch_acc > best_acc:
           best_acc = epoch_acc
           best_model_wts = copy.deepcopy(model.state_dict())
        if phase == 'val':
           val_acc_history.append(epoch_acc)

    print()


title: (Numpy or PyTorch) Sum array elements for given bins

body:
I would like this problem to be solved using PyTorch tensors. If there is no efficient solution in torch, then feel free to suggest a numpy solution.
Let a be a 1-dimensional tensor (or numpy array), and bin_indices be a tensor (np array) of integers between 0 and n excluded. I want to compute the array bins that at position i contains the sum of elements of a[bins_indices == i].
n = 3

a = [1, 4, 3, -2, 5]  # Values

bins_indices = [0, 0, 1, 2, 0]  # Correspondent bin indices

bins = [10, 3, -2]  # bins[0] = 1 + 4 + 5 etc. bins has 3 elements since n=3

If you can provide also a way of making this work for batches I would be immensely grateful to you!


title: Update net.parameters() without .data

body:
Is there any way to update the net parameters with some other tensors that carry gradients as well?
I want to do something like the following:
grads = torch.autograd.grad(loss, net.parameters(),
                                    create_graph=True)

for param gi in zip(net.parameters(), grads):
       param -= eps * gi

And I want each param to carry the grad_fn of gi.


title: Pytorch Dataset. Once catch exception, items do not seem to be added to list

body:
I have a dataset which contains huge amount of pictures (2 millions). A lot of pre-processing has been done and pictures are identified with id. Some of the ids do not exist, but they are generated (FYI, easier to code). This means that when I try to open an image, I surround it with a try/except block. If picture does not exist, I write to a log file and try to add that image identifier's name to a list. I might try to open the same file twice (actually needed for files which exist) and my reasoning was that if I add a picture's identifier to a list, I will not need to catch exception and code will run faster because I can just check if name of the file which does not exist is in the list and if it is, then I can just return None.
I provide some of the code:
     def __init__(self, real_frames_dataframe, fake_frames_dataframe,
                 augmentations, image_size=224):

        # Should increase training speed as on second epoch will not need to catch exceptions
        self.non_existent_files = []

    def __getitem__(self, index):
        row_real = self.real_df.iloc[index]
        row_fake = self.fake_df.iloc[index]

        real_image_name = row_real["image_path"]
        fake_image_name = row_fake["image_path"]

        # Will go here from second epoch
        if real_image_name in self.non_existent_files or fake_image_name in self.non_existent_files:
            return None

        try:
            img_real = Image.open(real_image_name).convert("RGB")
        except FileNotFoundError:
            log.info("Real Image not found: {}".format(real_image_name))
            self.non_existent_files.append(real_image_name)
            return None
        try:
            img_fake = Image.open(fake_image_name).convert("RGB")
        except FileNotFoundError:
            log.info("Fake Image not found: {}".format(fake_image_name))
            self.non_existent_files.append(fake_image_name)
            return None

The problem is that I can see the same identifier to be in the log file multiple times. For example:
Line 3201: 20:56:27, training.DeepfakeDataset, INFO Real Image not found: nvptcoxzah\nvptcoxzah_260.png
Line 3322: 21:23:13, training.DeepfakeDataset, INFO Real Image not found: nvptcoxzah\nvptcoxzah_260.png

I thought the identifier will be appended to non_existent_files and the next time I will not even try to open this file. However, this does not happen. Can anyone explain why?


title: Why does the output of the neural network is the same for all samples

body:
I'm using Pytorch to make a regression model (neural network). Just train and test it on the same sample (because i'm just learning how to build a neural network).
The data is the fish market dataset from kaggle:
https://www.kaggle.com/aungpyaeap/fish-market
Small sample from the training Tensor:
tensor([[ 0.0000, 23.2000, 25.4000, 30.0000, 11.5200,  4.0200],
        [ 0.0000, 24.0000, 26.3000, 31.2000, 12.4800,  4.3056],
        [ 0.0000, 23.9000, 26.5000, 31.1000, 12.3778,  4.6961],
        [ 0.0000, 26.3000, 29.0000, 33.5000, 12.7300,  4.4555],
        [ 0.0000, 26.5000, 29.0000, 34.0000, 12.4440,  5.1340],
        [ 0.0000, 26.8000, 29.7000, 34.7000, 13.6024,  4.9274],
        [ 0.0000, 26.8000, 29.7000, 34.5000, 14.1795,  5.2785],
        [ 0.0000, 27.6000, 30.0000, 35.0000, 12.6700,  4.6900],
        [ 0.0000, 27.6000, 30.0000, 35.1000, 14.0049,  4.8438],
        [ 0.0000, 28.5000, 30.7000, 36.2000, 14.2266,  4.9594],
        [ 0.0000, 28.4000, 31.0000, 36.2000, 14.2628,  5.1042],
        [ 0.0000, 28.7000, 31.0000, 36.2000, 14.3714,  4.8146],
        [ 0.0000, 29.1000, 31.5000, 36.4000, 13.7592,  4.3680],

Small Sample from the target tensor:
tensor([[ 242.0000],
        [ 290.0000],
        [ 340.0000],
        [ 363.0000],
        [ 430.0000],
        [ 450.0000],
        [ 500.0000],
        [ 390.0000],
        [ 450.0000],
        [ 500.0000],
        [ 475.0000],
        [ 500.0000],
        [ 500.0000],
        [ 340.0000],
        [ 600.0000],
        [ 600.0000],
        [ 700.0000],
        [ 700.0000],

The neural network:
class NeuralNetFish(nn.Module):

    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(6, 10)
        self.tanh1 = nn.Tanh()
        self.fc2 = nn.Linear(10, 10)
        self.tanh2 = nn.Tanh()
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        out = self.fc1(x)
        out = self.tanh1(out)
        out = self.fc2(out)
        out = self.tanh2(out)
        out = self.fc3(out)
        return out

model = NeuralNetFish()
criteron = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.02)

for i in range(10000):
    output = model(fish_tensor)
    loss = criteron(output, weight_tensor)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i % 1000 == 0:
        print("Epoch:", i, " ---  Loss:", loss.item())

OUTPUT:
Epoch: 0  ---  Loss: 286071.78125
Epoch: 1000  ---  Loss: 127342.515625
Epoch: 2000  ---  Loss: 127342.515625
Epoch: 3000  ---  Loss: 127342.515625
Epoch: 4000  ---  Loss: 127342.515625
Epoch: 5000  ---  Loss: 127342.515625
Epoch: 6000  ---  Loss: 127342.515625
Epoch: 7000  ---  Loss: 127342.515625
Epoch: 8000  ---  Loss: 127342.515625
Epoch: 9000  ---  Loss: 127342.515625

And i get the same output for all the samples.


The computer was able to run the code on the GPU.

I did a computer restart.

The computer cannot run the code and is recording the following error.

RuntimeError Traceback (most recent call last)
<ipython-input-8-186b7088820a> in <module>
      6 torch.cuda.empty_cache ()
      7
----> 8 print (torch.cuda.memory_allocated (0))

~ / anaconda3 / lib / python3.7 / site-packages / torch / cuda / memory.py in memory_allocated (device)
    276 details about GPU memory management.
    277 "" "
-> 278 return memory_stats (device = device) ["allocated_bytes.all.current"]
    279
    280

~ / anaconda3 / lib / python3.7 / site-packages / torch / cuda / memory.py in memory_stats (device)
    157 result.append ((prefix, obj))
    158
-> 159 stats = memory_stats_as_nested_dict (device = device)
    160 _recurse_add_to_result ("", stats)
    161 result.sort ()

~ / anaconda3 / lib / python3.7 / site-packages / torch / cuda / memory.py in memory_stats_as_nested_dict (device)
    167 "Returns the result of: func:` ~ torch.cuda.memory_stats` as a nested dictionary. "" "
    168 device = _get_device_index (device, optional = True)
-> 169 return torch._C._cuda_memoryStats (device)
    170
    171

RuntimeError: 0 <= device && device <device_num INTERNAL ASSERT FAILED at /pytorch/c10/cuda/CUDACachingAllocator.cpp:879, please report a bug to PyTorch. Invalid device argument.


title: Pytorch: convolution with non-constant kernel

body:
Is there a built-in function in any torch-compatible library that allows to consider convolutions (in 2 dimensions) with kernels that depend on the coordinates of the input.
More precisely, I would like a function such that:

the input has size (C_in,H,W),
kernel has size (C_in,C_out,H,W,K,K) (while usually it has size (C_in,C_out,K,K)),
the output has size (C_out,H,W),
C_in and C_out are the numbers of channels of the input and the output respectively,
H is the height,
W is the width,
K is the size of the kernel (in each of the two directions).

This can also be viewed as a sparse fully connected layer.
For those who know a bit about mathematics, this corresponds to integrating against a kernel with no translation invariance, i.e.
output(x,y)=\integral k(x,y)*input(y) dy

while the usual convolution in torch is:
output(x,y)=\integral k(x - y)*input(y) dy.

I implemented it twice by myself, once using torch.sparse, and once using multiple convolutions. It works but it is so much longer than a usual convolution while the number of basic arithmetic operations is the same (which means that it should take about the same computational time as a convolution).


title: Policy Gradient Loss in Pytorch

body:
Version 1
y = episode_a.argmax(-1)   # episode_a is in shape [T, n_actions]
action_preds = self.net(ep_s)  # action_preds is logits before softmax
neg_log_like = self.loss_fn(action_preds, y)
loss = torch.mean(r * neg_log_like)   # r is reward

Version 2
y = torch.tensor(episode_a, requires_grad=True)
action_preds = model(ep_s)
neg_log_like = -y * torch.log(action_preds)
loss = torch.sum(neg_log_like, 1).mean()

Versions 1 and 2 seem to have same loss value. The difference is, y does not require grad in version while it does in version 2. But it is like a supervised learning backprop operation, y should be not needing require_grad. I do not understand why version 1 cannot train up the policy but version 2 can?


title: cifar100 pytorch download root

body:
train_data=torchvision.datasets.CIFAR100(root='',....,download=True)

When I download CIFAR100 datasets and select the root,are the dataset downloaded in the root that I selected?
I cannot find any file or images.
How can I see the file?


title: Inplace parameter updation without torch.no_grad()

body:
I have just started learning this awesome tool called PyTorch but sadly I am stuck in an equivocal situation.
Below is a code snippet from one of the tutorials
 with torch.no_grad():
       weights -= weights.grad * lr
       bias -= bias.grad * lr
       weights.grad.zero_()
       bias.grad.zero_()

I am kind of confused that even if I will do parameter update without using torch.no_grad() ( i.e. only in-place ) like this:-
 # with torch.no_grad()
   weights -= weights.grad * lr
   bias -= bias.grad * lr
   weights.grad.zero_()
   bias.grad.zero_()

and since the backward call has already been made in the code above this snippet( not included in the snippet ) which basically means all the “grad” attributes are already computed and don’t require the “original” values again. Then, why is it illegal to do those operations without torch.no_grad()?
I know it will flag off the error in PyTorch but I just wanted to know where my line of thought is at fault?


title: Is there a way to figure out whether PyTorch model is on cpu or on the device?

body:
I would like to figure out, whether the PyTorch model is on cpu or cuda in order to
initialize some other variable as Torch.Tensor or Torch.cuda.Tensor depending on the model.
However, looking at the output of the dir() function I see only .cpu(), .cuda(), to() methods which put the model on device, GPU or other device, specified in to. For PyTorch tensor there is is_cuda attribute, but no analogue for the whole model.
Is there some way to deduce this for a model, or one needs to refer to a particular weight?


title: Get hash value of a pytorch architecture?

body:
I would like to automatically check whether a certain architecture has already been trained on a task. My thought is: If I can get a hash value of the architecture and store this value in a .json file, then I can check whether it has already been trained by checking whether the architecture's hash value is in the .json file.
However, I'm not exactly sure what to hash: If I hash the module object, then it will be different each time I run the program since it'll have a different id, because it's in a different memory location. Also, different random initializations will probably cause a different hash value.
Is there a way I can get a hash value that will be the same so long as the module consists of the same layers with the same dimensions?


title: Deploy pytorch model on webcam

body:
I am trying to deploy PyTorch classifier on webcam, but always getting errors, mostly "AttributeError: 'collections.OrderedDict' object has no attribute 'load_state_dict'".  The classifier is a binary classifier. Saved the model as .pt file.
Hope for your support to resolve the issue.
Here are the codes I am using:
import numpy as np
import torch
import torch.nn
import torchvision
from torch.autograd import Variable
from torchvision import transforms
import PIL
import cv2
#This is the Label
Labels = { 0 : 'Perfect',
           1 : 'Defected'
        }
# Let's preprocess the inputted frame
data_transforms = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")   ##Assigning the Device which will do the calculation
model  = torch.load("defect_classifier.pt") #Load model to CPU
model.load_state_dict(torch.load("defect_classifier.pt"))
model  = model.to(device)   #set where to run the model and matrix calculation
model.eval()                #set the device to eval() mode for testing
#Set the Webcam
def Webcam_720p():
    cap.set(3,1280)
    cap.set(4,720)
def argmax(prediction):
    prediction = prediction.cpu()
    prediction = prediction.detach().numpy()
    top_1 = np.argmax(prediction, axis=1)
    score = np.amax(prediction)
    score = '{:6f}'.format(score)
    prediction = top_1[0]
    result = Labels[prediction]
    return result,score
def preprocess(image):
    image = PIL.Image.fromarray(image) #Webcam frames are numpy array format
                                       #Therefore transform back to PIL image
    print(image)
    image = data_transforms(image)
    image = image.float()
    #image = Variable(image, requires_autograd=True)
    image = image.cuda()
    image = image.unsqueeze(0) #I don't know for sure but Resnet-50 model seems to only
                               #accpets 4-D Vector Tensor so we need to squeeze another
    return image                            #dimension out of our 3-D vector Tensor

#Let's start the real-time classification process!

cap = cv2.VideoCapture(0) #Set the webcam
Webcam_720p()
fps = 0
show_score = 0
show_res = 'Nothing'
sequence = 0
while True:
    ret, frame = cap.read() #Capture each frame

    if fps == 4:
        image        = frame[100:450,150:570]
        image_data   = preprocess(image)
        print(image_data)
        prediction   = model(image_data)
        result,score = argmax(prediction)
        fps = 0
        if result >= 0.5:
            show_res  = result
            show_score= score
        else:
            show_res   = "Nothing"
            show_score = score

    fps += 1
    cv2.putText(frame, '%s' %(show_res),(950,250), cv2.FONT_HERSHEY_SIMPLEX, 2, (255,255,255), 3)
    cv2.putText(frame, '(score = %.5f)' %(show_score), (950,300), cv2.FONT_HERSHEY_SIMPLEX, 1,(255,255,255),2)
    cv2.rectangle(frame,(400,150),(900,550), (250,0,0), 2)
    cv2.imshow("ASL SIGN DETECTER", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()
cv2.destroyWindow("ASL SIGN DETECTER")


title: How do you get an intermediate output from StyleGAN2-ADA?

body:
Starting from the official repo, I'm trying to generate intermediate outputs from a pre-trained model for a different downstream task. Since the network I'm feeding it to is 224x224 I only need to generate a 256x256 image, not the full 1024x1024. I realized I could do something like this:
with open('stylegan2-ada-pytorch_pretrained_metfaces.pkl', 'rb') as f:

    styleGAN2_ADA = pickle.load(f)
    G = styleGAN2_ADA['G_ema']
    G.synthesis.block_resolutions = [4, 8, 16, 32, 64, 128, 256,]

Which sort of works ... but the image is washed out. Am I missing something?
Example: result when block_resolutions are changed to above

Example: result without modification of block_resolutions


title: I have a Data in DICOM images i want to convert dicom images in to png or jpg format . but how to convert Number of folders at once with for loop

body:
I have Data in DICOM images, and I want to convert dicom images into png or jpg format. However, I need to convert the images in multiple folders at once using a for-loop. Here what I've done so far:
import pydicom as dicom
import cv2

# specify your image path
image_path = 'Data/LungSegmentation/NSCLC-Radiomics/LUNG1-001/09-18-2008-StudyID-69331/0.000000-82046/1-001.dcm'
ds = dicom.dcmread(image_path)

pixel_array_numpy = ds.pixel_array

image_format = '.jpg' # or '.png'
image_path = image_path.replace('.dcm', image_format)

cv2.imwrite(image_path, pixel_array_numpy)

#show image.....
plt.imshow(ds.pixel_array,cmap ='gray',vmin=0,vmax=255)


title: Choose 2nd GPU on server

body:
I am running code on server. There are 2 GPUs there, and the 1st one is busy. Yet, I can't find a way to switch between them. I am using pytorch if that is important. Following lines of code should be modified:
device = 'cuda' if torch.cuda.is_available() else 'cpu'

Modification may be stated only here.
Thanks.


title: Translating LSTM implementation from tensorflow to pytorch

body:
I'm attemping to translate an old TensorFlow code into a PyTorch equivalent, and I'm struggling with understanding TensorFlow's implementation of LSTMs.
Original code:
        rnn_cell_basic = tf.nn.rnn_cell.BasicLSTMCell(self.rnn_size, state_is_tuple=False)
        cell = tf.nn.rnn_cell.MultiRNNCell([rnn_cell_basic] * self.num_rnn_layers, state_is_tuple=False)

How does that translate into PyTorch?
        self.lstm = nn.LSTM(input_size=?, hidden_size =?, num_layers=?, batch_first=True)

My guess is num_layers = num_rnn_layers, however as for input_size and hidden_size, I'm not sure which one's which.


title: reading signed ints from openCV in/to libtorch tensor c++

body:
I have a cv::Mat which is CV_32SC3 type, it stores both positive and negative values.
When convert it to tensor, the values are messed up:
cout << in_img << endl;

 auto tensor_image = torch::from_blob(in_img.data, {1, in_img.rows, in_img.cols, 3}, torch::kByte);

The in_img has negative values, while after print out tensor_image, the values were all totally different than in_img.
the negative values are gone (it somehow seems to normilise it 255 range). I tried converting to Long like so:
auto tensor_image = torch::from_blob(in_img.data, {1, in_img.rows, in_img.cols, 3}, torch::kLong);

but when I print the values like so, I get seg fault:
  std::cout << "tensor_image: " << tensor_image << " values." << std::endl;

so, I tried looking at just the first element like so:
std::cout << "input_tensor[0][0][0][0]: " << tensor_image[0][0][0][0] << " values." << std::endl;

and the value is not the same as I see in the python implementation :((


title: How F.adaptive_max_pool work when input size smaller than the target size in PyTorch?

body:
In the below case, the input.shape is (5, 7) but target size is (7, 7). Accoding to the standard max pooling formula, the input size seems to be smaller than the target size if padding = 0.
I think there is a padding stage before the max_pooling operation, but how it work ?
Thanks for your help.
import torch
import torch.nn.functional as F

input = torch.rand((1, 3, 5, 7))
output = F.adaptive_max_pool2d(input, (7,7))


title: how to use tbb parallel for instead of for

body:
I am trying to use bb::parallel_for instead of for loop here, I read lots of documents but I am not able to find good examples
model->eval();
    torch::NoGradGuard no_grad;

    double running_loss = 0.0;
    size_t num_correct = 0;

    for (const auto& batch : *test_loader) {
        auto data = batch.data.to(device);
        auto target = batch.target.to(device);

        auto output = model->forward(data);

        auto loss = torch::nn::functional::cross_entropy(output, target);
        running_loss += loss.item<double>() * data.size(0);

        auto prediction = output.argmax(1);
        num_correct += prediction.eq(target).sum().item<int64_t>();
    }


title: PyTorch Lightning wants create a folder on import due to usage of wandb, which raises error on AWS Lambda

body:
So I want to build a Docker image with PyTorch Lightning that can be used with AWS lambda. However, when the function is invoked it raises an OS Error, that claims it uses a Read-only file system and wandb.py wants to write something.
I tried these things:

Overwrite the wandb.py file of pytroch lightning, that it does not init wandb --> Raises error
Execute a python script in Dockerfile, that the files are created on docker build and exist, when invoking the lambda function --> Same OS error

Does someone know a way to skip the wandb.py?
This is the error message:
START RequestId: ddae284d-4f32-4dc6-8160-d1fa62ba9772 Version: $LATEST
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
[ERROR] OSError: [Errno 30] Read-only file system: '/home/sbx_user1051'
Traceback (most recent call last):
  File "/var/lang/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/inference.py", line 5, in <module>
    import pytorch_lightning as pl
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/__init__.py", line 63, in <module>
    from pytorch_lightning.callbacks import Callback
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/callbacks/__init__.py", line 25, in <module>
    from pytorch_lightning.callbacks.swa import StochasticWeightAveraging
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/callbacks/swa.py", line 26, in <module>
    from pytorch_lightning.trainer.optimizers import _get_default_scheduler_config
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/trainer/__init__.py", line 18, in <module>
    from pytorch_lightning.trainer.trainer import Trainer
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 30, in <module>
    from pytorch_lightning.loggers import LightningLoggerBase
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/loggers/__init__.py", line 31, in <module>
    from pytorch_lightning.loggers.wandb import _WANDB_AVAILABLE, WandbLogger  # noqa: F401
  File "/var/lang/lib/python3.8/site-packages/pytorch_lightning/loggers/wandb.py", line 34, in <module>
    import wandb
  File "/var/lang/lib/python3.8/site-packages/wandb/__init__.py", line 131, in <module>
    api = InternalApi()
  File "/var/lang/lib/python3.8/site-packages/wandb/apis/internal.py", line 17, in __init__
    self.api = InternalApi(*args, **kwargs)
  File "/var/lang/lib/python3.8/site-packages/wandb/sdk/internal/internal_api.py", line 73, in __init__
    self._settings = Settings(
  File "/var/lang/lib/python3.8/site-packages/wandb/old/settings.py", line 25, in __init__
    self._global_settings.read([Settings._global_path()])
  File "/var/lang/lib/python3.8/site-packages/wandb/old/settings.py", line 105, in _global_path
    util.mkdir_exists_ok(config_dir)
  File "/var/lang/lib/python3.8/site-packages/wandb/util.py", line 687, in mkdir_exists_ok
    os.makedirs(path)
  File "/var/lang/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/var/lang/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/var/lang/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
END RequestId: ddae284d-4f32-4dc6-8160-d1fa62ba9772
REPORT RequestId: ddae284d-4f32-4dc6-8160-d1fa62ba9772  Duration: 27000.33 ms   Billed Duration: 27001 ms   Memory Size: 10240 MB   Max Memory Used: 241 MB
Unknown application error occurred


title: Visualizing CNN

body:
Hi I'm trying to visualize CNN.
I've been going through https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html
to study CNN by visualizing the structure. What I couldn't understand is its dimension.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

So the code part is supposed to be the CNN model structure from the image.
What I don't get is this.

Convolutions happen from the input to C1 and 3*3 kernel was used. In this case, shouldn't the dimension of C1 be 30 X 30 instead of 28 X 28?
According to the image, dimension of the input to layer F5 is 16 X 5 X 5 but the code says otherwise. It seems like layer F5 is taking an input of dimension 16 X 6 X 6.

I'm not sure whether I'm taking it incorrectly or the image is wrong.


title: Autoencoder outputs B&W images from color

body:
We are in the process of training an AE on CIFAR10 images. We used the following architecture:
class OurAE(nn.Module):
    def __init__(self, in_channels, z_channels):
        super(OurAE, self).__init__()

        self.tot_diff = None

        self.in_channels = in_channels

        curr_channels = in_channels

        encoder = []
        channels = [3, 16, 64] + [z_channels]

        for out_channels in channels:
            encoder += [
                nn.Conv2d(in_channels=curr_channels, out_channels=out_channels, kernel_size=3, padding=1, stride=2),
                nn.ReLU()
            ]
            curr_channels = out_channels

        self.encoder = nn.Sequential(*encoder)

        curr_channels = z_channels
        decoder = []
        channels = [64, 16, 3] + [in_channels]

        for out_channels in channels:
            decoder += [
                nn.ConvTranspose2d(in_channels=curr_channels, out_channels=out_channels, kernel_size=4, padding=1, stride=2),
                nn.ReLU()
            ]
            curr_channels = out_channels

        decoder = decoder[:-1] # removing the RELU layer
        decoder.append(nn.Sigmoid())

        self.decoder = nn.Sequential(*decoder)

    def forward(self, x):
        return self.decoder(self.encoder(x))

We are not sure why but we always get black and white images.
We tried to replace the Sigmoid with ReLU in the last layer but to no avail.
These are the loss function and the optimizers that we used:
optimizer = torch.optim.Adam(classifier.parameters(), lr=lr)
criterion = torch.nn.CrossEntropyLoss()

Here is an example of an input and output of the AE after training:


title: How can I emulate a GPU for testing code written in Pytorch?

body: Often, I code on my laptop which is not equipped with a GPU (MacBook, if it makes a difference). Then files are transferred to a server with a GPU. I just want to perform a sanity-check on my code, before running it on the server, in order to avoid errors related to tensors being on different devices. I am looking for a gpu emulator, which takes in some tenosrs and outputs some other random tensors.


title: why function .view(batch_size,-1) gives the same outputs?

body:
I am a freshman in neural network and I have built a vgg16 network.But in every batch all the inputs leads to the same outputs.So I checked the output of every layer and finally found that x=x.view(batch_size,-1) gives the same outputs!I have no idea why this would happen. here are part of my code:
class VGG16(torch.nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        self.conv1 = torch.nn.Conv2d(3, 64, padding=1, kernel_size=3)              #kernel
        self.conv2 = torch.nn.Conv2d(64, 64, padding=1, kernel_size=3)
        self.conv3 = torch.nn.Conv2d(64, 128, padding=1, kernel_size=3)
        self.conv4 = torch.nn.Conv2d(128, 128, padding=1, kernel_size=3)
        self.conv5 = torch.nn.Conv2d(128, 256, padding=1, kernel_size=3)
        self.conv6 = torch.nn.Conv2d(256, 256, padding=1, kernel_size=3)
        self.conv7 = torch.nn.Conv2d(256, 256, padding=1, kernel_size=3)
        self.conv8 = torch.nn.Conv2d(256, 512, padding=1 ,kernel_size=3)
        self.conv9 = torch.nn.Conv2d(512, 512, padding=1, kernel_size=3)
        self.conv10 = torch.nn.Conv2d(512, 512, padding=1, kernel_size=3)
        self.conv11 = torch.nn.Conv2d(512, 512, padding=1, kernel_size=3)
        self.conv12 = torch.nn.Conv2d(512, 512, padding=1, kernel_size=3)
        self.conv13 = torch.nn.Conv2d(512, 512, padding=1, kernel_size=3)
        self.pooling = torch.nn.MaxPool2d(2)                                        #pool
        self.fc1 = torch.nn.Linear(25088, 4096)                                     # 7 * 7 * 512 = 25088
        self.fc2 = torch.nn.Linear(4096, 4096)
        self.fc3 = torch.nn.Linear(4096, 2)

    def forward(self,x):
        batch_size = x.size(0)
        x = F.relu(self.conv1(x))                                                #layer1
        x = self.pooling(F.relu(self.conv2(x)))                                  #layer2
        x = F.relu(self.conv3(x))                                                #layer3
        x = self.pooling(F.relu(self.conv4(x)))                                  #layer4
        x = F.relu(self.conv5(x))                                                #layer5
        x = F.relu(self.conv6(x))                                                #layer6
        x = self.pooling(F.relu(self.conv7(x)))                                  #layer7
        x = F.relu(self.conv8(x))                                                #layer8
        x = F.relu(self.conv9(x))                                                #layer9
        x = self.pooling(F.relu(self.conv10(x)))                                 #layer10
        x = F.relu(self.conv11(x))                                               #layer11
        x = F.relu(self.conv12(x))                                               #layer12
        x = self.pooling(F.relu(self.conv13(x)))                                 #layer13
        x = x.view(batch_size,-1)                                                #flatten
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

this is the training part:
def train(epoch):
running_loss = 0.0
for batch_idx, data in enumerate(train_loader,0):
    inputs, true_labels = data
    optimizer.zero_grad()                        #clear the optimizer to avoid accumulating of grad
    #forward
    outputs = model(inputs)
    loss = criterion(outputs, true_labels)
    #backward
    loss.backward()
    #update
    optimizer.step()
    running_loss += loss.item()

    #output the train result every 10 loop
    if (batch_idx + 1) % 10 == 0:
        print('[%d %5d] loss: %.3f' %(epoch + 1, batch_idx + 1, running_loss/10 ))
        running_loss = 0.0

this is the outputs of layer13(before view):enter image description here
this is the outputs of x.view :enter image description here
I am searching for a long time on net.But no use.Any ideas?
Thanks in advance.


title: What does the * sign mean in this NN built by Pytorch?

body:
I was reading the code for Generative Adversarial Nets Code by https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/gan/gan.py, I would like to know what the * sign means here, I searched on Google and Stackoverflow but could not find a clear explanation.
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_feat, out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat, 0.8))
            layers.append(nn.LeakyReLU(0.2, inplace=True))
            return layers

        self.model = nn.Sequential(
            *block(opt.latent_dim, 128, normalize=False),
            *block(128, 256),
            *block(256, 512),
            *block(512, 1024),
            nn.Linear(1024, int(np.prod(img_shape))),
            nn.Tanh()
        )


title: Running Pytorch training Script of YOLOV5 in Azure ML

body:
from azureml.core import ScriptRunConfig
from azureml.core.runconfig import MpiConfiguration

src = ScriptRunConfig(source_directory=project_folder,
                      script='train.py',
                      arguments=["-m", "torch.distributed.launch", "--nproc_per_node", 2, "--img",
                      640, "--batch", 16, "--epochs", 3, "--data", "coco128.yaml", "--weights",
                      "yolov5x.pt"],
                      compute_target=compute_target,
                      distributed_job_config=MpiConfiguration(process_count_per_node=1,
                      node_count=2),
                      environment=pytorch_env)

Based on my understanding right now the code is running as :
$ python train.py -m torch.distributed.launch --nproc_per_node 2 --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5x.pt

But I want to run the code as :
$ python -m torch.distributed.launch --nproc_per_node 2 train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5x.pt

Can anyone tell me how to execute this command in Azure ML??
Thanks in advance!!


title: Why putting value on GPU slow the calculation?

body:
I'm a new one in GPU parallezation. I found putting values on GPU in advance slows the calculations and indexing. My code is as follows:
import torch

A = torch.rand(600, 600, device='cuda:0')
row0 = torch.tensor(100, device='cuda:0')
col0 = torch.tensor(100, device='cuda:0')
row1 = torch.tensor(356, device='cuda:0')
col1 = torch.tensor(356, device='cuda:0')
B = torch.rand(256, 256, device='cuda:0')
a = 10

%timeit B[:] = A[row0:row1, col0:col1]
# 395 µs ± 4.01 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit a*A + a**2
# 17 µs ± 256 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

A = torch.rand(600, 600, device='cuda:0')
row0 = 100
col0 = 100
row1 = 356
col1 = 356
B = torch.rand(256, 256, device='cuda:0')
a1 = torch.as_tensor(a).cuda()

%timeit B[:] = A[row0:row1, col0:col1]
# 10.6 µs ± 141 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
%timeit a1*A + a1**2
# 30.2 µs ± 584 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Could anyone explain the mechanism behind it?


title: Why do we need to pass the gradient parameter to the backward function in PyTorch?

body:
According to the docs, when we call the backward function to the tensor if the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying gradient.
import torch
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([1.,1.]))

print(a.grad)

Output: tensor([20., 20.])
Now scaling the external gradient:
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([2.,2.])) #modified

print(a.grad)

Output: tensor([40., 40.])
So, passing the gradient argument to backward seems to scale the gradients.Also, by default F.backward() is F.backward(gradient=torch.Tensor([1.]))
Apart from scaling the grad value how does the gradient parameter passed to the backward function helps to compute the derivatives when we have a non-scalar tensor?Why can't PyTorch calculate the derivative implicitly without asking explicit gradient parameter as it did for the scalar tensor?


title: Federated reinforcement learning

body:
I am implementing federated deep Q-learning by PyTorch, using multiple agents, each running DQN.
My problem is that when I use multiple replay buffers for agents, each appending experiences at the corresponding agent, two elements of experiences in each agent replay buffer, i. e., "current_state" and "next_state" becomes the same after the first time slot. I mean in each buffer, we see the same values for current states and the same values for next states.
I have included simplified parts of the code and results below. Whay is it changing the current states and next states already exixting in the buffer when doing append? Is there something wrong with defining the buffers as a global variable? or do you have another idea?
<<< time 0 and agent 0:
current_state[0] = [1,2]
next_state[0] = [11,12]
*** experience: (array([ 1., 2.]), 2.0, array([200]), array([ 11., 12.]), 0)
*** buffer: deque([(array([ 1., 2.]), 2.0, array([200]), array([ 11., 12.]), 0)], maxlen=10000)

<<< time 0 and agent 1:
current_state[1] = [3, 4]
next_state[1] = [13, 14]
*** experience: (array([ 3., 4.]), 4.0, array([400]), array([ 13., 14.]), 0)
*** buffer: deque([(array([ 1., 2.]), 4.0, array([400]), array([ 11., 12.]), 0)], maxlen=10000)

<<< time 1 and agent 0:
current_state = [11,12]
next_state[0] = [110, 120]
*** experience: (array([ 11., 12.]), 6.0, array([600]), array([ 110., 120.]), 0)
*** buffer: deque([(array([ 11., 12.]), 2.0, array([200]), array([ 110., 120.]), 0),(array([ 11., 12.]), 6.0, array([600]), array([ 110., 120.]), 0)], maxlen=10000)

<<< time 1 and agent 1:
current_state = [13, 14]
next_state[1] = [130, 140]
*** experience: (array([ 13., 14.]), 8.0, array([800]), array([ 130., 140.]), 0)
*** buffer: deque([(array([ 13., 14.]), 4.0, array([400]), array([ 130., 140.]), 0),(array([ 13., 14.]), 8.0, array([800]), array([ 130., 140.]), 0)], maxlen=10000)

class BasicBuffer:
def __init__(self, max_size):
    self.max_size = max_size
    self.buffer = deque(maxlen=10000)

def add(self, current_state, action, reward, next_state, done):
    ## """"Add a new experience to buffer.""""
    experience = (current_state, action, np.array([reward]), next_state, done)
    self.buffer.append(experience)

def DQNtrain(env, state_size, agent):
for time in range(time_max):
    for e in range(agents_numbers):
       current_state[e,:]
        next_state_edge[e, :]
        ## """"Add a new experience to buffer.""""
        replay_buffer_t[e].add(current_state, action, reward, next_state, done)
        current_state[e, :] = next_state[e, :]

if __name__ == '__main__':
   DQNtrain(env, state_size, agent)
   replay_buffer_t = [[] for _ in range(edge_max)]
   for e in range(edge_max):
       replay_buffer_t[e] = BasicBuffer(max_size=agent_buffer_size)


title: Why the two outputs aren't equal in Pytorch derivation?

body:
I thought from the view in the derivation of composite functions(Chain rule), these two representations should be equal.
import torch

x = torch.rand(3, requires_grad=True)
y = x + 2
z = 2 * y * y
z = z.mean()
z.backward()
print(4 * (x + 2))
print(x.grad)

But the outputs are as follows:
tensor([ 8.5011,  8.6625, 11.9508], grad_fn=<MulBackward0>)
tensor([2.8337, 2.8875, 3.9836])

Why?


title: How to implement binary mask matrix in Keras?

body:
I'm currently working on a project and part of it is reimplementing a model written for a paper in PyTorch in Keras. The overal model classifies proteins based on three elements of their properties: sequence, interaction with other proteins, and domains in their sequence (motifs). The part I'm working on recreating currently is the Protein-Protein Interaction part. Firstly, the input vectors simply go through some fully connected layers which is easy enough to implement in keras. However, the outputs from this model are fed into a 'weight classifier model' which applies a binary mask matrix to inputs using a layer created specifically for this model using PyTorch's nn.functional API.
Here is the code I am struggling to implement in keras:
class Weight_classifier(nn.Module):
    def __init__(self, func):
        super(Weight_classifier, self).__init__()
        # self.weight_layer = nn.Linear(OUT_nodes[func]*3, OUT_nodes[func])
        self.weight_layer = MaskedLinear(OUT_nodes[func]*3, OUT_nodes[func], 'data/{}_maskmatrix.csv'.format(func)).cuda()
        self.outlayer= nn.Linear(OUT_nodes[func], OUT_nodes[func])

    def forward(self, weight_features):
        weight_out = self.weight_layer(weight_features)
        # weight_out = F.sigmoid(weight_out)
        weight_out = F.relu(weight_out)
        weight_out = F.sigmoid(self.outlayer(weight_out))
        return weight_out

class MaskedLinear(nn.Linear):
    def __init__(self, in_features, out_features, relation_file, bias=True):
        super(MaskedLinear, self).__init__(in_features, out_features, bias)

        mask = self.readRelationFromFile(relation_file)
        self.register_buffer('mask', mask)
        self.iter = 0

    def forward(self, input):
        masked_weight = self.weight * self.mask
        return F.linear(input, masked_weight, self.bias)

    def readRelationFromFile(self, relation_file):
        mask = []
        with open(relation_file, 'r') as f:
            for line in f:
                l = [int(x) for x in line.strip().split(',')]
                for item in l:
                    assert item == 1 or item == 0  # relation 只能为0或者1
                mask.append(l)
        return Variable(torch.Tensor(mask))

And this is the paper I am working to, it contains several diagrams and explanations of the models if I have not explained the issue sufficiently.
Many thanks.


title: PyTorch Backwards Automatic Diff. with Weighted Loss

body:
I'm currently trying to implement a method where I perform gradient descent using a weighted loss, and was wondering if someone could help me with the pytorch implementation of this method, since my current one is erroring out
The Psuedo Code for it is below

My implementation (ignoring the for loop):
  model.train()
  ## Create a Dummy Model
  dummy = L2RWCNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM,
                DROPOUT, PAD_IDX)
  dummy.state_dict(model.state_dict())

  ## Get Training and Validation Data AKA Lines 2 & 3
  text_t, labels_t = next(iter(train_iterator))
  text_v, labels_v = next(iter(valid_iterator))

  y_f_hat = dummy(text_t) ## Line 4

  ## Line 5
  eps = Variable(torch.zeros(y_f_hat.size()), requires_grad=True)
  cost = F.binary_cross_entropy_with_logits(y_f_hat.squeeze(), labels_t, reduce = False)
  l_f_meta = torch.sum(eps * cost)
  dummy.zero_grad()
  #Line 6
  grads = torch.autograd.grad(l_f_meta, (dummy.params()), create_graph = True, retain_graph=True)
  #Line 7
  dummy.update_params(l_f_meta, source_params=grads)
  #Line 8
  y_g_hat = dummy(text_v)
  #Line 9
  l_g_meta = F.binary_cross_entropy_with_logits(y_g_hat.squeeze(), labels_v, reduce = True)
  #line 10
  grad_eps = torch.autograd.grad(l_g_meta, eps)[0] ## THIS ERRORS OUT

At line 10, the code crashes due to eps not being in the computational graph. I was wondering if there was a better way to do line 7 in the pseudo-code. Currently, I use a custom function to do it, and I think that might be the issue, since theoretically the computational graph should have a path of eps->l_f_meta->model parameters (aka theta) -> y_g - > l_g, which means both eps and l_g are related, so there theoretically should be a gradient.
My update_params code:
def update_params(self, lr_inner, first_order=False, source_params=None, detach=False):
    if source_params is not None:
        for tgt, src in zip(self.named_params(self), source_params):
            name_t, param_t = tgt

            # name_s, param_s = src
            # grad = param_s.grad
            # name_s, param_s = src
            grad = src
            if first_order:
                grad = to_var(grad.detach().data)
            tmp = param_t - lr_inner * grad
            self.set_param(self, name_t, tmp)
    else:

        for name, param in self.named_params(self):
            if not detach:
                grad = param.grad
                if first_order:
                    grad = to_var(grad.detach().data)
                tmp = param - lr_inner * grad
                self.set_param(self, name, tmp)
            else:
                param = param.detach_()
                self.set_param(self, name, param)

def set_param(self,curr_mod, name, param):
    if '.' in name:
        n = name.split('.')
        module_name = n[0]
        rest = '.'.join(n[1:])
        for name, mod in curr_mod.named_children():
            if module_name == name:
                self.set_param(mod, rest, param)
                break
    else:
        param = nn.Parameter(param)
        setattr(curr_mod, name, param)

If anyone could either give me a better way to update model parameters with a weighted loss, just a better way to implement all of my code, or a solution to the eps not being in the computational graph, That would be greatly appreciated as I've been stuck on this for a long time.


title: My data set size is larger than the memory, how to modify my code？

body:
The code below is part of the data set code I defined, which neglect the getitem(self, index) part. But when I train the model with the dataset, because the dataset is too large, cpu memory can not support the dataset. So I wondering how to modify my datset code!
class VimeoDataset(Dataset):
def __init__(self, dataset_name, batch_size=32):
    self.batch_size = batch_size
    self.path = '/data/train_sample/dataset/'
    self.dataset_name = dataset_name
    self.load_data()
    self.h = 256
    self.w = 448
    xx = np.arange(0, self.w).reshape(1,-1).repeat(self.h,0) #xx shape is(256,448)
    yy = np.arange(0, self.h).reshape(-1,1).repeat(self.w,1) #yy shape is(448,256)
    self.grid = np.stack((xx,yy),2).copy()

def __len__(self):
    return len(self.meta_data)

def load_data(self):
    self.train_data = []
    self.flow_data = []
    self.val_data = []
    for i in range(100):
        f = np.load('/data/train_sample/dataset/{}.npz'.format(i))
        if i < 80:
            self.train_data.append(f['i0i1gt'])
            self.flow_data.append(f['ft0ft1'])
        else:
            self.val_data.append(f['i0i1gt'])
    if self.dataset_name == 'train':
        self.meta_data = self.train_data
    else:
        self.meta_data = self.val_data
    self.nr_sample = len(self.meta_data)

def getimg(self, index):
    data = self.meta_data[index]
    img0 = data[0:3].transpose(1, 2, 0)
    img1 = data[3:6].transpose(1, 2, 0)
    gt = data[6:9].transpose(1, 2, 0)
    flow_gt = (self.flow_data[index]).transpose(1, 2, 0)
    return img0, gt, img1, flow_gt

def __getitem__(self, index):
    img0, gt, img1, flow_gt = self.getimg(index)
    if self.dataset_name == 'train':
        ...

And below is my use example:
dataset = VimeoDataset('train')
sampler = DistributedSampler(dataset)
train_data = DataLoader(dataset, batch_size=args.batch_size, num_workers=8, pin_memory=True, drop_last=True, sampler=sampler)


title: Visualize graph of an object detection network through torch.utils.tensorboard add_graph

body:
I've been trying to visualize an object detection network (faster rcnn) through pytorch build-in torch.utils.tensorboard.
I saw plenty of tutorial/ informations for classifing network, but none for object detection.
I am using the same method that the one recommended with classification:
from torch.utils.tensorboard import SummaryWriter

data_loader = torch.utils.data.DataLoader(dataset, batch_size=10, shuffle=True, num_workers=4, collate_fn=utils.collate_fn)

#download the image net pretrained model and change the number of classes
def get_object_detection_model(num_classes):
        # load a model pre-trained pre-trained on COCO
        model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

        # get number of input features for the classifier
        in_features = model.roi_heads.box_predictor.cls_score.in_features
        # replace the pre-trained head with a new one
        model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

        return model

model = get_object_detection_model(num_classes)

writer = SummaryWriter('path/to/summary/folder')

for epoch in range(num_epochs):

        train_one_epoch(model, optimizer, data_loader, device, epoch, writer, print_freq=10)

def train_one_epoch(model, optimizer, data_loader, device, epoch, writer, print_freq):
    model.train()
    metric_logger = utils.MetricLogger(delimiter="  ")
    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
    header = 'Epoch: [{}]'.format(epoch)

    lr_scheduler = None
    if epoch == 0:
        warmup_factor = 1. / 1000
        warmup_iters = min(1000, len(data_loader) - 1)

        lr_scheduler = utils.warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor)

    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)

        losses = sum(loss for loss in loss_dict.values())
        grid = torchvision.utils.make_grid(images)
        writer.add_image("images", grid)
        #input("...")
        for img in images:
            print(type(img))
            print(img.size())

            #writer.add_image('imresult', img, epoch)
            #

            #input("...")
        # reduce losses over all GPUs for logging purposes
        writer.add_graph(model,images)
        loss_dict_reduced = utils.reduce_dict(loss_dict)
        losses_reduced = sum(loss for loss in loss_dict_reduced.values())

        loss_value = losses_reduced.item()
        writer.add_scalar("Loss/train", loss_value, epoch)
        if not math.isfinite(loss_value):
            print("Loss is {}, stopping training".format(loss_value))
            print(loss_dict_reduced)
            sys.exit(1)

        optimizer.zero_grad()
        losses.backward()
        writer.add_scalar("Loss/train", losses, epoch)
        optimizer.step()

        if lr_scheduler is not None:
            lr_scheduler.step()

        metric_logger.update(loss=losses_reduced, **loss_dict_reduced)
        metric_logger.update(lr=optimizer.param_groups[0]["lr"])

any other addition such as add_image or add_scalar works perfecly fine. But when I try to add an object detection graph, I got this error:
File "/pytorch-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 534, in _slow_forward
result = self.forward(*input, **kwargs)
TypeError: forward() takes from 2 to 3 positional arguments but 11 were given
Has anyone clues or manage to worked over that before?
Thanks!


title: Getting error while implementing DCGAN in pytorch: Kernel size can't be greater than actual input size

body:
I have been trying to train a DCGAN implementation in PyTorch. I am getting the following error:

RuntimeError: Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size

I have modified the discriminator network defined in this implementation.
The modified discriminator network is as follows:
self.main = nn.Sequential(

            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf * 8, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf * 8, ndf * 16, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 16),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf * 16, ndf * 16, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 16),
            nn.LeakyReLU(0.2, inplace=True),

            nn.Conv2d(ndf * 16, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

Thanks in advance!


title: PyTorch: Zero all elements of vector except top k?

body:
I am trying to create a new activation layer, let’s call it topk, that would work as follows. It will take a vector x of size n as input (result of multiplying previous layer output by weight matrix and adding bias) and a positive integer k and would output a vector topk(x) of size n whose elements are:
              x_i (if x_i is one of the top k elements of x)
topk(x)_i =
              0 (otherwise)

While calculating gradient of topk(x), top k elements of x should have gradient 1, everything else 0.
How should I implement this? Any help will be appreciated.


title: CNN-LSTM can not converge with CTC loss

body:
I want to train a LPR(License Plate Recognition) model using ResNet50-LSTM to recognize chinese license plate. First, I use license plates from ten provinces randomly，the model can converge very well in one epoch, but when I add one more province into training datasets,whatever any province, it can not converge.
There are 2000 pics in every provinces,it looks like the follow pics:train dataset pics
The model
class res50LPRModel(Module):

def __init__(self, n_classes, input_shape=(3, 64, 128), pretrained=False,LSTMbidirectional=True,lstmHidenSize=256*2):
    super(res50LPRModel, self).__init__()
    self.input_shape = input_shape
    self.resnet50 = models.resnet50(pretrained=pretrained)
    self.resnet50 = nn.Sequential(*list(self.resnet50.children())[:7])
    if LSTMbidirectional:
        self.fc = nn.Sequential(nn.Linear(in_features=lstmHidenSize*2, out_features=128),nn.Dropout(0.25),nn.Linear(in_features=128, out_features=n_classes))
    else:
        self.fc = nn.Sequential(nn.Linear(in_features=lstmHidenSize, out_features=(lstmHidenSize/2)),nn.Dropout(0.5),nn.Linear(in_features=(lstmHidenSize/2), out_features=n_classes))
    self.conv1 = nn.Sequential(nn.Conv2d(1024, 512, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(512),
                               nn.ReLU(inplace=True))
    self.conv2 = nn.Sequential(nn.Conv2d(512, 256, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(256),
                               nn.ReLU(inplace=True))
    self.conv3 = nn.Sequential(nn.Conv2d(256, 128, kernel_size=3, stride=1, padding=1), nn.BatchNorm2d(128))
    self.lstm = nn.LSTM(input_size=self.infer_features(), hidden_size=lstmHidenSize, num_layers=2,bidirectional=LSTMbidirectional)

def infer_features(self):
    x = torch.zeros((1,) + self.input_shape)
    x = self.resnet50(x)
    x = self.conv1(x)
    x = self.conv2(x)
    x = self.conv3(x)
    x = x.reshape(x.shape[0], -1, x.shape[-1])
    return x.shape[1]

def forward(self, x):
    x = self.resnet50(x)
    x = self.conv1(x)
    x = self.conv2(x)
    x = self.conv3(x)
    x = x.reshape(x.shape[0], -1, x.shape[-1])
    x = x.permute(2, 0, 1)
    x, _ = self.lstm(x)
    x = self.fc(x)
    return x

data loader:
class labelTestDataLoader(Dataset):
def __init__(self, img_dir, imgSize,
             provinces=["皖", "沪", "津", "渝", "冀", "晋", "蒙", "辽", "吉", "黑", "苏", "浙", "京", "闽", "赣", "鲁", "豫", "鄂",
                        "湘", "粤", "桂", "琼", "川", "贵", "云", "藏", "陕", "甘", "青", "宁", "新", "警", "学"],
             alphabets=['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'R', 'S', 'T',
                        'U', 'V', 'W', 'X', 'Y', 'Z'],
             ads=['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'J', 'K', 'L', 'M', 'N', 'P','Q', 'R', 'S', 'T', 'U', 'V',
                  'W', 'X', 'Y', 'Z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'], transform=None):
    self.img_dir = img_dir
    # self.img_paths = []
    # for i in range(len(img_dir)):
    #     print(i)
    #     self.img_paths += [el for el in paths.list_images(img_dir[i])]
    self.img_paths = os.listdir(img_dir)
    # print(self.img_paths)
    self.img_size = imgSize
    self.transform = transform

    self.provinces = provinces
    self.alphabets = alphabets
    self.ads = ads
    self.all_class = self.provinces
    self.all_class.extend(self.alphabets)
    self.all_class.extend(self.ads)
    self.all_class = list(set(self.all_class))
    self.all_class.sort()
    characters = ['_']
    characters.extend(self.all_class)
    self.all_class = characters

def __len__(self):
    return len(self.img_paths)

def __getitem__(self, index):
    img_name = self.img_paths[index]
    img = cv2.imread(os.path.join(self.img_dir, img_name))
    # img = img.astype('float32')
    resizedImage_out = cv2.resize(img, self.img_size)
    # resizedImage = np.transpose(resizedImage, (2, 0, 1))

    resizedImage = self.transform(resizedImage_out)
    # resizedImage /= 255.0
    lbl = img_name.split('/')[-1].split('.')[0].split('-')[-3]
    new_label = []
    label_text =""
    # 01-0_1-249&528_393&586-392&584_249&586_250&530_393&528-0_0_25_27_7_26_29-131-21.jpg
    # 01-0_1-249&528_393&586-392&584_249&586_250&530_393&528-0_0_25_27_7_26_29-131-21.jpg
    # 01-0_1-249&528_393&586-392&584_249&586_250&530_393&528-0_0_2_4_5_23_22-131-21.jpg
    for label_index, label_ in enumerate(lbl.split('_')):
        # print(label_)
        if label_index == 0:
            new_label.append(self.all_class.index(self.provinces[int(label_)]))
            label_text=label_text+(self.provinces[int(label_)])
        elif label_index == 1:
            new_label.append(self.all_class.index(self.alphabets[int(label_)]))
            label_text=label_text+(self.alphabets[int(label_)])
        else:
            new_label.append(self.all_class.index(self.ads[int(label_)]))
            label_text=label_text+(self.ads[int(label_)])
    # label_text=torch.as_tensor(label_text)
    new_label = torch.tensor(new_label)
    # resizedImage = to_tensor(resizedImage)
    label_length = int(len(new_label))  # 预测的长度是多少
    input_length = int(label_length * 3)  # 一般是预测一个数据中间会放一个_来做为间隔数据
    input_length = torch.full(size=(1,), fill_value=input_length, dtype=torch.long)  # 后续中ctc会用到
    target_length = torch.full(size=(1,), fill_value=label_length, dtype=torch.long)
    return resizedImage, new_label, img_name, input_length, target_length,resizedImage_out,label_text

traing code：
import torch.nn.functional as F
from tqdm import tqdm
from PIL import ImageFont, ImageDraw, Image
from torch.utils.data import Dataset, DataLoader
import numpy as np
import torch
from tensorboardX import SummaryWriter
from pylab import mpl
import setproctitle
from model.allModel import rawModel, res18LPRModel, res50LPRModel
from data import labelTestDataLoader
from torchvision.utils import make_grid
import cv2 as cv
import matplotlib.pylab as plt
import torchvision
from torchvision.transforms import ToTensor,Normalize

setproctitle.setproctitle("lpr")

writer = SummaryWriter(log_dir="outlog")
mpl.rcParams['font.sans-serif'] = ['FangSong']  # 指定默认字体
mpl.rcParams['axes.unicode_minus'] = False  # 解决保存图像是负号'-'显示为方块的问题

def decode(sequence):
  a = ''.join([characters[x] for x in sequence])
  s = ''.join([x for j, x in enumerate(a[:-1]) if x != characters[0] and x != a[j + 1]])
  if len(s) == 0:
      return ''
  if a[-1] != characters[0] and s[-1] != a[-1]:
      s += a[-1]
  return s

def decode_target(sequence):
  return ''.join([characters[x] for x in sequence]).replace(' ', '')

def calc_acc(target, output):
  output_argmax = output.detach().permute(1, 0, 2).argmax(dim=-1)
  target = target.cpu().numpy()
  output_argmax = output_argmax.cpu().numpy()
  a = np.array([decode_target(true) == decode(pred) for true, pred in zip(target, output_argmax)])
  return a.mean()

def train(model, optimizer, epoch, dataloader):
  model.train()
  loss_mean = 0
  acc_mean = 0
  with tqdm(dataloader) as pbar:
      for batch_index, (data, target, img_name, input_lengths, target_lengths,imgs,label_text) in enumerate(pbar):

          data, target = data.cuda(), target.cuda()

          optimizer.zero_grad()
          output = model(data)
          # print(input_lengths)
          # print(target_lengths)
          output_log_softmax = F.log_softmax(output, dim=-1)
          loss = F.ctc_loss(output_log_softmax, target, input_lengths, target_lengths)

          loss.backward()
          optimizer.step()

          loss = loss.item()
          acc = calc_acc(target, output)

          if batch_index == 0:
              loss_mean = loss
              acc_mean = acc

          loss_mean = 0.1 * loss + 0.9 * loss_mean
          acc_mean = 0.1 * acc + 0.9 * acc_mean

          pbar.set_description(f'Epoch: {epoch} Loss: {loss_mean:.4f} Acc: {acc_mean:.4f} ')
          writer.add_scalar("train_loss", loss_mean, epoch)
          writer.add_scalar("train_acc", acc_mean, epoch)

          rawImgTensorList=[]
          for index,img in enumerate(imgs):
              tempImg=cv.putText(img.numpy(),label_text[index],(0,20),cv.FONT_HERSHEY_SIMPLEX,0.7,(255,255,255))
              rawImgTensorList.append(torch.from_numpy(tempImg).permute(2, 0, 1))
          showGrid=make_grid(rawImgTensorList,padding=50)
          writer.add_image("batchImg",showGrid,global_step=epoch)

def valid(model, optimizer, epoch, dataloader):
  model.eval()
  with tqdm(dataloader) as pbar, torch.no_grad():
      loss_sum = 0
      acc_sum = 0
      for batch_index, (data, target, img_name, input_lengths, target_lengths) in enumerate(pbar):
          data, target = data.cuda(), target.cuda()

          output = model(data)

          output_argmax = output.detach().permute(1, 0, 2).argmax(dim=-1)
          # predict=decode(output_argmax[0])
          # print('pred:', decode(output_argmax[0]))

          output_log_softmax = F.log_softmax(output, dim=-1)
          loss = F.ctc_loss(output_log_softmax, target, input_lengths, target_lengths)

          loss = loss.item()
          acc = calc_acc(target, output)

          loss_sum += loss
          acc_sum += acc

          loss_mean = loss_sum / (batch_index + 1)
          acc_mean = acc_sum / (batch_index + 1)

          pbar.set_description(f'Test : {epoch} Loss: {loss_mean:.4f} Acc: {acc_mean:.4f} ')
          writer.add_scalar("test_loss", loss_mean, epoch)
          writer.add_scalar("test_acc", acc_mean, epoch)

def text_name(name, image, position, size, color, fontpath="msyh.ttf"):
  fontpath = fontpath  # 32为字体大小
  font = ImageFont.truetype(fontpath, size)
  img_pil = Image.fromarray(image)
  draw = ImageDraw.Draw(img_pil)
  # 绘制文字信息<br># (100,300/350)为字体的位置，(255,255,255)为白色，(0,0,0)为黑色
  draw.text(position, name, font=font, fill=color)
  bk_img = np.array(img_pil)
  return bk_img

if __name__ == '__main__':
  ##车牌标签
  provinces = ["皖", "沪", "津", "渝", "冀", "晋", "蒙", "辽", "吉", "黑", "苏", "浙", "京", "闽", "赣", "鲁", "豫", "鄂",
               "湘", "粤", "桂", "琼", "川", "贵", "云", "藏", "陕", "甘", "青", "宁", "新", "警", "学"]
  alphabets = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'R', 'S', 'T',
               'U', 'V', 'W', 'X', 'Y', 'Z']
  ads = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'J', 'K', 'L', 'M', 'N', 'P', 'Q', 'R', 'S', 'T', 'U', 'V',
         'W', 'X', 'Y', 'Z', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

  batch_size = 16

  ##车牌拍标签拼接
  all_class = provinces
  all_class.extend(alphabets)
  all_class.extend(ads)
  all_class = list(set(all_class))
  all_class.sort()
  characters = ['_']  # 增加预测不出来的数据结果，用_表示
  characters.extend(all_class)  # 所有预测结果的拼接
  ##预测车牌数据为7位数据,鲁K.12345
  target_len = 7
  model_output_len = 7 * 3  ##每个预测目标之间需要穿插一个占位符

  width, height = int(model_output_len * 16), 168  ##每16个wid 输出一个1，预测7个数要21个字符，21个字符 *16 = 336
  n_classes = len(all_class)  # 预测类别

  transformImage = torchvision.transforms.Compose(
      [ToTensor(), Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

  train_set = labelTestDataLoader(r'License plate recognition\datasets\eleven',
                                  (width, height),transform=transformImage)  # 训练数据集
  valid_set = labelTestDataLoader(r'License plate recognition\datasets\LicensePlaceCropped\valid',
                                  (width, height),transform=transformImage)  # 测试数据集

  train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=0)  ##要调成0 否则报进程错还在修改中

  valid_loader = DataLoader(valid_set, batch_size=batch_size, shuffle=True, num_workers=0)
  # 数据展示
  for batch_index, (resizedImage, new_labels, img_name, input_lengths, target_lengths,img,label_text) in enumerate(
          train_loader):  # torch.Size([64, 3, 168, 336])
      if batch_index > 2:
          break

  model = res50LPRModel(n_classes, input_shape=(3, height, width),pretrained=True)
  # model=rawModel(n_classes, input_shape=(3, height, width))
  # model.load_state_dict(torch.load("ctc_LPR_model_dict.pt"))
  # model=torch.load('ctc_carCCPD.pth')
  # torch.save(model.state_dict(),"ctc_LPR_model_dict.pt")
  print(model)
  model = model.cuda()
  #
  # for param in model.parameters():
  #     param.requires_grad = False
  # for param in model.lstm.parameters():
  #     param.requires_grad = True
  # for param in model.fc.parameters():
  #     param.requires_grad = True
  """update parameters using different lr"""
  finetuneLr=8e-5
  lr=1e-4
  optimizer = torch.optim.Adam(
      [{'params': model.resnet50.parameters(), 'lr': finetuneLr},
       {'params': model.lstm.parameters(), 'lr': lr},
       {'params': model.conv1.parameters(), 'lr': lr},
       {'params': model.conv2.parameters(), 'lr': lr},
       {'params': model.conv3.parameters(), 'lr': lr},
       {'params': model.fc.parameters(), 'lr': lr}], amsgrad=True, weight_decay=1e-5)
  # model_train 训练
  epochs = 1
  for epoch in range(1, epochs + 1):
      train(model, optimizer, epoch, train_loader)
      # valid(model, optimizer, epoch, valid_loader)
  ##训练调整参数
  finetuneLr = 1e-5
  lr = 1e-4
  optimizer = torch.optim.Adam(
      [{'params': model.resnet50.parameters(), 'lr': finetuneLr},
       {'params': model.lstm.parameters(), 'lr': lr},
       {'params': model.conv1.parameters(), 'lr': lr},
       {'params': model.conv2.parameters(), 'lr': lr},
       {'params': model.conv3.parameters(), 'lr': lr},
       {'params': model.fc.parameters(), 'lr': lr}], amsgrad=True, weight_decay=1e-5)

  epochs = 20
  for epoch in range(1, epochs + 1):
      train(model, optimizer, epoch, train_loader)
      # valid(model, optimizer, epoch, valid_loader)
  torch.save(model.state_dict(), 'ctc_car_FiveClass_20210409.pth')

It confused me for a long time.Does anybody know why this is happening?
Many thanks for your help!


title: How to do next frame predictions using Pytorch CNN-LSTM?

body:
I'm doing next frame prediction from static images extracted from video and save into disk. I'm using CNN-LSTM, during training feed the model 5 frames and predict the 6th frame, but during evaluation I want the CNN-LSTM model to take it's prediction and use it as input to predict the next future frame it should repeat until predict the 6th frame.
How do you pass the predicted frame from then CNN-LSTM as an input in the next step for 6 iterations? Here is the sample code for my CNN-LSTM model:
class CRNN(nn.Module):
def __init__(self, in_channels=3, sample_size=64, num_classes=100,
            hidden_size=512, num_layers=1, rnn_unit='LSTM'):
    super(CRNN, self).__init__()
    self.in_channels=in_channels
    self.sample_size = sample_size
    self.num_classes = num_classes
    self.rnn_unit=rnn_unit

    # network params
    self.ch1, self.ch2= 64, 128
    self.k1, self.k2 = (5, 5), (5, 5)
    self.s1, self.s2 = (1, 1), (1, 1)
    self.p1, self.p2 = (0, 0), (0, 0)
    self.d1, self.d2 = (1, 1), (1, 1)
    self.input_size  = self.ch2
    self.hidden_size = hidden_size
    self.num_layers  = num_layers

    # network architecture
    # in_channels=3 for rgb
    self.conv1 = nn.Sequential(
        nn.Conv2d(in_channels=self.in_channels, out_channels=self.ch1, kernel_size=self.k1, stride=self.s1, padding=self.p1, dilation=self.d1),
        nn.BatchNorm2d(self.ch1, momentum=0.01),
        nn.ReLU(inplace=True),
        nn.Dropout(p=0.5),
        nn.Conv2d(in_channels=self.ch1, out_channels=self.ch1, kernel_size=1, stride=1),
        nn.MaxPool2d(kernel_size=2),
    )
    self.conv2 = nn.Sequential(
        nn.Conv2d(in_channels=self.ch1, out_channels=self.ch2, kernel_size=self.k2, stride=self.s2, padding=self.p2, dilation=self.d2),
        nn.BatchNorm2d(self.ch2, momentum=0.01),
        nn.ReLU(inplace=True),
        nn.Dropout(p=0.5),
        nn.Conv2d(in_channels=self.ch2, out_channels=self.ch2, kernel_size=1, stride=1),
        nn.MaxPool2d(kernel_size=2),
    )

    if self.rnn_unit=='LSTM':
        self.lstm = nn.LSTM(
        input_size=self.input_size,
        hidden_size=self.hidden_size,
        dropout= 0.5 if self.num_layers > 1 else 0,
        num_layers=self.num_layers,
        batch_first=True,
    )
    if self.rnn_unit=='GRU':
        self.lstm = nn.GRU(
        input_size=self.input_size,
        hidden_size=self.hidden_size,
        dropout= 0.5 if self.num_layers > 1 else 0,
        num_layers=self.num_layers,
        batch_first=True,
    )
    if self.rnn_unit=='RNN':
        self.lstm = nn.RNN(
        input_size=self.input_size,
        hidden_size=self.hidden_size,
        dropout= 0.5 if self.num_layers > 1 else 0,
        num_layers=self.num_layers,
        batch_first=True,
    )

    self.fc1 = nn.Linear(self.hidden_size, self.num_classes)
    self.act = nn.Sigmoid()

def forward(self, X, n_steps=None):
    # CNN
    # x: (batch_size, channel, t, h, w)
    cnn_embed_seq = []
    if n_steps is None: # Code for training
        for t in range(X.size(2)):
            # Conv
            x = self.conv1(X[:, :, t, :, :])
            x = self.conv2(x)

            x = x.view(x.size(0), -1)
            cnn_embed_seq.append(x)

        cnn_embed_seq = torch.stack(cnn_embed_seq, dim=0)

        # batch first
        cnn_embed_seq = cnn_embed_seq.transpose_(0, 1)

        # LSTM
        hidden=None
        # use faster code paths
        self.lstm.flatten_parameters()
        out, hidden = self.lstm(cnn_embed_seq, hidden)
        # MLP
        # out: (batch, seq, feature), choose the last time step
        out = self.fc1(out[:, -1, :])
        out = self.act(out)

    if n_steps is not None: # Code for prediction
        for t in range(n_steps):
            # Conv
            x = self.conv1(X[:, :, 0, :, :])
            x = self.conv2(x)

            x = x.view(x.size(0), -1)
            # cnn_embed_seq.append(x)

            # cnn_embed_seq = torch.stack(cnn_embed_seq, dim=0)
            # batch first
            # x = x.transpose_(0, 1)
            # print('pred:',x.shape, t)

            x = x[ :, np.newaxis, :]
            # LSTM
            hidden=None
            # use faster code paths
            self.lstm.flatten_parameters()
            out, hidden = self.lstm(x, hidden)
            # print('LSTM out', out.shape)
            # MLP
            # out: (batch, seq, feature), choose the last time step
            out = self.fc1(out[:, -1, :])
            # X[:, 1, :, :, :] = out.view(-1, 1, self.sample_size,self.sample_size)
            out = self.act(out)
            X[:, 1, :, :, :] = out.view(-1, 1, self.sample_size,self.sample_size)

    return out


title: pytorch images to tensor of different size

body:
I tried to create my own dataset.
I have got images of different size in the folder
files = [fn for fn in os.listdir('images')
              if any(fn.endswith(ext) for ext in ['jpg'])]
data=[]
for f in files:
    data.append(np.asarray(Image.open('images/'+f)))

next, I got the labels :
labels_name = [fn for fn in os.listdir('images')
              if any(fn.endswith(ext) for ext in ['txt'])]
labels=[]
for label in labels_name:
    labels.append(np.loadtxt('images/'+str(label)))

Next step is to split data:
print(type(labels),len(labels),labels[x].shape)
print(type(data),len(data),data[x].shape)
#<class 'list'> 3385 (5,) --< yes 5 labels to each image
#<class 'list'> 3385 (436, 540, 3) <--W,H,C

X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, random_state=42)

So there is the question -

ValueError: expected sequence of length 256 at dim 1 (got 400)
error with resizing, but there is transform, isn't it?

How to choose the correct size? I have different image sizes, how do I choose the optimum value?

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Resize(256),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
train_dataset=torch.utils.data.TensorDataset(torch.tensor(X_train), torch.tensor(y_train),transform)
train_dataset=torch.utils.data.TensorDataset(torch.tensor(X_test), torch.tensor(y_test),transform)


title: the Training is stopped for my PyTorch model

body:
I am creating a model to identify names of items in RNN (LITMS)
i get data , then transform data then create Baches then create Model then create train function Correctly but the training is stop here(not working):

this this my code
        for e in range(epochs):

            # initialize hidden state
            h = net.init_hidden(batch_size)

            for x, y in get_batches(data, batch_size, seq_length):
                print ("the login the loob get_batches is succressfuly")
                counter += 1

                # One-hot encode our data and make them Torch tensors
                x = one_hot_encode(x, n_chars)
                inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

                if(train_on_gpu):
                    inputs, targets = inputs.cuda(), targets.cuda()

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                h = tuple([each.data for each in h])

                # zero accumulated gradients
                net.zero_grad()

                # get the output from the model
                output, h = net(inputs, h)

                # calculate the loss and perform backprop
                loss = criterion(output, targets.view(batch_size*seq_length))
                loss.backward()
                # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
                nn.utils.clip_grad_norm_(net.parameters(), clip)
                opt.step()

                # loss stats
                if counter % print_every == 0:
                    # Get validation loss
                    val_h = net.init_hidden(batch_size)
                    val_losses = []
                    net.eval()
                    for x, y in get_batches(val_data, batch_size, seq_length):
                        # One-hot encode our data and make them Torch tensors
                        x = one_hot_encode(x, n_chars)
                        x, y = torch.from_numpy(x), torch.from_numpy(y)

                        # Creating new variables for the hidden state, otherwise
                        # we'd backprop through the entire training history
                        val_h = tuple([each.data for each in val_h])

                        inputs, targets = x, y
                        if(train_on_gpu):
                            inputs, targets = inputs.cuda(), targets.cuda()

                        output, val_h = net(inputs, val_h)
                        val_loss = criterion(output, targets.view(batch_size*seq_length))

                        val_losses.append(val_loss.item())

                    net.train() # reset to train mode after iterationg through validation data

                    print("Epoch: {}/{}...".format(e+1, epochs),
                        "Step: {}...".format(counter),
                        "Loss: {:.4f}...".format(loss.item()),
                        "Val Loss: {:.4f}".format(np.mean(val_losses)))

I don't know why.
some times i got this error after many trials

please help me if you can
You can find the notebook file here.


title: How to handle variable length sentences in PyTorch with Glove embedding layer?

body:
I am building a text classifier using an RNN in PyTorch. The embeddings i'm using are GLOVE. However i am feeding variable length index references in to the model. This will lead to variable length embeddings, which i take it will not work. How do i get around this and make the embedding output the same length for all sentences?
def forward(self, sentence):
        embeds = self.embedding(sentence)
        hidden = self.__init__hidden(size)
        output, hidden = self.rnn(embeds, hidden)
        out = self.hidden2out(output)

Also, if someone could tell me how to choose the hidden layer size that would be great.


title: Errors rewriting a Tensorflow model in Pytorch

body:
I have model written in Tensorflow and I would like to convert it to Pytorch.
However I am having a lot of problems in converting because I am getting confused between functions, for instance, for Tensorflow BatchNormalization should I use BatchNorm1d or BatchNorm2?
The code in Tensorflow is this:
    input = Input(shape=(216, 1025, 1))
    norm_inp1 = BatchNormalization()(input)
    img_1 = Convolution2D(16, kernel_size=(3, 7), activation=activations.relu)(norm_inp1)
    img_1 = Convolution2D(16, kernel_size=(3, 7), activation=activations.relu)(img_1)
    img_1 = MaxPooling2D(pool_size=(3, 7))(img_1)
    img_1 = Dropout(rate=0.1)(img_1)
    img_1 = Convolution2D(32, kernel_size=3, activation=activations.relu)(img_1)
    img_1 = Convolution2D(32, kernel_size=3, activation=activations.relu)(img_1)
    img_1 = MaxPooling2D(pool_size=(3, 3))(img_1)
    img_1 = Dropout(rate=0.1)(img_1)
    img_1 = Convolution2D(128, kernel_size=3, activation=activations.relu)(img_1)
    img_1 = GlobalMaxPool2D()(img_1)
    img_1 = Dropout(rate=0.1)(img_1)

I have created a class that extends torch.nn.Module in which I apply this sequential:
features = torch.nn.Sequential(
            torch.nn.BatchNorm2d(1025),
            torch.nn.Conv1d(16, 1, kernel_size=(3, 7)),
            torch.nn.ReLU(inplace=True),
            torch.nn.Conv2d(16, 16, kernel_size=(3, 7)),
            torch.nn.ReLU(inplace=True),
            torch.nn.MaxPool2d(kernel_size=(3,7)),
            torch.nn.Dropout2d(0.1),
            torch.nn.Conv2d(16, 32, kernel_size=3),
            torch.nn.ReLU(inplace=True),
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(inplace=True),
            torch.nn.MaxPool2d(kernel_size=(3,3)),
            torch.nn.Dropout2d(0.1),
            torch.nn.Conv2d(32, 128, kernel_size=3),
            torch.nn.ReLU(inplace=True),
            torch.nn.MaxPool2d(kernel_size=(20,44)),
            torch.nn.Dropout2d(0.1)
        )

Could someone help me to understand the errors in the second one? I have perfectly understood the code in Tensorflow.
For complete network schema you can consult this link


title: convert Float32 array to image in coreml

body:
I converted a deelplab v3 model from Pytorch to coreml and seeing the outputs as MultiArray, so I need to convert the output to image. I've been using many different advice (e.g. from coreml survival guide) but still haven't been successful. Could you anyone kindly help me out here? Thanks a lot.
FYI, this is the model spec from Python:
    spec desc is
     input {
      name: "input.1"
      type {
           imageType {
            width: 513
            height: 513
           colorSpace: RGB
           }
          }
    }
    output {
     name: "1436"
     type {
          multiArrayType {
          dataType: FLOAT32
          }
     }
    }

In Xcode, the output the I printout in Xcode is "Float32 1 × 14 × 513 × 513 array", which I assume that 1 is the number of channels, 14 is the number of labels, and 513 x 514 is Width x Height. How can I convert this array into Int32 513 × 513 matrix or to an Image?
Thanks for your help!
Edited: I added my model output's structure compared with Apple's DeeplabV3's (visualized through Netron) below for your reference. Any guidance is appreciated!
1/ My model output's architecture
2/ Apple's DeeplabV3


title: How to find back the architecture of a pytorch model having only the weight dictionnary?

body:
I wanted to use the  multilingual-codesearch model but first the code doesn't work and outputs the following error which suggest that it cannot load with only weights:

    from transformers import AutoTokenizer, AutoModel

    tokenizer = AutoTokenizer.from_pretrained("ncoop57/multilingual-codesearch")

    model = AutoModel.from_pretrained("ncoop57/multilingual-codesearch")

ValueError: Unrecognized model in ncoop57/multilingual-codesearch. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: gpt_neo, big_bird, speech_to_text, vit, wav2vec2, m2m_100, convbert, led, blenderbot-small, retribert, ibert, mt5, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, mpnet, bart, blenderbot, reformer, longformer, roberta, deberta-v2, deberta, flaubert, fsmt, squeezebert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag, tapas

Then I downloaded the pytorch bin file but it only contains the weight dictionnary (state dictionnary as mentioned here), which means that if I want to use the model I have to initialize the good architecture and then load the weights.

But how am I supposed to find the architecture fitting the weight of a model that complex ? I saw that some method could find back the model based on the weight dictionnary but I didn't manage to make them work (I think about enter link description here).
How can one find back the architecture of a weight dictionnary in order to make the model work ? Is it even possible ?