title: Error while applying image augmentation transformations to data in FastAI

body:
I am trying to replicate this kaggle notebook https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-with-resnet50-oversampling on Google Colab. The code was working fine till yesterday but today it is throwing a runtime error. Below is the problematic code:

tfms = get_transforms(do_flip=True,flip_vert=True,max_rotate=360,max_warp=0,max_zoom=1.1,max_lighting=0.1,p_lighting=0.5)
src = (ImageList.from_df(df=df,path=data_path,cols='path') #get dataset from dataset //ImageItemList threw errors so changed to ImageList
        .split_by_idx(range(len(train_df)-1,len(df))) #Splitting the dataset
        .label_from_df(cols='level') #obtain labels from the level column
      )
data= (src.transform(tfms,size=sz) #Data augmentation
        .databunch(bs=bs,num_workers=0) #DataBunch
        .normalize(imagenet_stats) #Normalize
       )

I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in _check_kwargs(ds, tfms, **kwargs)
    593         x = ds[0]
--> 594         try: x.apply_tfms(tfms, **kwargs)
    595         except Exception as e:

8 frames
/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in apply_tfms(self, tfms, do_resolve, xtra, size, resize_method, mult, padding_mode, mode, remove_out)
    122                     x = tfm(x, size=_get_crop_target(size,mult=mult), padding_mode=padding_mode)
--> 123             else: x = tfm(x)
    124         return x.refresh()

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in __call__(self, x, *args, **kwargs)
    523         "Randomly execute our tfm on `x`."
--> 524         return self.tfm(x, *args, **{**self.resolved, **kwargs}) if self.do_run else x
    525

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in __call__(self, p, is_random, use_on_y, *args, **kwargs)
    469         "Calc now if `args` passed; else create a transform called prob `p` if `random`."
--> 470         if args: return self.calc(*args, **kwargs)
    471         else: return RandTransform(self, kwargs=kwargs, is_random=is_random, use_on_y=use_on_y, p=p)

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in calc(self, x, *args, **kwargs)
    474         "Apply to image `x`, wrapping it if necessary."
--> 475         if self._wrap: return getattr(x, self._wrap)(self.func, *args, **kwargs)
    476         else:          return self.func(x, *args, **kwargs)

/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in affine(self, func, *args, **kwargs)
    182         m = tensor(func(*args, **kwargs)).to(self.device)
--> 183         self.affine_mat = self.affine_mat @ m
    184         return self

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
<ipython-input-74-31aae73a70fc> in <module>()
      6       )
      7 print(src)
----> 8 data= (src.transform(tfms,size=sz) #Data augmentation
      9         .databunch(bs=bs,num_workers=0) #DataBunch
     10         .normalize(imagenet_stats) #Normalize

/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in transform(self, tfms, **kwargs)
    503         if not tfms: tfms=(None,None)
    504         assert is_listy(tfms) and len(tfms) == 2, "Please pass a list of two lists of transforms (train and valid)."
--> 505         self.train.transform(tfms[0], **kwargs)
    506         self.valid.transform(tfms[1], **kwargs)
    507         if self.test: self.test.transform(tfms[1], **kwargs)

/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in transform(self, tfms, tfm_y, **kwargs)
    722     def transform(self, tfms:TfmList, tfm_y:bool=None, **kwargs):
    723         "Set the `tfms` and `tfm_y` value to be applied to the inputs and targets."
--> 724         _check_kwargs(self.x, tfms, **kwargs)
    725         if tfm_y is None: tfm_y = self.tfm_y
    726         tfms_y = None if tfms is None else list(filter(lambda t: getattr(t, 'use_on_y', True), listify(tfms)))

/usr/local/lib/python3.6/dist-packages/fastai/data_block.py in _check_kwargs(ds, tfms, **kwargs)
    594         try: x.apply_tfms(tfms, **kwargs)
    595         except Exception as e:
--> 596             raise Exception(f"It's not possible to apply those transforms to your dataset:\n {e}")
    597
    598 class LabelList(Dataset):

Exception: It's not possible to apply those transforms to your dataset:
 Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out

I changed nothing in this code, it is the same as it was yesterday but for some reason it gives me an error today. Kindly Help.

Edit: I found out it is working perfectly fine on my local Jupyter notebook. Still shows error for Colab though


title: How to save weights in .npy file in a similar format which is being used in CNN?

body:
I am using a github repository containing a trained CNN with weight parameters given in .npy file. Model is loading the weights and using the model parameters like this:-

model = CNN_Model(batch_size)
filename = "weight_file.npy"
dtype = torch.FloatTensor
model.load_state_dict(load_weights(model, weight_file, dtype))

And load_weights is defined as:-

def load_weights(model, filename, dtype):
    model_params = model.state_dict()
    data_dict = np.load(filename, encoding='latin1').item()
    model_params["conv1.weight"] = torch.from_numpy(data_dict["conv1"] ["weights"]).type(dtype).permute(3,2,0,1)
    model_params["conv1.bias"] = torch.from_numpy(data_dict["conv1"]["biases"]).type(dtype)
    model_params["bn1.weight"] = torch.from_numpy(data_dict["bn_conv1"]["scale"]).type(dtype)
    model_params["bn1.bias"] = torch.from_numpy(data_dict["bn_conv1"]["offset"]).type(dtype)
    return model_params

I have added a training module to it and trying to fine tune the weights  on my own dataset. After training i want to save new weights in .npy file with same indexes of data_dict as there were in previously loaded weight file so i can use them again for CNN model.

How should i do indexing with similar names before saving the data_dict array using:

np.save("trained_weight_file.npy", data_dict)

EDIT 1:-
So on recommendation of @a-d i did

data_dict = model.state_dict()

What it did is it saved all the weights with index of model_params. Output of print data_dict was:-

OrderedDict([('conv1.weight', tensor([[[[....]]]])), ('conv1.bias', tensor([....])), , ('bn1.weight', tensor([....])), ('bn1.bias', tensor([....]))])

But what i need is to store in data_dict index so i can read it with same algo from .npy file. Also i tried returning data_dict along with model_params from load_weights definition and then tried to use data_dict = model.state_dict() but it gave me error on `model.load_state_dict(load_weights(model, weight_file, dtype))' line that is:-

Traceback (most recent call last):
  model.load_state_dict(load_weights(model, weight_file, dtype))
  state_dict = state_dict.copy()
AttributeError: 'tuple' object has no attribute 'copy'


title: How to make a dataset from video datasets(tensorflow first)

body:
everyone.

Now I have an object classification task, and I have a dataset containing a large number of videos. In every video, some frames(not every frame, about 160 thousand frames) have its labels, since a frame may have multiple objects.

I have some confusion about creating the dataset. My idea is to convert videos to frames firstly, then every frame only with labels will be made as tfrecord or hdf5 format. Finally, I would write every frame's path into csv files (training and validation) using for my task.

My question is :
1. Is there efficient enough(tfrecord or hdf5)? Should I preprocess every frame such as compression for save the storage space before creating tfrecord or hdf5 files?
2. Is there a way to handle the video dataset directly in tensorflow or pytorch?

I want to find an efficient and conventional way to handle video datasets. Really looking forward to every answer.


title: Pytorch list of supported GPU hardware for each release (Ubuntu18.04)

body:
I have an old GPU and pytorch says it is too old to support:

    Found GPU0 GeForce GTX 670 which is of cuda capability 3.0.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability that we support is 3.5.

First question is what is this 3.0 / 3.5 referring to ? Clearly not cuda, nvidia driver nor pytorch?
Secondly, I know I can build Pytorch from source in order to have more support, however it's unclear what hardware pytorch supports when built from source, or even when installed with pip. I'm at the point where the likely pytorch version I need, requires python2, and cuda 9, so I'm spending solid time changing a bunch of software versions without knowing whether any of it will work.

Also if I were to get a newer GPU, I don't know whether it is currently supported.

Any way to have a list of supported NVIDIA hardware of pytorch? Ideally for each releases, but at least for current release?

I run an Nvidia GTX670 with Nvidia Driver 430.50. I changed from cuda 10 to 9, but seeing I might have to build pytorch 0.3.1, I think cuda8 is needed since when runnin python setup.py install in v0.3.1 i get:

CMakeFiles/THC.dir/build.make:560: recipe for target 'CMakeFiles/THC.dir/THC_generated_THCHalf.cu.o' failed
make[2]: *** [CMakeFiles/THC.dir/THC_generated_THCHalf.cu.o] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/THC.dir/all' failed
make[1]: *** [CMakeFiles/THC.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2


title: How to get weights for custom loss function in pytorch?

body:
I have a model in pytorch and would like to add L1 regularization insde the loss_function. But I don't want to pass the weights to the loss_function() - is there a better way of doing this? For details see the loss_function() below.

class AutoEncoder(nn.Module):
    def __init__(self, inp_size, hid_size):
        super(AutoEncoder, self).__init__(

        self.lambd = 1.

        # Encoder
        self.e1 = nn.Linear(inp_size, hid_size)

        # Decoder
        self.d1 = nn.Linear(hid_size, inp_size)
        self.sigmoid = nn.Sigmoid()

        pass

   def forward(self,x):
       encode = self.e1(x)
       decode = self.sigmoid(self.d1(encode))
       return decode

   def loss_function(self, recon_x, x):
       l2_loss = nn.MSELoss()

       # Here I would like to compute the L1 regularization of the weight parameters
       loss = l2_loss(recon_x, x) + self.lambd(l1_loss(self.e1) + l1_loss(self.e2))
       return loss


title: Partitioned matrix multiplication in tensorflow or pytorch

body:
Assume I have matrices P with the size [4, 4] which partitioned (block) into 4 smaller matrices [2,2]. How can I efficiently multiply this block-matrix into another matrix (not partitioned matrix but smaller)?

Let's Assume our original matric is:

P = [ 1 1 2 2
      1 1 2 2
      3 3 4 4
      3 3 4 4]

Which split into submatrices:

P_1 = [1 1    , P_2 = [2 2  , P_3 = [3 3   P_4 = [4 4
       1 1]            2 2]          3 3]         4 4]

Now our P is:

P = [P_1 P_2
     P_3 p_4]

In the next step, I want to do element-wise multiplication between P and smaller matrices which its size is equal to number of sub-matrices:

P * [ 1 0   =   [P_1  0  = [1 1 0 0
      0 0 ]      0    0]    1 1 0 0
                            0 0 0 0
                            0 0 0 0]


title: Resized copy of Pytorch Tensor/Dataset

body:
I have a homemade dataset with a few million rows.   I am trying to make truncated copies.  So I clip the tensors that I'm using to make the original dataset and create a new dataset.   However, when I save the new dataset, which is only 20K rows, it's the same size on disk as the original dataset.  Otherwise everything seems kosher, including, when I check, the size of the new tensors.   What am I doing wrong?

#original dataset - 2+million rows
dataset = D.TensorDataset(training_data, labels)
torch.save(dataset, filename)

#20k dataset for experiments
d = torch.Tensor(training_data[0:20000])
l = torch.Tensor(labels[0:20000])
ds_small = D.TensorDataset(d,l)
#this is the same size as the one above on disk... approx 1.45GB
torch.save(ds_small, filename_small)

Thanks


title: Could not find a version that satisfies the requirement torch~=1.4.0 (from syft) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)

body:
I get the above error while installing syft package through Anaconda command. I followed the below link exactly as is,
https://medium.com/secure-and-private-ai-writing-challenge/installing-pysyft-package-ffa1ff0ad83c

Following commands were used:

conda create -n pysyft python=3
conda activate pysyft
pip install syft

Some of the links suggested to update the Pytorch version. I have 1.5.0+cpu version already and updating didnt help. I also tried using pip install syft without creating a conda environment, that didnt solve the problem either.
I went through the below stackoverflow link too which explains similar error, but it didnt help either:
Issues installing PyTorch 1.4 - "No matching distribution found for torch===1.4.0"

Any advice? Thanks in advance


title: Index pytorch 4d tensor by values in 2d tensor

body:
I have two pytorch tensors:

X with shape (A, B, C, D)
I with shape (A, B)

Values in I are integers in range [0, C).

What is the most efficient way to get tensor Y with shape (A, B, D), such that:

Y[i][j][k] = X[i][j][ I[i][j] ][k]


title: Convert pytorch tensor to numpy, and reshape

body: I have a pytorch tensor [100, 1, 32, 32] corresponding to batch size of 100 images, 1 channel, height 32 and width 32. I want to reshape this tensor to have dimension [32*10, 32*10], such that the images are represented as a 10x10 grid, with the first 10 images on row 1, and so on. How to achieve this?


title: How to apply OpenCV filters on Pytorch dataset?

body:
I have used below steps to pre-process single image using OpenCV. Now, I want to apply these pre-processing steps to my entire dataset before training the model in Pytorch. How can this be done?

im = cv2.imread(image_path)
im_nonoise = cv2.medianBlur(im, 3)
imgray = cv2.cvtColor(im_nonoise,cv2.COLOR_BGR2GRAY)
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
cl1 = clahe.apply(imgray)
ret,thresh = cv2.threshold(cl1,110,255,0)
image, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
img = cv2.drawContours(image, contours, -1, (250,100,120))

And I load the data using

data = datasets.ImageFolder(train_dir,transform=transform)
train_loader = torch.utils.data.DataLoader(data,batch_size=batch_size,sampler=train_sampler)


title: Installed pytorch through conda, but cannot import in Windows 10

body:
When I issue an Anaconda prompt conda search pytorch
then I get pytorch installed even issuing conda list command gives me:
pytorch                   1.5.1           py3.7_cuda102_cudnn7_0    pytorch

But when I start python on command prompt and then issue import pytorch i get ModuleNotFoundError: No module named 'pytorch'.
Even I tried to do the same after issuing
conda create -n pytorch_env -c pytorch pytorch torchvision
conda activate pytorch_env
conda install -c pytorch pytorch torchvision

as written in Installing PyTorch via Conda but of no use. BTW does it require a restart of the machine after installing Anaconda? Also, let me know how I can use PyTorch in Jupyter notebooks.


title: Adding xavier initiliazation in pytorch

body:
I want to add Xavier initialization to the first layer of my Neural Network, but I am getting an error in this class:
class DemoNN(nn.Module):
    def __init__(self):
        super().__init__()
        torch.manual_seed(0)
        self.net = nn.Sequential(
            nn.Linear(2,2),
            torch.nn.init.xavier_uniform((nn.Linear(2,2)).weights),
            nn.Sigmoid(),

            nn.Linear(2,2),
            nn.Sigmoid(),

            nn.Linear(2,4),
            nn.Softmax()
            )

    def forward(self, X):
        self.net(X)


title: PyTorch RNN is more efficient with `batch_first=False`?

body:
In machine translation, we always need to slice out the first timestep (the SOS token) in the annotation and prediction.
When using batch_first=False, slicing out the first timestep still keeps the tensor contiguous.
import torch
batch_size = 128
seq_len = 12
embedding = 50

# Making a dummy output that is `batch_first=False`
batch_not_first = torch.randn((seq_len,batch_size,embedding))
batch_not_first = batch_first[1:].view(-1, embedding) # slicing out the first time step

However, if we use batch_first=True, after slicing, the tensor is no longer contiguous. We need to make it contiguous before we can do different operations such as view.
batch_first = torch.randn((batch_size,seq_len,embedding))
batch_first[:,1:].view(-1, embedding) # slicing out the first time step

output>>>
"""
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-a9bd590a1679> in <module>
----> 1 batch_first[:,1:].view(-1, embedding) # slicing out the first time step

RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
"""

Does that mean batch_first=False is better, at least, in the context of machine translation? Since it saves us from doing the contiguous() step. Is there any cases that batch_first=True works better?


title: Out of memory error during evaluation but training works fine

body:
I have recently upgraded PyTorch from 0.2 to 0.3. Surprisingly my old programs are throwing an out of memory error during evaluation (in eval() mode) but training works just fine. I am using the same batch size for training and evaluation. I am totally clueless what is happening? Did anyone face similar issue? Is there any possible solution?

I tried using volatile=True param on the variables and it didn't help. Please note, I am not doing anything special to use cuDNN. I am using the default setting.

def validate(self, dev_corpus):
    # Turn on evaluation mode which disables dropout.
    self.model.eval()

    dev_batches = helper.batchify(dev_corpus.data, self.config.batch_size)
    print('number of dev batches = ', len(dev_batches))

    dev_loss = 0
    num_batches = len(dev_batches)
    for batch_no in range(1, num_batches + 1):
        session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels = helper.session_to_tensor(
            dev_batches[batch_no - 1], self.dictionary)
        if self.config.cuda:
            session_queries = session_queries.cuda()
            session_query_length = session_query_length.cuda()
            rel_docs = rel_docs.cuda()
            rel_docs_length = rel_docs_length.cuda()
            doc_labels = doc_labels.cuda()

        loss = self.model(session_queries, session_query_length, rel_docs, rel_docs_length, doc_labels)
        if loss.size(0) > 1:
            loss = loss.mean()
        dev_loss += loss.data[0]

    return dev_loss / num_batches

I am using the above function for evaluation. Here, session_queries, session_query_length, .... rest variables are created by enabling volatile=True.

Please help!!


title: Pytorch:Apply cross entropy loss with custom weight map

body:
I am solving multi-class segmentation problem using u-net architecture in pytorch.
As specified in U-NET paper, I am trying to implement custom weight maps to counter class imbalances.

Below is the opertion which I want to apply -

Also, I reduced the batch_size=1 so that I can remove that dimension while passing it to precompute_to_masks function.
I tried the below approach-

def precompute_for_image(masks):
    masks = masks.cpu()
    cls = masks.unique()
    res = torch.stack([torch.where(masks==cls_val, torch.tensor(1), torch.tensor(0)) for cls_val in cls])
    return res

def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):

        ###################
        # train the model #
        ###################
        model.train()
        for batch_idx, (data, target) in enumerate(final_train_loader):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            optimizer.zero_grad()
            output = model(data)
            temp_target = precompute_for_image(target)
            w = weight_map(temp_target)
            loss = criterion(output,target)
            loss = w*loss
            loss.backward()
            optimizer.step()

            train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

    return model

where weight_map is the function to calculate weight mask which I got from here
The issue, I am facing is I am getting memory error when I apply the following method. I am using 61gb RAM and Tesla V100 GPU.
I really think I am applying it in incorrect way.
How to do it?
I am omitting the non-essential details from the training loop.
Below is my weight_map function:

from skimage.segmentation import find_boundaries

w0 = 10
sigma = 5

def make_weight_map(masks):
    """
    Generate the weight maps as specified in the UNet paper
    for a set of binary masks.

    Parameters
    ----------
    masks: array-like
        A 3D array of shape (n_masks, image_height, image_width),
        where each slice of the matrix along the 0th axis represents one binary mask.

    Returns
    -------
    array-like
        A 2D array of shape (image_height, image_width)

    """
    nrows, ncols = masks.shape[1:]
    masks = (masks > 0).astype(int)
    distMap = np.zeros((nrows * ncols, masks.shape[0]))
    X1, Y1 = np.meshgrid(np.arange(nrows), np.arange(ncols))
    X1, Y1 = np.c_[X1.ravel(), Y1.ravel()].T
    for i, mask in enumerate(masks):
        # find the boundary of each mask,
        # compute the distance of each pixel from this boundary
        bounds = find_boundaries(mask, mode='inner')
        X2, Y2 = np.nonzero(bounds)
        xSum = (X2.reshape(-1, 1) - X1.reshape(1, -1)) ** 2
        ySum = (Y2.reshape(-1, 1) - Y1.reshape(1, -1)) ** 2
        distMap[:, i] = np.sqrt(xSum + ySum).min(axis=0)
    ix = np.arange(distMap.shape[0])
    if distMap.shape[1] == 1:
        d1 = distMap.ravel()
        border_loss_map = w0 * np.exp((-1 * (d1) ** 2) / (2 * (sigma ** 2)))
    else:
        if distMap.shape[1] == 2:
            d1_ix, d2_ix = np.argpartition(distMap, 1, axis=1)[:, :2].T
        else:
            d1_ix, d2_ix = np.argpartition(distMap, 2, axis=1)[:, :2].T
        d1 = distMap[ix, d1_ix]
        d2 = distMap[ix, d2_ix]
        border_loss_map = w0 * np.exp((-1 * (d1 + d2) ** 2) / (2 * (sigma ** 2)))
    xBLoss = np.zeros((nrows, ncols))
    xBLoss[X1, Y1] = border_loss_map
    # class weight map
    loss = np.zeros((nrows, ncols))
    w_1 = 1 - masks.sum() / loss.size
    w_0 = 1 - w_1
    loss[masks.sum(0) == 1] = w_1
    loss[masks.sum(0) == 0] = w_0
    ZZ = xBLoss + loss
    return ZZ

Traceback of the error-

MemoryError                               Traceback (most recent call last)
<ipython-input-30-f0a595b8de7e> in <module>
      1 # train the model
      2 model_scratch = train(20, final_train_loader, unet, optimizer,
----> 3                       criterion, train_on_gpu, 'model_scratch.pt')

<ipython-input-29-b481b4f3120e> in train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path)
     24             loss = criterion(output,target)
     25             target.requires_grad = False
---> 26             w = make_weight_map(target)
     27             loss = W*loss
     28             loss.backward()

<ipython-input-5-e75a6281476f> in make_weight_map(masks)
     33         X2, Y2 = np.nonzero(bounds)
     34         xSum = (X2.reshape(-1, 1) - X1.reshape(1, -1)) ** 2
---> 35         ySum = (Y2.reshape(-1, 1) - Y1.reshape(1, -1)) ** 2
     36         distMap[:, i] = np.sqrt(xSum + ySum).min(axis=0)
     37     ix = np.arange(distMap.shape[0])

MemoryError:


title: problem with CUDA in the learning process

body:
In the learning process, I get an cuda error:

 CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

I have not encountered such a problem before. In google I did not find a solution that could help me.
Perhaps this is somehow connected with the fact that I have little gpu memory(2gb)

Code of model:

num_epochs = 2
batch_size = 4

for epoch in range(num_epochs):
    # In each epoch, we do a full pass over the training data:
    start_time = time.time()
    model.train(True) # enable dropout / batch_norm training behavior
    for (X_batch, y_batch) in train_batch_gen:
        loss = compute_loss(X_batch, y_batch)
        loss.backward()
        opt.step()
        opt.zero_grad()
        train_loss.append(loss.data.cpu().numpy())
    model.train(False) # disable dropout / use averages for batch_norm
    for X_batch, y_batch in val_batch_gen:
        logits = model(Variable(torch.FloatTensor(X_batch)).cuda())
        y_pred = logits.max(1)[1].data
        val_accuracy.append(np.mean( (y_batch.cpu() == y_pred.cpu()).numpy() ))

Traceback:

RuntimeError                              Traceback (most recent call last)
 in
     11     for (X_batch, y_batch) in train_batch_gen:
     12         # train on batch
---> 13         loss = compute_loss(X_batch, y_batch)
     14         loss.backward()
     15         opt.step()

 in compute_loss(X_batch, y_batch)
      2     X_batch = Variable(torch.FloatTensor(X_batch)).cuda()
      3     y_batch = Variable(torch.LongTensor(y_batch)).cuda()
----> 4     logits = model.cuda()(X_batch)
      5     return F.cross_entropy(logits, y_batch).mean()

c:\program files\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\program files\python38\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102

c:\program files\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\program files\python38\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

c:\program files\python38\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1608     if input.dim() == 2 and bias is not None:
   1609         # fused op is marginally faster
-> 1610         ret = torch.addmm(bias, input, weight.t())
   1611     else:
   1612         output = input.matmul(weight.t())

RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`


title: RNN model (GRU) of word2vec to regression not learning

body:
I am converting Keras code into PyTorch because I am more familiar with the latter than the former. However, I found that it is not learning (or only barely).

Below I have provided almost all of my PyTorch code, including the initialisation code so that you can try it out yourself. The only thing you would need to provide yourself, is the word embeddings (I'm sure you can find many word2vec models online). The first input file should be a file with tokenised text, the second input file should be a file with floating-point numbers, one per line. Because I have provided all the code, this question may seem huge and too broad. However, my question is specific enough I think: what is wrong in my model or training loop that causes my model to not or barely improve. (See below for results.)

I have tried to provide many comments where applicable, and I have provided the shape transformations as well so you do not have to run the code to see what is going on. The data prep methods are not important to inspect.

The most important parts are the forward method of the RegressorNet, and the training loop of RegressionNN (admittedly, these names were badly chosen). I think the mistake is there somewhere.

from pathlib import Path
import time

import numpy as np
import torch
from torch import nn, optim
from torch.utils.data import DataLoader
import gensim

from scipy.stats import pearsonr

from LazyTextDataset import LazyTextDataset

class RegressorNet(nn.Module):
    def __init__(self, hidden_dim, embeddings=None, drop_prob=0.0):
        super(RegressorNet, self).__init__()
        self.hidden_dim = hidden_dim
        self.drop_prob = drop_prob

        # Load pretrained w2v model, but freeze it: don't retrain it.
        self.word_embeddings = nn.Embedding.from_pretrained(embeddings)
        self.word_embeddings.weight.requires_grad = False
        self.w2v_rnode = nn.GRU(embeddings.size(1), hidden_dim, bidirectional=True, dropout=drop_prob)

        self.dropout = nn.Dropout(drop_prob)
        self.linear = nn.Linear(hidden_dim * 2, 1)
        # LeakyReLU rather than ReLU so that we don't get stuck in a dead nodes
        self.lrelu = nn.LeakyReLU()

    def forward(self, batch_size, sentence_input):
        # shape sizes for:
        # * batch_size 128
        # * embeddings of dim 146
        # * hidden dim of 200
        # * sentence length of 20

        # sentence_input: torch.Size([128, 20])
        # Get word2vec vector representation
        embeds = self.word_embeddings(sentence_input)
        # embeds: torch.Size([128, 20, 146])

        # embeds.view(-1, batch_size, embeds.size(2)): torch.Size([20, 128, 146])
        # Input vectors into GRU, only keep track of output
        w2v_out, _ = self.w2v_rnode(embeds.view(-1, batch_size, embeds.size(2)))
        # w2v_out = torch.Size([20, 128, 400])

        # Leaky ReLU it
        w2v_out = self.lrelu(w2v_out)

        # Dropout some nodes
        if self.drop_prob > 0:
            w2v_out = self.dropout(w2v_out)
        # w2v_out: torch.Size([20, 128, 400

        # w2v_out[-1, :, :]: torch.Size([128, 400])
        # Only use the last output of a sequence! Supposedly that cell outputs the final information
        regression = self.linear(w2v_out[-1, :, :])
        regression: torch.Size([128, 1])

        return regression

class RegressionRNN:
    def __init__(self, train_files=None, test_files=None, dev_files=None):
        print('Using torch ' + torch.__version__)

        self.datasets, self.dataloaders = RegressionRNN._set_data_loaders(train_files, test_files, dev_files)
        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

        self.model = self.w2v_vocab = self.criterion = self.optimizer = self.scheduler = None

    @staticmethod
    def _set_data_loaders(train_files, test_files, dev_files):
        # labels must be the last input file
        datasets = {
            'train': LazyTextDataset(train_files) if train_files is not None else None,
            'test': LazyTextDataset(test_files) if test_files is not None else None,
            'valid': LazyTextDataset(dev_files) if dev_files is not None else None
        }
        dataloaders = {
            'train': DataLoader(datasets['train'], batch_size=128, shuffle=True, num_workers=4) if train_files is not None else None,
            'test': DataLoader(datasets['test'], batch_size=128, num_workers=4) if test_files is not None else None,
            'valid': DataLoader(datasets['valid'], batch_size=128, num_workers=4) if dev_files is not None else None
        }

        return datasets, dataloaders

    @staticmethod
    def prepare_lines(data, split_on=None, cast_to=None, min_size=None, pad_str=None, max_size=None, to_numpy=False,
                      list_internal=False):
        """ Converts the string input (line) to an applicable format. """
        out = []
        for line in data:
            line = line.strip()
            if split_on:
                line = line.split(split_on)
                line = list(filter(None, line))
            else:
                line = [line]

            if cast_to is not None:
                line = [cast_to(l) for l in line]

            if min_size is not None and len(line) < min_size:
                # pad line up to a number of tokens
                line += (min_size - len(line)) * ['@pad@']
            elif max_size and len(line) > max_size:
                line = line[:max_size]

            if list_internal:
                line = [[item] for item in line]

            if to_numpy:
                line = np.array(line)

            out.append(line)

        if to_numpy:
            out = np.array(out)

        return out

    def prepare_w2v(self, data):
        idxs = []
        for seq in data:
            tok_idxs = []
            for word in seq:
                # For every word, get its index in the w2v model.
                # If it doesn't exist, use @unk@ (available in the model).
                try:
                    tok_idxs.append(self.w2v_vocab[word].index)
                except KeyError:
                    tok_idxs.append(self.w2v_vocab['@unk@'].index)
            idxs.append(tok_idxs)
        idxs = torch.tensor(idxs, dtype=torch.long)

        return idxs

    def train(self, epochs=10):
        valid_loss_min = np.Inf
        train_losses, valid_losses = [], []
        for epoch in range(1, epochs + 1):
            epoch_start = time.time()

            train_loss, train_results = self._train_valid('train')
            valid_loss, valid_results = self._train_valid('valid')

            # Calculate Pearson correlation between prediction and target
            try:
                train_pearson = pearsonr(train_results['predictions'], train_results['targets'])
            except FloatingPointError:
                train_pearson = "Could not calculate Pearsonr"

            try:
                valid_pearson = pearsonr(valid_results['predictions'], valid_results['targets'])
            except FloatingPointError:
                valid_pearson = "Could not calculate Pearsonr"

            # calculate average losses
            train_loss = np.mean(train_loss)
            valid_loss = np.mean(valid_loss)

            train_losses.append(train_loss)
            valid_losses.append(valid_loss)

            # print training/validation statistics
            print(f'----------\n'
                  f'Epoch {epoch} - completed in {(time.time() - epoch_start):.0f} seconds\n'
                  f'Training Loss: {train_loss:.6f}\t Pearson: {train_pearson}\n'
                  f'Validation loss: {valid_loss:.6f}\t Pearson: {valid_pearson}')

            # validation loss has decreased
            if valid_loss <= valid_loss_min and train_loss > valid_loss:
                print(f'!! Validation loss decreased ({valid_loss_min:.6f} --> {valid_loss:.6f}).  Saving model ...')
                valid_loss_min = valid_loss

            if train_loss <= valid_loss:
                print('!! Training loss is lte validation loss. Might be overfitting!')

            # Optimise with scheduler
            if self.scheduler is not None:
                self.scheduler.step(valid_loss)

        print('Done training...')

    def _train_valid(self, do):
        """ Do training or validating. """
        if do not in ('train', 'valid'):
            raise ValueError("Use 'train' or 'valid' for 'do'.")

        results = {'predictions': np.array([]), 'targets': np.array([])}
        losses = np.array([])

        self.model = self.model.to(self.device)
        if do == 'train':
            self.model.train()
            torch.set_grad_enabled(True)
        else:
            self.model.eval()
            torch.set_grad_enabled(False)

        for batch_idx, data in enumerate(self.dataloaders[do], 1):
            # 1. Data prep
            sentence = data[0]
            target = data[-1]
            curr_batch_size = target.size(0)

            # Returns list of tokens, possibly padded @pad@
            sentence = self.prepare_lines(sentence, split_on=' ', min_size=20, max_size=20)
            # Converts tokens into w2v IDs as a Tensor
            sent_w2v_idxs = self.prepare_w2v(sentence)
            # Converts output to Tensor of floats
            target = torch.Tensor(self.prepare_lines(target, cast_to=float))

            # Move input to device
            sent_w2v_idxs, target = sent_w2v_idxs.to(self.device), target.to(self.device)

            # 2. Predictions
            pred = self.model(curr_batch_size, sentence_input=sent_w2v_idxs)
            loss = self.criterion(pred, target)

            # 3. Optimise during training
            if do == 'train':
                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

            # 4. Save results
            pred = pred.detach().cpu().numpy()
            target = target.cpu().numpy()

            results['predictions'] = np.append(results['predictions'], pred, axis=None)
            results['targets'] = np.append(results['targets'], target, axis=None)
            losses = np.append(losses, float(loss))

        torch.set_grad_enabled(True)

        return losses, results

if __name__ == '__main__':
    HIDDEN_DIM = 200

    # Load embeddings from pretrained gensim model
    embed_p = Path('path-to.w2v_model').resolve()
    w2v_model = gensim.models.KeyedVectors.load_word2vec_format(str(embed_p))
    # add a padding token with only zeros
    w2v_model.add(['@pad@'], [np.zeros(w2v_model.vectors.shape[1])])
    embed_weights = torch.FloatTensor(w2v_model.vectors)

    # Text files are used as input. Every line is one datapoint.
    # *.tok.low.*: tokenized (space-separated) sentences
    # *.cross: one floating point number per line, which we are trying to predict
    regr = RegressionRNN(train_files=(r'train.tok.low.en',
                                      r'train.cross'),
                         dev_files=(r'dev.tok.low.en',
                                    r'dev.cross'),
                         test_files=(r'test.tok.low.en',
                                     r'test.cross'))
    regr.w2v_vocab = w2v_model.vocab
    regr.model = RegressorNet(HIDDEN_DIM, embed_weights, drop_prob=0.2)
    regr.criterion = nn.MSELoss()
    regr.optimizer = optim.Adam(list(regr.model.parameters())[0:], lr=0.001)
    regr.scheduler = optim.lr_scheduler.ReduceLROnPlateau(regr.optimizer, 'min', factor=0.1, patience=5, verbose=True)

    regr.train(epochs=100)

For the LazyTextDataset, you can refer to the class below.

from torch.utils.data import Dataset

import linecache

class LazyTextDataset(Dataset):
    def __init__(self, paths):
        # labels are in the last path
        self.paths, self.labels_path = paths[:-1], paths[-1]

        with open(self.labels_path, encoding='utf-8') as fhin:
            lines = 0
            for line in fhin:
                if line.strip() != '':
                    lines += 1

            self.num_entries = lines

    def __getitem__(self, idx):
        data = [linecache.getline(p, idx + 1) for p in self.paths]
        label = linecache.getline(self.labels_path, idx + 1)

        return (*data, label)

    def __len__(self):
        return self.num_entries

As I wrote before, I am trying to convert a Keras model to PyTorch. The original Keras code does not use an embedding layer, and uses pre-built word2vec vectors per sentence as input. In the model below, there is no embedding layer. The Keras summary looks like this (I don't have access to the base model setup).

Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
bidirectional_1 (Bidirectional)  (200, 400)            417600
____________________________________________________________________________________________________
dropout_1 (Dropout)              (200, 800)            0           merge_1[0][0]
____________________________________________________________________________________________________
dense_1 (Dense)                  (200, 1)              801         dropout_1[0][0]
====================================================================================================

The issue is that with identical input, the Keras model works and gets a +0.5 Pearson correlation between predicted and actual labels. The PyTorch model above, though, does not seem to work at all. To give you an idea, here is the loss (mean squared error) and Pearson (correlation coefficient, p-value) after the first epoch:

Epoch 1 - completed in 11 seconds
Training Loss: 1.684495  Pearson: (-0.0006077809280690612, 0.8173368901481127)
Validation loss: 1.708228    Pearson: (0.017794288315261794, 0.4264098054188664)

And after the 100th epoch:

Epoch 100 - completed in 11 seconds
Training Loss: 1.660194  Pearson: (0.0020315421756790806, 0.4400929436716754)
Validation loss: 1.704910    Pearson: (-0.017288118524826892, 0.4396865964324158)

The loss is plotted below (when you look at the Y-axis, you can see the improvements are minimal).

A final indicator that something may be wrong, is that for my 140K lines of input, each epoch only takes 10 seconds on my GTX 1080TI. I feel that his is not much and I would guess that the optimisation is not working/running. I cannot figure out why, though. To issue will probably be in my train loop or the model itself, but I cannot find it.

Again, something must be going wrong because:
  - the Keras model does perform well;
  - the training speed is 'too fast' for 140K sentences
  - almost no improvemnts after training.

What am I missing? The issue is more than likely present in the training loop or in the network structure.


title: How to apply image transform to a list of images and maintain the right dimensions?

body:
I'm using the Omniglot dataset, which is a set of 19,280 images, each which is 105 x 105 (grayscale).

I defined a custom Dataset class with the following transform:

class OmniglotDataset(Dataset):

    def __init__(self, X, transform=None):
        self.X = X
        self.transform = transform

    def __len__(self):
        return self.X.shape[0]

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        img = self.X[idx]
        if self.transform:
            img = self.transform(img)
        return img

img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

X_train.shape
(19280, 105, 105)
train_dataset = OmniglotDataset(X_train, transform=img_transform)

When I index a single image, it returns the right dimensions:

train_dataset[0].shape
torch.Size([1, 105, 105])

But when I index several images, it returns the dimensions in the wrong order (I expect 3 x 105 x 105):

train_dataset[[1,2,3]].shape
torch.Size([105, 3, 105])


title: Fixing torch.cuda.is_available() when it is False

body:
How can I change the result of torch.cuda.is_available() to True?
These are some the information I can provide about my system:
$ python
Python 3.6.7 | packaged by conda-forge | (default, Nov  6 2019, 16:19:42)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.version.cuda
'8.0.61'
>>> torch.cuda.is_available()
False
>>> torch.backends.cudnn.enabled
True
>>> torch.__version__
'1.0.0.dev20190328'

and
$ uname -a
Linux goku.bu.edu 3.10.0-1127.18.2.el7.x86_64 #1 SMP Sun Jul 26 15:27:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.8.2003 (Core)
Release:        7.8.2003
Codename:       Core

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.130

$ nvidia-smi
Sat Sep 19 00:04:00 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                  N/A |
|  0%   24C    P8    12W / 250W |     55MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
|  0%   27C    P8    12W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2637      G   /usr/bin/X                         39MiB |
|    0   N/A  N/A      2903      G   /usr/bin/gnome-shell               12MiB |
+-----------------------------------------------------------------------------+

Regarding GPU:
$ lspci | grep ' VGA ' | cut -d" " -f 1 | xargs -i lspci -v -s {}
05:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 6598
        Flags: bus master, fast devsel, latency 0, IRQ 88, NUMA node 0
        Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        [virtual] Expansion ROM at fb000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_drm, nvidia

06:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: eVga.com. Corp. Device 6598
        Flags: bus master, fast devsel, latency 0, IRQ 89, NUMA node 0
        Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at d000 [size=128]
        [virtual] Expansion ROM at f9000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nouveau, nvidia_drm, nvidia


title: Model parameter initialization

body:
I am new to PyTorch. I am writing a simple program for linear regression and i want to compare the results by using different methods (SGD,momentum,ADAM,etc). The problem I have is that I want every time a loop ends for the model parameters to be reinitialized to the same value that the previous model started with  so the comparison is valid.

This is what i have so far, this is my training data:

      x1=np.arange(0,10,1).reshape(10,1)
      y1=2*x1+1+np.random.normal(0,1,(10,1))
      x=torch.from_numpy(x1)
      y=torch.from_numpy(y1)

Here i train the data

      from torch.utils.data import TensorDataset,DataLoader
      train=TensorDataset(xdata,ydata)
      size_batch=10
      dl=DataLoader(train,size_batch,shuffle=True)

Define the model and opt

     model=nn.Linear(1,1)
     opt=torch.optim.SGD(model.parameters(), lr=1e-4)
     import torch.nn.functional as F
     loss1=F.mse_loss
     loss=loss1(model(x),y)

Function

     def fitmodel(nepochs, model, loss1, opt):
         for epoch in range(nepochs):
             for xm,ym in dl:
                 predict = model(xm)
                 loss = loss1(predict, ym)
                 loss.backward()
                 opt.step()
                 opt.zero_grad()

Call the function

     fitmodel(1000,model,loss1,opt)

Now i want to rerun the above but for different optimization algorithms. If I just rerun fitmodel it will use some of the parameters it already has calculated. I want to start from the same 'initial conditions' as i started the previous run. Anyone has any ideas how to do that?

Edit
Before i run the fitmodel I copy the initial bias and weight

   w1=model.weight
   b1=model.bias
   fitmodel(1000,model,loss1,opt)
   model.weight=w1
   model.bias=b1
   loss=[]

But i get this error:
TypeError: cannot assign 'list' as parameter 'bias' (torch.nn.Parameter or None expected)


title: How to do sequence classification with pytorch nn.Transformer?

body:
I am doing a sequence classification task using nn.TransformerEncoder(). Whose pipeline is similar to nn.LSTM().

I have tried several temporal features fusion methods:

Selecting the final outputs as the representation of the whole sequence.
Using an affine transformation to fuse these features.
Classifying the sequence frame by frame, and then select the max values to be the category of the whole sequence.

But, all these 3 methods got a terrible accuracy, only 25% for 4 categories classification. While using nn.LSTM with the last hidden state, I can achieve 83% accuracy easily. I tried plenty of hyperparameters of nn.TransformerEncoder(), but without any improvement for the accuracy.

I have no idea about how to adjust this model now. Could you give me some practical advice? Thanks.

For LSTM: the forward() is:

    def forward(self, x_in, x_lengths, apply_softmax=False):

        # Embed
        x_in = self.embeddings(x_in)

        # Feed into RNN
        out, h_n = self.LSTM(x_in) #shape of out: T*N*D

        # Gather the last relevant hidden state
        out = out[-1,:,:] # N*D

        # FC layers
        z = self.dropout(out)
        z = self.fc1(z)
        z = self.dropout(z)
        y_pred = self.fc2(z)

        if apply_softmax:
            y_pred = F.softmax(y_pred, dim=1)
        return y_pred

For transformer:

    def forward(self, x_in, x_lengths, apply_softmax=False):

        # Embed
        x_in = self.embeddings(x_in)

        # Feed into RNN
        out = self.transformer(x_in)#shape of out T*N*D

        # Gather the last relevant hidden state
        out = out[-1,:,:] # N*D

        # FC layers
        z = self.dropout(out)
        z = self.fc1(z)
        z = self.dropout(z)
        y_pred = self.fc2(z)

        if apply_softmax:
            y_pred = F.softmax(y_pred, dim=1)
        return y_pred


title: Why are the images generated by a GAN get darker as the network trains more?

body:
I created a simple DCGAN with 6 layers and trained it on CelebA dataset (a portion of it containing 30K images).
I noticed my network generated images are dimmed looking and as the network trains more, the bright colors fade into dim ones!

here are some example:
This is how CelebA images look like (real images used for training)  :

and these are the generated ones ,the number shows  the epoch number(they were trained for 30 epochs ultimately) :

What is the cause for this phenomenon?
I tried to do all the general tricks concerning GANs, such as rescaling the input image between -1 and 1, or not using BatchNorm in the first layer of the Discriminator, and for the last layer of the Generator or
using LeakyReLU(0.2) in the Discriminator, and ReLU for the Generator. yet I have no idea why the images are this dim/dark!
Is this caused by simply less training images?
or is it caused by the networks deficiencies ? if so what is the source of such deficiencies?
Here are how these networks implemented :

def conv_batch(in_dim, out_dim, kernel_size, stride, padding, batch_norm=True):
    layers = nn.ModuleList()
    conv = nn.Conv2d(in_dim, out_dim, kernel_size, stride, padding, bias=False)
    layers.append(conv)
 if batch_norm:
        layers.append(nn.BatchNorm2d(out_dim))
 return nn.Sequential(*layers)

class Discriminator(nn.Module):
    def __init__(self, conv_dim=32, act = nn.ReLU()):
        super().__init__()

        self.conv_dim = conv_dim
        self.act = act
        self.conv1 = conv_batch(3, conv_dim, 4, 2, 1, False)
        self.conv2 = conv_batch(conv_dim, conv_dim*2, 4, 2, 1)
        self.conv3 = conv_batch(conv_dim*2, conv_dim*4, 4, 2, 1)
        self.conv4 = conv_batch(conv_dim*4, conv_dim*8, 4, 1, 1)
        self.conv5 = conv_batch(conv_dim*8, conv_dim*10, 4, 2, 1)
        self.conv6 = conv_batch(conv_dim*10, conv_dim*10, 3, 1, 1)

        self.drp = nn.Dropout(0.5)
        self.fc = nn.Linear(conv_dim*10*3*3, 1)

    def forward(self, input):
        batch = input.size(0)
        output = self.act(self.conv1(input))
        output = self.act(self.conv2(output))
        output = self.act(self.conv3(output))
        output = self.act(self.conv4(output))
        output = self.act(self.conv5(output))
        output = self.act(self.conv6(output))

        output = output.view(batch, self.fc.in_features)
        output = self.fc(output)
        output = self.drp(output)

        return output

def deconv_convtranspose(in_dim, out_dim, kernel_size, stride, padding, batchnorm=True):
    layers = []
    deconv = nn.ConvTranspose2d(in_dim, out_dim, kernel_size = kernel_size, stride=stride, padding=padding)
    layers.append(deconv)
    if batchnorm:
        layers.append(nn.BatchNorm2d(out_dim))
    return nn.Sequential(*layers)

class Generator(nn.Module):
    def __init__(self, z_size=100, conv_dim=32):
        super().__init__()
         self.conv_dim = conv_dim
         # make the 1d input into a 3d output of shape (conv_dim*4, 4, 4 )
         self.fc = nn.Linear(z_size, conv_dim*4*4*4)#4x4
         # conv and deconv layer work on 3d volumes, so we now only need to pass the number of fmaps and the
         # input volume size (its h,w which is 4x4!)
         self.drp = nn.Dropout(0.5)
        self.deconv1 = deconv_convtranspose(conv_dim*4, conv_dim*3, kernel_size =4, stride=2, padding=1)
        self.deconv2 = deconv_convtranspose(conv_dim*3, conv_dim*2, kernel_size =4, stride=2, padding=1)
        self.deconv3 = deconv_convtranspose(conv_dim*2, conv_dim, kernel_size =4, stride=2, padding=1)
        self.deconv4 = deconv_convtranspose(conv_dim, conv_dim, kernel_size =3, stride=2, padding=1)
        self.deconv5 = deconv_convtranspose(conv_dim, 3, kernel_size =4, stride=1, padding=1, batchnorm=False)

    def forward(self, input):
        output = self.fc(input)
        output = self.drp(output)
        output = output.view(-1, self.conv_dim*4, 4, 4)
        output = F.relu(self.deconv1(output))
        output = F.relu(self.deconv2(output))
        output = F.relu(self.deconv3(output))
        output = F.relu(self.deconv4(output))
        # we create the image using tanh!
        output = F.tanh(self.deconv5(output))

        return output

# testing nets
dd = Discriminator()
zd = np.random.rand(2,3,64,64)
zd = torch.from_numpy(zd).float()
# print(dd)
print(dd(zd).shape)

gg = Generator()
z = np.random.uniform(-1,1,size=(2,100))
z = torch.from_numpy(z).float()
print(gg(z).shape)


title: Pytorch module not found

body:
I'm using Python 3.6, and trying to use Pytorch. I've uninstalled it using pip3, and re-installed it

Hugos-MacBook-Pro-2:project hugokitano$ pip3 install torch
Requirement already satisfied: torch in /usr/local/lib/python3.7/site-packages (1.3.1)
Requirement already satisfied: numpy in /Users/hugokitano/Library/Python/3.7/lib/python/site-packages (from torch) (1.17.2)

However, when I try to import torch, the module is not found. I've also tried to install via conda, running

  conda install pytorch torchvision -c pytorch

and it was successful, installing to

  environment location: /Users/hugokitano/anaconda

However, "import torch" still doesn't work. Any thoughts? Thanks!


title: ModuleNotFoundError: No module named 'allennlp.data.iterators'

body:
In Google colab I am to trying import BucketIterator using:
from allennlp.data.iterators import BucketIterator

But it is raising the same error again and again-
ModuleNotFoundError: No module named 'allennlp.data.iterators

After installing allennlp with the imports:
from allennlp.data.token_indexers import TokenIndexer, SingleIdTokenIndexer
from allennlp.data.tokenizers.character_tokenizer import CharacterTokenizer
from allennlp.data.vocabulary import Vocabulary
from allennlp.modules.seq2vec_encoders import PytorchSeq2VecWrapper

are working fine. Is there a way to solve this issue?


title: What does Pytorch's nn.Linear(x,y) return?

body:
I am new to object orientation, and I am having troubles understanding the following:
import torch.nn as nn

class mynet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(20, 64)

    def forward(self, x):
        x = self.fc1(x)

The line self.fc1 = nn.Linear(20, 64)  is supposed to create a member variable fc1 to my class, right? But what is the return value of nn.Linear(20, 64)?
According to the documentation, nn.Linear is defined as
class torch.nn.Linear(in_features: int, out_features: int, bias: bool = True).
However, in my basic OOP tutorial I have only seen something like class CLASSNAME(BASECLASS) so that the class CLASSNAME inherits from BASECLASS. What does the documentation mean with its way of writing all that stuff in between the brackets?
Also, the line x=fc1(x) somehow makes it look as if fc1 was a function now.
I seem to lack OOP knowledge here... Any help appreciated!


title: Pytorch RNN memory allocation error in DataLoader

body:
I am writing an RNN in Pytorch.
I have the following line of code:

data_loader = torch.utils.data.DataLoader(
    data,
    batch_size=args.batch_size,
    shuffle=True,
    num_workers=args.num_workers,
    drop_last=True)

If I set num_workers to 0, I get a segmentation fault.
If I set num_workers to > 0, then I have the traceback:

Traceback (most recent call last):
File "rnn_model.py", line 352, in <module>
train_model(train_data, dev_data, test_data, model, args)
File "rnn_model.py", line 212, in train_model
loss = run_epoch(train_data, True, model, optimizer, args)
File "rnn_model.py", line 301, in run_epoch
for batch in tqdm.tqdm(data_loader):
File "/home/username/miniconda3/lib/python2.7/site-packages/tqdm/_tqdm.py",
line 872, in __iter__
for obj in iterable:
File "/home/username/miniconda3/lib/python2.7/site-
packages/torch/utils/data/dataloader.py", line 303, in __iter__
return DataLoaderIter(self)
File "/home/username/miniconda3/lib/python2.7/site-
packages/torch/utils/data/dataloader.py", line 162, in __init__
w.start()
File "/home/username/miniconda3/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/home/username/miniconda3/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory


title: How to get rid of zombie processes using torch.multiprocessing.Pool (Python)

body:
I am using torch.multiprocessing.Pool to speed up my NN in inference, like this:

import torch.multiprocessing as mp
mp = torch.multiprocessing.get_context('forkserver')

def parallel_predict(predict_func, sequences, args):
    predicted_cluster_ids = []
    pool = mp.Pool(args.num_workers, maxtasksperchild=1)
    out = pool.imap(
        func=functools.partial(predict_func, args=args),
        iterable=sequences,
        chunksize=1)
    for item in tqdm(out, total=len(sequences), ncols=85):
        predicted_cluster_ids.append(item)
    pool.close()
    pool.terminate()
    pool.join()
    return predicted_cluster_ids

Note 1) I am using imap because I want to be able to show a progress bar with tqdm.
Note 2) I tried with both forkserver and spawn but no luck. I cannot use other methods because of how they interact (poorly) with CUDA.
Note 3) I am using maxtasksperchild=1 and chunksize=1 so for each sequence in sequences it spawns a new process.
Note 4) Adding or removing pool.terminate() and pool.join() makes no difference.
Note 5) predict_func is a method of a class I created. I could also pass the whole model to parallel_predict but it does not change anything.

Everything works fine except the fact that after a while I run out of memory on the CPU (while on the GPU everything works as expected). Using htop to monitor memory usage I notice that, for every process I spawn with pool I get a zombie that uses 0.4% of the memory. They don't get cleared, so they keep using space. Still, parallel_predict does return the correct result and the computation goes on. My script is structured in a way that id does validation multiple times so next time parallel_predict is called the zombies add up.

This is what I get in htop:

Usually, these zombies get cleared after ctrl-c but in some rare cases I need to killall.

Is there some way I can force the Pool to close them?

UPDATE:
I tried to kill the zombie processes using this:

def kill(pool):
    import multiprocessing
    import signal
    # stop repopulating new child
    pool._state = multiprocessing.pool.TERMINATE
    pool._worker_handler._state = multiprocessing.pool.TERMINATE
    for p in pool._pool:
        os.kill(p.pid, signal.SIGKILL)
    # .is_alive() will reap dead process
    while any(p.is_alive() for p in pool._pool):
        pass
    pool.terminate()

But it does not work. It gets stuck at pool.terminate()

UPDATE2:
I tried to use the initializer arg in imap to catch signals like this:

def process_initializer():
    def handler(_signal, frame):
        print('exiting')
        exit(0)
    signal.signal(signal.SIGTERM, handler)

def parallel_predict(predict_func, sequences, args):
    predicted_cluster_ids = []
    with mp.Pool(args.num_workers, initializer=process_initializer, maxtasksperchild=1) as pool:
        out = pool.imap(
            func=functools.partial(predict_func, args=args),
            iterable=sequences,
            chunksize=1)
        for item in tqdm(out, total=len(sequences), ncols=85):
            predicted_cluster_ids.append(item)
        for p in pool._pool:
            os.kill(p.pid, signal.SIGTERM)
        pool.close()
        pool.terminate()
        pool.join()
    return predicted_cluster_ids

but again it does not free memory.


title: Same random seed for Numpy and PyTorch

body:
I am trying to compare my functions in vanilla Python/Numpy to those of PyTorch. For that I would like to set the seed of both to make the results comparable.

Is it possible to set the seed for numpy like np.random.seed(x) and PyTorch like torch.manual_seed(x) so that both produce the same random numbers?

Right now, with the same x value np.random.uniform(0, 1) and torch.rand(1) produce different outputs.


title: PyTorch: accuracy of validation set greater than 100% during training

body:
1 ) Problem

I observe an odd behaviour during training where my validation-accuracy is above 100% right from the start.

Epoch 0/3
----------
100%|██████████| 194/194 [00:50<00:00,  3.82it/s]
train Loss: 1.8653 Acc: 0.4796
100%|██████████| 194/194 [00:32<00:00,  5.99it/s]
val Loss: 1.7611 Acc: 1.2939

Epoch 1/3
----------
100%|██████████| 194/194 [00:42<00:00,  4.61it/s]
train Loss: 0.8704 Acc: 0.7467
100%|██████████| 194/194 [00:31<00:00,  6.11it/s]
val Loss: 1.0801 Acc: 1.4694

The output indicates that one epoch iterates over 194 batches, which does seem to be  correct for the training data (which has a length of 6186, batch_size is 32, hence 32*194 = 6208 and this is ≈6186) but does not match the size of the validation-data (length of 3447, batch_size = 32).

Hence I would expect my validation-loop to generate 108 (3447 / 32 ≈ 108) batches insted of 194.

I thought this behaviour is handled within my for loop at:

for dataset in tqdm(dataloaders[phase]):

But somehow I can't figure out what is wrong here. See point 3) below for my entire code.

2 ) Question

If my assumption above is correct i.e. that this error stems from the for-loop within in my code then I would like to know the following:

How do I need to adjust the for-loop during the validation phase to handle the number of batches that are being used for validation correctly?

3 ) Background:

Following two tutorials, one on how to do transfer-learning (https://discuss.pytorch.org/t/transfer-learning-using-vgg16/20653) and one on how to do data-loading (https://pytorch.org/tutorials/beginner/data_loading_tutorial.html) in pytorch, I am trying to customize the code such that I can perform transfer-learning on a new custom dataset which I want to provide via pandas dataframes.

As such, my training- and validation-data is provided via two dataframes (df_train & df_val) which both contain two columns, one for the path and one for the target. E.g. like this:

    url                                 target
0   C:/Users/aaron/Desktop/pics/4ebd... 9
1   C:/Users/aaron/Desktop/pics/7153... 3
2   C:/Users/aaron/Desktop/pics/3ee6... 3
3   C:/Users/aaron/Desktop/pics/4652... 16
4   C:/Users/aaron/Desktop/pics/28ce... 15
...

And their respective length:

print(len(df_train))
print(len(df_val))
>> 6186
>> 3447

My pipeline looks like this:

class CustomDataset(Dataset):
    def __init__(self, df, transform=None):

        self.dataframe = df_train
        self.transform = transform

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, idx):
        img_name = self.dataframe.iloc[idx, 0]
        img = Image.open(img_name)
        img_normalized = self.transform(img)

        landmarks = self.dataframe.iloc[idx, 1]
        sample = {'data': img_normalized, 'label': int(landmarks)}

        return sample

train_dataset = CustomDataset(df_train,transform=transforms.Compose([
                                                transforms.Resize(224),
                                               transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]))

val_dataset = CustomDataset(df_val,transform=transforms.Compose([
                                                transforms.Resize(224),
                                               transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]))

train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=32,shuffle=True, num_workers=0)
val_loader = torch.utils.data.DataLoader(val_dataset,batch_size=32,shuffle=True, num_workers=0)

dataloaders = {'train': train_loader, 'val': val_loader}
dataset_sizes = {'train': len(df_train) ,'val': len(df_val)}

################### Training

from tqdm import tqdm

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for dataset in tqdm(dataloaders[phase]):

                inputs, labels = dataset["data"], dataset["label"]
                #print(inputs.type())
                inputs = inputs.to(device, dtype=torch.float)
                labels = labels.to(device,dtype=torch.long)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, len(le.classes_))

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=4)


title: batch_size not match in torchtext BucketIterator

body:
I have set batch_size equals to 64, but when i print out the train_batch and val_batch, the size is not equal to 64.

The train data and val data are in the below format:

First, i define TEXT and LABEL field.

tokenize = lambda x: x.split()

TEXT = data.Field(sequential=True, tokenize=tokenize)
LABEL = data.Field(sequential=False)

And then i keep trying follow tutorials, and wrote things below:

train_data, valid_data = data.TabularDataset.splits(
        path='.',
        train='train_intent.csv', validation='val.csv',
        format='csv',
        fields= {'sentences': ('text', TEXT),
                'labels': ('label',LABEL)}
)

test_data = data.TabularDataset(
        path='test.csv',
        format='csv',
        fields={'sentences': ('text', TEXT)}

)
TEXT.build_vocab(train_data)
LABEL.build_vocab(train_data)

BATCH_SIZE = 64

train_iter, val_iter = data.BucketIterator.splits(
    (train_data, valid_data),
    batch_sizes=(BATCH_SIZE, BATCH_SIZE),
    sort_key=lambda x: len(x.text),
    sort_within_batch=False,
    repeat=False,
    device=device
)

But when i want to know the iter is fine or not, i just find the below strange things:

train_batch = next(iter(train_iter))
print(train_batch.text.shape)
print(train_batch.label.shape)
[output]
torch.Size([15, 64])
torch.Size([64])

And the train process output errorValueError: Expected input batch_size (15) to match target batch_size (64).:

def train(model, iterator, optimizer, criterion):

    epoch_loss = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        predictions = model(batch.text)
        loss = criterion(predictions, batch.label)
        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()

    return epoch_loss / len(iterator)

Anyone could give me a hint would be highly appreciated. Thanks!


title: How to get log probabilities in TensorFlow?

body:
I am trying to convert a pytorch script to tensorflow and I need to get log probabilities from a categorical distribution. But the tensorflow calculated log probabilities is different from pytorch's log prob even after using same seed. This is what I have done so far

import torch
from torch.distributions import Categorical
import tensorflow as tf
import tensorflow_probability as tfp

torch.manual_seed(1)
tf.random.set_seed(1)

probs =[0.4,0.6]
m = Categorical(torch.tensor(probs))
action = m.sample()

n = tfp.distributions.Categorical(probs)
print("pytorch",m.log_prob(action))
print("Tensorflow", tf.math.log(n.prob(action.item())))


title: Pytorch error Illegal instruction (core dumped)

body:
I have install pytorch version 0.4.1

I installed it directly with pip without conda, I've also noted the issue is with the binary and from my research on processor incompatibility with C gcc version.
My version of gcc is 7.3.0

And my processor type AMD A8-7410 APU with AMD Radeon R5 Graphics

gcc location.

(data-science) sam@sam-Lenovo-G51-35:~/code/data science projects/pytorch$ which gcc
/usr/bin/gcc

This is how I get my error.

The following code runs...

from torchvision import datasets, transforms
from torch import nn
import torch
import torch.nn.functional as F

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                              ])

# Download and load the training data
trainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

dataiter = iter(trainloader)
images, labels = dataiter.next()

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        # Defining the layers, 128, 64, 10 units each
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 64)
        # Output layer, 10 units - one for each digit
        self.fc3 = nn.Linear(64, 10)

    def forward(self, x):
        ''' Forward pass through the network, returns the output logits '''

        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = F.softmax(x, dim=1)

        return x

# Create the network and look at it's text representation
model = Network()

# print(model.fc1.weight)
# print(model.fc1.bias)

# Set biases to all zeros
model.fc1.bias.data.fill_(0)

# sample from random normal with standard dev = 0.01
model.fc1.weight.data.normal_(std=0.01)

# Grab some data
dataiter = iter(trainloader)
images, labels = dataiter.next()

# Resize images into a 1D vector, new shape is (batch size, color channels, image pixels)
images.resize_(64, 1, 784)
# or images.resize_(images.shape[0], 1, 784) to automatically get batch size

all of this successfully.

But when I run this other line

# Forward pass through the network
img_idx = 0
ps = model.forward(images[img_idx,:])

I get this error

Illegal instruction (core dumped)


title: Text classification using BERT - how to handle misspelled words

body:
I am not sure if this is the best place to submit that kind of question, perhaps CrossValdation would be a better place.

I am working on a text multiclass classification problem.
I built a model based on BERT concept implemented in PyTorch (huggingface transformer library). The model performs pretty well, except when the input sentence has an OCR error or equivalently it is misspelled.

For instance, if the input is "NALIBU DRINK" the Bert tokenizer generates ['na', '##lib', '##u', 'drink'] and model's prediction is completely wrong. On the other hand, if I correct the first character, so my input is "MALIBU DRINK", the Bert tokenizer generates two tokens ['malibu', 'drink'] and the model makes a correct prediction with very high confidence.

Is there any way to enhance Bert tokenizer to be able to work with misspelled words?


title: 'NoneType' object has no attribute 'size' - how to face detection using mtcnn?

body:
i'm trying to build face recognition in real time in neural network(facenet network) using pytorch and face detection using MTCNN
i've tried this for detecting faces in real time  (from webcam) but doesnt work
read frames then going through mtcnn detector

import cv2
capture = cv2.VideoCapture(0)
while(True):
   ret, frame = capture.read()
   frames_tracked = []
   print('\rTracking frame: {}'.format(i + 1), end='')
   boxes,_ = mtcnn.detect(frame)
   frame_draw = frame.copy()
   draw = ImageDraw.Draw(frame_draw)
   for box in boxes:
       draw.rectangle(box.tolist(), outline=(255, 0, 0), width=6)
       frames_tracked.append(frame_draw.resize((640, 360), Image.BILINEAR))
   d = display.display(frames_tracked[0], display_id=True)
   i = 1
   try:
      while True:
          d.update(frames_tracked[i % len(frames_tracked)])
          i += 1
   except KeyboardInterrupt:
      pass
   if cv2.waitKey('q') == 27:
     break

  capture.release()
  cv2.destroyAllWindows()

but it will rise this error :

this is the entire traceback http://dpaste.com/0HR58RQ

  AttributeError: 'NoneType' object has no attribute 'size'

is there a solution for this problem ? what caused this error? thanks for your advice


title: How to get the value of a feature in a layer that match a the state dict in PyTorch?

body:
I have some cnn, and I want to fetch the value of some intermediate layer corresponding to a some key from the state dict.
How could this be done?
Thanks.


title: Why is very simple PyTorch LSTM model not learning?

body:
I am trying to do very simple learning so that I can better understand how PyTorch and LSTMs work. To that end, I am trying to learn a mapping from an input tensor to an output tensor (same shape) that is twice the value. So [1 2 3] as input should learn [2 4 6] as an output. To that end, I have a dataloader:

class AudioDataset(Dataset):
    def __init__(self, corrupted_path, train_set=False, test_set=False):
        torch.manual_seed(0)
        numpy.random.seed(0)

    def __len__(self):
        return len(self.file_paths)

    def __getitem__(self, index):
        random_tensor = torch.rand(1, 5) * 2
        random_tensor = random_tensor - 1

        return random_tensor, random_tensor * 2

My LSTM itself is pretty simple:

class MyLSTM(nn.Module):
    def __init__(self, input_size=4000):
        super(MyLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=input_size, hidden_size=input_size,
                            num_layers=2)

    def forward(self, x):
        y = self.lstm(x)
        return y

My training looks like:

    train_loader = torch.utils.data.DataLoader(
        train_set, batch_size=1, shuffle=True, **kwargs)

    model = MyLSTM(input_size=5)
    optimizer = optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0001)
    loss_fn = torch.nn.MSELoss(reduction='sum')

    for epoch in range(300):
        for i, data in enumerate(train_loader):
            inputs = data[0]
            outputs = data[1]

            print('inputs', inputs, inputs.size())
            print('outputs', outputs, outputs.size())
            optimizer.zero_grad()

            pred = model(inputs)
            print('pred', pred[0], pred[0].size())

            loss = loss_fn(pred[0], outputs)

            model.zero_grad()

            loss.backward()
            optimizer.step()

After 300 epochs, my loss looks like tensor(1.4892, grad_fn=<MseLossBackward>). Which doesn't appear to be very good.

Randomly looking at some of the inputs / outputs and predictions:

inputs tensor([[[0.5050, 0.4669, 0.8310,  ..., 0.0659, 0.5043, 0.8885]]]) torch.Size([1, 1, 4000])
outputs tensor([[[1.0100, 0.9338, 1.6620,  ..., 0.1319, 1.0085, 1.7770]]]) torch.Size([1, 1, 4000])
pred tensor([[[ 0.6930,  0.0231, -0.6874,  ..., -0.5225,  0.1096,  0.5796]]],
       grad_fn=<StackBackward>) torch.Size([1, 1, 4000])

We see that it hasn't learned very much at all. I can't understand what I'm doing wrong; if someone can guide me, that would be greatly appreciated.


title: Input dimension reshape when using PyTorch model with CoreML

body:
I have a seq2seq model in PyTorch that I want to run with CoreML. When exporting the model to ONNX the input dimensions are fixed to the shape of the tensor used during export, and again with the conversion from ONNX to CoreML.

import torch
from onnx_coreml import convert

x = torch.ones((32, 1, 1000))  # N x C x W
model = Model()
torch.onnx.export(model, x, 'example.onnx')

mlmodel = convert(model='example.onnx', minimum_ios_deployment_target='13')
mlmodel.save('example.mlmodel')

For the ONNX export you can export dynamic dimension -

torch.onnx.export(
    model, x, 'example.onnx',
    input_names = ['input'],
    output_names = ['output'],
    dynamic_axes={
        'input' : {0 : 'batch', 2: 'width'},
        'output' : {0 : 'batch', 1: 'owidth'},
    }
)

But this leads to a RunTimeWarning when converting to CoreML -

  RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  Blob with zero size found:

For inference in CoreML I would like the batch (first) and width (last) dimension to either be dynamic or have the ability to statically change them.

Is that possible?


title: Pytorch vectors with empty dimensions

body:
I noticed that some vectors in pytorch have empty dimensions like a=torch.randn(1) has size (1L,). What are these types of vectors called and why are they missing the second dimension? I'm on OSX and Ubuntu - not sure if that makes a difference.

Example:

>>> import torch
>>> a=torch.randn(1)
>>> a

 2.1358
[torch.FloatTensor of size 1]

>>> a.size()
(1L,)
>>> a=torch.randn(1).size()
>>> a
(1L,)


title: Using Multiple GPUs outside of training in PyTorch

body:
I'm calculating the accumulated distance between each pair of kernel inside a nn.Conv2d layer. However for large layers it runs out of memory using a Titan X with 12gb of memory. I'd like to know if it is possible to divide such calculations across two gpus.
The code follows:

def ac_distance(layer):
    total = 0
    for p in layer.weight:
      for q in layer.weight:
         total += distance(p,q)
    return total

Where layer is instance of nn.Conv2d and distance returns the sum of the differences between p and q. I can't detach the graph, however, for I need it later on. I tried wrapping my model around a nn.DataParallel, but all calculations in ac_distance are done using only 1 gpu, however it trains using both.


title: pytorch loss.backward() keeps running for hours

body:
I am using pytorch to train some x-ray images but I ran into the following issue:

in the line : loss.backward(), the program just keeps running and never end, and there is no error or warning.

            loss, outputs = self.forward(images, targets)
            loss = loss / self.accumulation_steps
            print("loss calculated: " + str(loss))

            if phase == "train":
                print("running loss backwarding!")
                loss.backward()
                print("loss is backwarded!")
                if (itr + 1 ) % self.accumulation_steps == 0:
                    self.optimizer.step()
                    self.optimizer.zero_grad()

The loss calculated before this is something like tensor(0.8598, grad_fn=<DivBackward0>).

Could anyone help me with why this keeps running or any good ways to debug the backward() function?

I am using torch 1.2.0+cu92 with the compatible cuda 10.0.

Thank you so much!!


title: net.load_state_dict(torch.load('rnn_x_epoch.net')) not working on cpu

body:
I am using pytorch to train a Neural Network. When I train and test on GPU, it works fine.
But When I try to load the model parameters on CPU using:

net.load_state_dict(torch.load('rnn_x_epoch.net'))

I get the following error:

RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:51

I have searched for the error, it's mainly because of CUDA driver dependency, but since I'm running on CPU when I get this error,it must be something else, or may be I missed something.
Since it's working fine using GPU, I could just run it on GPU but I'm trying to train the network on GPU, store the parameters and then load it on CPU mode for predictions.
I am just looking for a way to load the parameters while on CPU mode.

I tried this as well to load the parameters:

check = torch.load('rnn_x_epoch.net')

It did not work.

I tried to save the model parameters in two ways, to see if any of these would work, but didn't:
1)

checkpoint = {'n_hidden': net.n_hidden,
          'n_layers': net.n_layers,
          'state_dict': net.state_dict(),
          'tokens': net.chars}

with open('rnn_x_epoch.net', 'wb') as f:
    torch.save(checkpoint, f)

2)

torch.save(model.state_dict(), 'rnn_x_epoch.net')

TraceBack:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-e61f28013b35> in <module>()
----> 1 net.load_state_dict(torch.load('rnn_x_epoch.net'))

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
    301         f = open(f, 'rb')
    302     try:
--> 303         return _load(f, map_location, pickle_module)
    304     finally:
    305         if new_fd:

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    467     unpickler = pickle_module.Unpickler(f)
    468     unpickler.persistent_load = persistent_load
--> 469     result = unpickler.load()
    470
    471     deserialized_storage_keys = pickle_module.load(f)

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in persistent_load(saved_id)
    435             if root_key not in deserialized_objects:
    436                 deserialized_objects[root_key] = restore_location(
--> 437                     data_type(size), location)
    438             storage = deserialized_objects[root_key]
    439             if view_metadata is not None:

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in default_restore_location(storage, location)
     86 def default_restore_location(storage, location):
     87     for _, _, fn in _package_registry:
---> 88         result = fn(storage, location)
     89         if result is not None:
     90             return result

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in _cuda_deserialize(obj, location)
     68     if location.startswith('cuda'):
     69         device = max(int(location[5:]), 0)
---> 70         return obj.cuda(device)
     71
     72

/opt/conda/lib/python3.6/site-packages/torch/_utils.py in _cuda(self, device, non_blocking, **kwargs)
     66         if device is None:
     67             device = -1
---> 68     with torch.cuda.device(device):
     69         if self.is_sparse:
     70             new_type = getattr(torch.cuda.sparse,
self.__class__.__name__)

/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py in __enter__(self)
    223         if self.idx is -1:
    224             return
--> 225         self.prev_idx = torch._C._cuda_getDevice()
    226         if self.prev_idx != self.idx:
    227             torch._C._cuda_setDevice(self.idx)

RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:51

Also may be the save/load operations in Pytorch are only for GPU mode, but I am not really convinced by that.


title: nvidia_deeplearningexamples_tacotron2 :RuntimeError: CUDA error: invalid device function

body:
Set up runtime: python3 and GPU.

Run the code step by step.
I only successfully run the code at first time.

After that, when run the below part, occured "RuntimeError: CUDA error: invalid device function"

sequence = np.array(tacotron2.text_to_sequence(text, ['english_cleaners']))[None, :]
sequence = torch.from_numpy(sequence).to(device='cuda', dtype=torch.int64)

with torch.no_grad():
    _, mel, _, _ = tacotron2.infer(sequence)
    audio = waveglow.infer(mel)
audio_numpy = audio[0].data.cpu().numpy()
rate = 22050

Do you know the root cause? And does the pre-trained model be run on local CPU?


title: Load csv and Image dataset in pytorch

body:
I am doing image classification with PyTorch. I have a separate Images folder and train and test csv file with images ids and labels . I don’t have any an idea about how to combine those images and ID and converting into tensors.

train.csv : contains all ID of Image like 4325.jpg, 2345.jpg,…so on and contains Labels like cat,dog.
Image_data : contains all the images of with ID name.


title: torch.pow does not work

body:
I'm trying to create a custom loss function using PyTorch, and am running into a simple error.

When I try to use torch.pow to take the exponent of a PyTorch Variable, I get the following error message:

AttributeError: 'torch.LongTensor' object has no attribute 'pow'

In the python terminal, I created a simple Variable, and attempted to do the same, and received the same error. Here's a snippet that should recreate the problem:

import torch
from torch.autograd import Variable
import numpy as np

v = Variable(torch.from_numpy(np.array([1, 2, 3, 4])))
torch.pow(v, 2)

I can't find any information on this issue, and nothing is showing up in search results. Help?

EDIT: this problem also occurs when I try to use torch.sqrt()
EDIT: same problem also happens if I try to do

v.pow(2)

pow is definitely a method of v, and the docs clearly state that pow is a method that exists and takes a tensor as it's argument. I really don't see how this is happening, and it seems to me that the docs are just flat out wrong and these methods don't actually work.


title: Understanding Pytorch vanilla RNN architectures

body:
Standard interpretation: in the original RNN, the hidden state and output are calculated as

In other words, we obtain the the output from the hidden state.

According to Wiki, the RNN architecture can be unfolded like this:

And the code I have been using is like:

class Model(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(Model, self).__init__()
        self.hidden_dim = hidden_dim
        self.rnn = nn.RNN(input_size, hidden_dim, 1)
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x):
        batch_size = x.size(0)

        out, hidden = self.rnn(x)

        # getting output from the hidden state
        out = out..view(-1, self.hidden_dim)
        out = self.fc(out)

        return out, hidden

RNN as "pure" feed-forward layers: but today, I see another implementation from the Pytorch Tutorial

And their code is like

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

The hidden layer calculation is same as the standard interpretation, but the output is is calculated independently from the current hidden state h.

To me, the math behind this implementation is:

So, this implementation is different from the original RNN implementation?

I have been using RNN for almost 1 year and I thought I understand it, not until today when I see this post from Pytorch. I am really confused now.


title: `return_sequences = False` equivalent in pytorch LSTM

body:
In tensorflow/keras, we can simply set return_sequences = False for the last LSTM layer before the classification/fully connected/activation (softmax/sigmoid) layer to get rid of the temporal dimension.

In PyTorch, I don't find anything similar. For the classification task, I don't need a sequence to sequence model but many to one architecture like this:

Here's my simple bi-LSTM model.

import torch
from torch import nn

class BiLSTMClassifier(nn.Module):
    def __init__(self):
        super(BiLSTMClassifier, self).__init__()
        self.embedding = torch.nn.Embedding(num_embeddings = 65000, embedding_dim = 64)
        self.bilstm = torch.nn.LSTM(input_size = 64, hidden_size = 8, num_layers = 2,
                                    batch_first = True, dropout = 0.2, bidirectional = True)
        # as we have 5 classes
        self.linear = nn.Linear(8*2*512, 5) # last dimension
    def forward(self, x):
        x = self.embedding(x)
        print(x.shape)
        x, _ = self.bilstm(x)
        print(x.shape)
        x = self.linear(x.reshape(x.shape[0], -1))
        print(x.shape)

# create our model

bilstmclassifier = BiLSTMClassifier()

If I observe the shapes after each layer,

xx = torch.tensor(X_encoded[0]).reshape(1,512)
print(xx.shape)
# torch.Size([1, 512])
bilstmclassifier(xx)
#torch.Size([1, 512, 64])
#torch.Size([1, 512, 16])
#torch.Size([1, 5])

What can I do so that the last LSTM returns a tensor with shape (1, 16) instead of (1, 512, 16)?


title: How to implement differentiable hamming loss in pytorch?

body:
How to implement a differentiable loss function that counts the number of wrong predictions?

output = [1,0,4,10]
target = [1,2,4,15]
loss = np.count_nonzero(output != target) / len(output) # [0,1,0,1] -> 2 / 4 -> 0.5

I have tried a few implementations but they are not differentiable. RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

def hamming_loss(output, target):
  #loss = torch.tensor(torch.nonzero(output != target).size(0)).double() / target.size(0)
  #loss = torch.sum((output != target), dim=0).double() / target.size(0)
  loss = torch.mean((output != target).double())
  return loss

Maybe there is some similar but differential loss function?