title: Error with pip while building singularity image

body:
I'm trying to build a .simg file from a .recipe. In the recipe I upgraded pip. In the building process I get following error after downloading pytorch:
can someone tell me what's wrong with pip?
Collecting torch==1.5.1
  Downloading torch-1.5.1-cp35-cp35m-manylinux1_x86_64.whl (753.2 MB)
     |############################### | 753.2 MB 6.8 MB/s eta 0:00:01
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/base_command.py", line 188, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
    return func(self, options, args)
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/commands/install.py", line 333, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/operations/prepare.py", line 469, in prepare_linked_requirement
    hashes=hashes,
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/operations/prepare.py", line 259, in unpack_url
    hashes=hashes,
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/operations/prepare.py", line 130, in get_http_url
    link, downloader, temp_dir.path, hashes
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/operations/prepare.py", line 281, in _download_http_url
    for chunk in download.chunks:
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/progress_bars.py", line 166, in iter
    for x in it:
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/network/utils.py", line 39, in response_chunks
    decode_content=False,
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/urllib3/response.py", line 564, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/urllib3/response.py", line 507, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 65, in read
    self._close()
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 52, in _close
    self.__callback(self.__buf.getvalue())
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/cachecontrol/controller.py", line 309, in cache_response
    cache_url, self.serializer.dumps(request, response, body=body)
  File "/usr/local/lib/python3.5/dist-packages/pip/_vendor/cachecontrol/serialize.py", line 72, in dumps
    return b",".join([b"cc=4", msgpack.dumps(data, use_bin_type=True)])
MemoryError

The .recipe file I used looks as follows:
As you can see I tried to update pip, so it should be the most recent version.
#header
Bootstrap: docker
From: nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04

#Sections

%post
#commands executed inside the container after os has been installed. Used for setup of the container
#devine env variables for build time

# Install the commonly used packages (from repo)
    apt-get -y update
    apt-get install -y python-pip
    apt-get install -y python3-pip
    pip3 install --upgrade setuptools pip

# Install the commonly used packages (from repo)
    apt-get update && apt-get install -y --no-install-recommends \
        apt-utils \
        build-essential \
        curl \
        git \
        libopenblas-dev \
        libcurl4-openssl-dev \
        libfreetype6-dev \
        libpng-dev \
        libzmq3-dev \
        locales \
        pkg-config \
        python \
        python-dev \
        python-setuptools \
        rsync \
        software-properties-common \
        unzip \
        vim \
        zip \
        zlib1g-dev

    pip3 install --no-cache-dir --upgrade future matplotlib scipy

    # Install other commonly-needed python packages
    pip3 install --no-cache-dir --upgrade \
        future \
        matplotlib \
        scipy

pip3 install opencv-contrib-python
python3 -m pip install keras
pip3 install tensorflow-gpu
CUDA_PATH=/usr/local/cuda pip3 install cupy-cuda101
CUDA_ROOT=/usr/local/cuda PATH=$PATH:/usr/local/cuda/bin pip3 install scikit-cuda
pip3 install pyro4
pip3 install pandas
pip3 install torchvision
pip3 install torch
pip3 install backpack-for-pytorch
apt-get update
apt-get -y install python3-tk
apt-get -y install python-opencv
pip3 install opencv-contrib-python
pip3 install appdirs requests


title: Calculate number of parameters in neural network

body:
I am wondering would the number of parameters in the models like ResNet18, Vgg16, and DenseNet201 would change if we change the input size to the model?
I did measure the number of parameters with the following command
pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

Also, I have tried this snippet, and the number of parameters did not change for different input size
import torchvision.models as models
model= models.resnet18(pretrained = False)
model.cuda()
summary(model, (1,64,64))


title: Pytorch: Mask dilation / extension

body: I wonder how to extend / dilate binary mask in pytorch? i.e. it should be something like cv2.dilate from opencv.


title: Implementing Gradcam on Siamese network?

body:
I am a beginner in deep learning and I am experimenting on siamese networks. I learnt that we can visualize any CNN network using grad cam. I want to visualize how the network is working using grad cam but I could not find any implementations related to that on the web and couldn't code it on my own too for siamese network. Can anyone help me out?
This is the network I created but don't know how to use grad cam to visualize it.
Thank you
class SiameseNetwork(nn.Module):
    def __init__(self):
        super(SiameseNetwork, self).__init__()
        self.cnn1 = nn.Sequential(
            nn.ReflectionPad2d(1),
            nn.Conv2d(1, 4, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(4),

            nn.ReflectionPad2d(1),
            nn.Conv2d(4, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),

            nn.ReflectionPad2d(1),
            nn.Conv2d(8, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),

        )

        self.fc1 = nn.Sequential(
            nn.Linear(8*200*200, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 5))

    def forward_once(self, x):
        output = self.cnn1(x)
        output = output.view(output.size()[0], -1)
        output = self.fc1(output)
        return output

    def forward(self, input1, input2):
        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)
        return output1, output2


title: PyTorch on Linux\ARM (BeagleBone Black or other)

body:
we're embarking on a project with Linux\ARM and Deep Learning. Currently prototyping with PyTorch on PC and looking for a target Linux\ARM platform which has good and straightforward support.

any recommendation on a supported Linux\ARM platform for TensorFlow or PyTorch would be much appreciated.
out current Linux\ARM reference device is BeagleBone Black. is there a supported, straightforward of installing TF\PyTorch on BBB?
same as above for OpenCV

Thanks!
FWIW, see logs of trying to install PyTorch and OpenCV on BBB with PIP. Looking for a way that will work out of the box, preferably on BBB but other Linux\ARM devices could work as well.
debian@beaglebone:~$ pip3 install torch==1.6.0 torchvision==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.6.0
Could not find a version that satisfies the requirement torch==1.6.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
No matching distribution found for torch==1.6.0
debian@beaglebone:~$ pip3 install cv2
Collecting cv2
Could not find a version that satisfies the requirement cv2 (from versions: )
No matching distribution found for cv2


title: Getting an error when inverting an integer type matrix in torch

body:
I was attempting to find the inverse of a 3x3 matrix that is composed of random integers using: torch.randint(). However, when doing so, I was getting the error: "inverse_cpu" not implemented for 'Long'
The code:
A = torch.randint(0, 10, (3, 3))
A_inv = A.inverse()
print(A @ A_inv, "\n", A_inv @ A)

I believe that A.inverse() is expecting the inverse of matrix A to also be of type integer, but it's not. Maybe we can have it such that matrix A is of type float like torch.Tensor(), or have A_inv invert it regardless. Though I'm not quite sure how to do either.
Thanks for your assistance!


title: Adding noise when using embedding layer in pytorch

body:
I'm building a generator g,  that receives a latent-code (vector of shape 100) and outputs an image. Specifically, I have 1000 MNIST images, and I want the network to learn a latent code z_i for each image x_i, such that g(z_i)=x_i (this approach is known as Generative Latent Optimization). So I've used nn.Embedding(1000,embedding_dim=100) and standard generator architecture, that receives the code from embedding and outputs an image. As for loss, I combine reconstruction loss with regularization on the embedding-vector weights.
My probelm is: I'd like to add noise to the latent-code vector before it is inserted to the generator (in order to make the latent-code compact). However I'm a beginner, and I don't know whether I should call detach() when adding the noise or not. I'm not sure of my approach entirely. I don't want to learn the scale of the noise or anything.
Here's my attempt:
class net(nn.Module):
  def __init__():
    self.embed = nn.Embedding(1000,embedding_dim=100)
    self.generator = nn.sequential( nn.Linear(100, 84), .... )
  def forward(batch_indices):
    batch_codes = self.embed(batch_indices)
    noise = torch.randn_like(batch_codes) * sigma
    noisy_batch_codes = batch_codes + noise # SHOULD THIS BE batch_codes.detach() + noise ??
    return self.generator(noisy_batch_codes)

g = net()
optim = SGD(g.parameters(), lr=0.01)
for epoch in range(num_epochs):
  for orig_images, orig_images_idx in trainloader:
    optim.zero_grad()
    output = g(orig_images_idx)
    reconstruction_loss = nn.MSELoss()(output, orig_images)
    embed_vector_weights = g.embed.weight[orig_images_idx]
    reg_loss = torch.norm(embed_vector_weights) * reg_coeff
    loss = reconstruction_loss + reg_loss
    loss.backward()
    optim.step()


title: Does anyone know why SHAP's Deep Explainer fails on ResNet-50 pretrained model?

body:
I have trained a ResNet-50 model and I am attempting to implement the explainability method SHAP. I have found this has been an issue for others in https://github.com/slundberg/shap/issues/1479.
My code closely follows the instructions in the SHAP pytorch example in https://github.com/slundberg/shap/blob/master/notebooks/image_examples/image_classification/PyTorch%20Deep%20Explainer%20MNIST%20example.ipynb:
def main():
    output_format = Pupils()

    model_path = ".../trained_model_30_epochs"
    ellipse_overlay_path = ".../ellipse_overlay.png"

    path_to_dataset = [DatasetPath(path='...',
                                   image_type='png')]

    path_to_dataset_to_explain = [DatasetPath(path='...',
                                   image_type='png')]

    dataset = RealDataset(path_to_dataset,
                          output_format.ground_truth_from_annotations,
                          transform=get_transforms(is_training=False, is_synthetic=False),
                          load_into_ram=True
                          )

    dataset_to_explain = RealDataset(path_to_dataset_to_explain,
                                     output_format.ground_truth_from_annotations,
                                     transform=get_transforms(is_training=False, is_synthetic=False),
                                     load_into_ram=True
                                     )
    dataloader = DataLoader(dataset,
                            batch_size=100,
                            shuffle=False)

    dataloader_to_explain = DataLoader(dataset_to_explain,
                            batch_size=32,
                            shuffle=False)

    resnet = models.resnet50(pretrained=True)
    resnet.fc = torch.nn.Linear(2048, 5)
    resnet.load_state_dict(torch.load(model_path))
    resnet.eval()

    batch = next(iter(dataloader))
    images = batch['image'].expand(-1, 3, -1, -1)

    # select a set of background examples to take an expectation over
    background = images[:100]  # dataset[0]['image']
    # background = background.expand(3, -1, -1).unsqueeze(0)

    batch = next(iter(dataloader_to_explain))
    test_images = batch['image'].expand(-1, 3, -1, -1) # dataset_to_explain[1]['image']
    # test_images = test_images.expand(3, -1, -1).unsqueeze(0)

    e = shap.DeepExplainer(resnet, background)
    shap_values = e.shap_values(test_images)

    shap_numpy = [np.swapaxes(np.swapaxes(s, 1, -1), 1, 2) for s in shap_values]
    test_numpy = np.swapaxes(np.swapaxes(test_images[:3].numpy(), 1, -1), 1, 2)

    # plot the feature attributions
    shap.image_plot(shap_numpy, -test_numpy)

and the traceback error is:
Traceback (most recent call last):
  File "/.../PycharmProjects/thesis/SHAP.py", line 133, in <module>
    main()
  File ".../PycharmProjects/thesis/SHAP.py", line 123, in main
    shap_values = e.shap_values(test_images)
  File ".../anaconda3/lib/python3.8/site-packages/shap/explainers/_deep/__init__.py", line 124, in shap_values
    return self.explainer.shap_values(X, ranked_outputs, output_rank_order, check_additivity=check_additivity)
  File ".../anaconda3/lib/python3.8/site-packages/shap/explainers/_deep/deep_pytorch.py", line 185, in shap_values
    sample_phis = self.gradient(feature_ind, joint_x)
  File ".../anaconda3/lib/python3.8/site-packages/shap/explainers/_deep/deep_pytorch.py", line 121, in gradient
    grad = torch.autograd.grad(selected, x,
  File ".../anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 202, in grad
    return Variable._execution_engine.run_backward(
  File ".../anaconda3/lib/python3.8/site-packages/shap/explainers/_deep/deep_pytorch.py", line 226, in deeplift_grad
    return op_handler[module_type](module, grad_input, grad_output)
  File ".../anaconda3/lib/python3.8/site-packages/shap/explainers/_deep/deep_pytorch.py", line 358, in nonlinear_1d
    grad_output[0] * (delta_out / delta_in).repeat(dup0))
RuntimeError: The size of tensor a (512) must match the size of tensor b (2048) at non-singleton dimension 1

Does anyone have any ideas what could be causing this? I have reviewed the code so many times and I cannot find where it goes wrong... There shouldn't be mismatches in Tensor sizes on a trained network architecture.
Note that I have intentionally removed any personally identifying paths :).
Many thanks for any input!


title: In PyTorch, How to COMPLETELY release gpu memory of a model after training and do other things

body:
When training, the model requires gradients and consumes much gpu memory, after training, I want to evaluate the model performance of different steps/epochs in parallel(multi-processing), where each of the subprocess loads different parameters for the model without gradients. In order to spawn as many subprocesses as there could be, I want to COMPLETELY free the gpu memory occupied by the previous model (in training phase).
I've tried del model and torch.empty_cache() as many suggested, but the nvidia-smi still shows the gpu memory is not released, which will prevent me from creating as many evaluation subprocesses as expected (raising the error of CUDA out of memory).
So anyone can help me completely delete the model from gpu? In addition, I don't mind loading the model repeatedly in each subprocess because the model in training is far bigger than the model in evaluating. Or tell me why my proposal is not realistic.


title: PyTorch not working when using Pytorch with cuda 11.1: Dataloader

body:
So, I'm running this deep neural network. And normally, when i train it with the cpu using the pytorch library WITHOUT cuda, it runs fine.
However, I noticed when I installed pytorch+cuda 11.1, whenever I try to enumerate over the Dataloader, the following error gets thrown out:
OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\Name\Desktop\Folder\lib\site-packages\torch\lib\cudnn_adv_train64_8.dll" or one of its dependencies.

But, when i use pip to uninstall pytorch+cuda and install pytorch without cuda, my script runs fine.
both versions of pytorch are 1.8.0
Does anyone know how to go about fixing this? Would love to be able to use this gpu.
Note: This is how i set up the dataloader:
    train_dataloader = DataLoader(
        train_dataset,
        batch_size=args.batch_size,
        num_workers=args.num_workers,
        shuffle=True,
        pin_memory=(device == "cuda"),
    )


im trying to work with bert but i keep getting the error RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding). I am not really sure what im doing wrong at this stage.    this is the traceback i receive:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-89-dcab947253f7> in <module>
     27
     28     # Forward pass
---> 29     loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
     30     train_loss_set.append(loss.item())
     31     # Backward pass

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\pytorch_pretrained_bert\modeling.py in forward(self, input_ids, token_type_ids, attention_mask, labels)
    987
    988     def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
--> 989         _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
    990         pooled_output = self.dropout(pooled_output)
    991         logits = self.classifier(pooled_output)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\pytorch_pretrained_bert\modeling.py in forward(self, input_ids, token_type_ids, attention_mask, output_all_encoded_layers)
    728         extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
    729
--> 730         embedding_output = self.embeddings(input_ids, token_type_ids)
    731         encoded_layers = self.encoder(embedding_output,
    732                                       extended_attention_mask,

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\pytorch_pretrained_bert\modeling.py in forward(self, input_ids, token_type_ids)
    265             token_type_ids = torch.zeros_like(input_ids)
    266
--> 267         words_embeddings = self.word_embeddings(input_ids)
    268         position_embeddings = self.position_embeddings(position_ids)
    269         token_type_embeddings = self.token_type_embeddings(token_type_ids)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127
    128     def extra_repr(self) -> str:

~\Anaconda3\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.IntTensor instead (while checking arguments for embedding)

The code im using is the following:
train_loss_set = []

# Number of training epochs (authors recommend between 2 and 4)
epochs = 10

# trange is a tqdm wrapper around the normal python range
for _ in trange(epochs, desc="Epoch"):

  # Training

  # Set our model to training mode (as opposed to evaluation mode)
  model.train()

  # Tracking variables
tr_loss = 0
nb_tr_examples, nb_tr_steps = 0, 0

  # Train the data for one epoch
for step, batch in enumerate(train_dataloader):
    # Add batch to CPU
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_input_mask, b_labels = batch
    # Clear out the gradients (by default they accumulate)
    optimizer.zero_grad()

    # Forward pass
    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
    train_loss_set.append(loss.item())
    # Backward pass
    loss.backward()
    # Update parameters and take a step using the computed gradient
    optimizer.step()

    # Update tracking variables
    tr_loss += loss.item()
    nb_tr_examples += b_input_ids.size(0)
    nb_tr_steps += 1

print("Train loss: {}".format(tr_loss/nb_tr_steps))

  # Validation

  # Put model in evaluation mode to evaluate loss on the validation set
model.eval()

  # Tracking variables
eval_loss, eval_accuracy = 0, 0
nb_eval_steps, nb_eval_examples = 0, 0

  # Evaluate data for one epoch
for batch in validation_dataloader:
    # Add batch to cpu
    batch = tuple(t.to(device) for t in batch)
    # Unpack the inputs from our dataloader
    b_input_ids, b_input_mask, b_labels = batch
    # Telling the model not to compute or store gradients, saving memory and speeding up validation
    with torch.no_grad():
      # Forward pass, calculate logit predictions
        logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)

        # Move logits and labels to CPU
    logits = logits.detach().cpu().numpy()
    label_ids = b_labels.to('cpu').numpy()

    tmp_eval_accuracy = flat_accuracy(logits, label_ids)

    eval_accuracy += tmp_eval_accuracy
    nb_eval_steps += 1

print("Validation Accuracy: {}".format(eval_accuracy/nb_eval_steps))


title: Dice Loss and samples containing no data in target tensors?

body:
I have a highly imbalanced 3D dataset, where about 80% of the volume is background data, I am only interested in the foreground elements which constitute about 20% of the total volume at random locations. These locations are noted in the label tensor given to the network. The target tensor is binary where 0 represents the background and 1 represents the areas we are interested in or want to segment.

The size of each volume is [30,512,1024]. I am iterating over each volume using blocks of size [30,64,64]. Thus most of my blocks have only 0 values in the target tensor.

I read that DiceLoss is perfect for such problems and is used successfully in segmenting 3D MRI scans. One simple implementation is from here: https://github.com/pytorch/pytorch/issues/1249#issuecomment-305088398

def dice_loss(input, target):
    smooth = 1.

    iflat = input.view(-1)
    tflat = target.view(-1)
    intersection = (iflat * tflat).sum()

    return 1 - ((2. * intersection + smooth) /
              (iflat.sum() + tflat.sum() + smooth))

This is not working for me, I mean for a patch where all I have are background the tflat.sum() would be 0. This would make intersection 0 as well, thus for majority of my patches or blocks I will get a return of 1.

Is this right? This is not how it is supposed to work. But I am struggling with this as this is my network output:

idx:  0 of  312 - Training Loss:  1.0 - Training Accuracy:  3.204042239857152e-11
idx:  5 of  312 - Training Loss:  0.9876335859298706 - Training Accuracy:  0.0119545953348279
idx:  10 of  312 - Training Loss:  1.0 - Training Accuracy:  7.269467666715101e-11
idx:  15 of  312 - Training Loss:  0.7320756912231445 - Training Accuracy:  0.22638492286205292
idx:  20 of  312 - Training Loss:  0.3599294424057007 - Training Accuracy:  0.49074622988700867
idx:  25 of  312 - Training Loss:  1.0 - Training Accuracy:  1.0720428988975073e-09
idx:  30 of  312 - Training Loss:  1.0 - Training Accuracy:  1.19782361807097e-09
idx:  35 of  312 - Training Loss:  1.0 - Training Accuracy:  1.956790285362331e-09
idx:  40 of  312 - Training Loss:  1.0 - Training Accuracy:  1.6055999862985004e-09
idx:  45 of  312 - Training Loss:  1.0 - Training Accuracy:  7.580232552761856e-10
idx:  50 of  312 - Training Loss:  1.0 - Training Accuracy:  9.510597864803572e-10
idx:  55 of  312 - Training Loss:  1.0 - Training Accuracy:  1.341515676323013e-09
idx:  60 of  312 - Training Loss:  0.7165247797966003 - Training Accuracy:  0.02658153884112835
idx:  65 of  312 - Training Loss:  1.0 - Training Accuracy:  4.528208030762926e-09
idx:  70 of  312 - Training Loss:  0.3205708861351013 - Training Accuracy:  0.6673439145088196
idx:  75 of  312 - Training Loss:  0.9305377006530762 - Training Accuracy:  2.3437689378624782e-05
idx:  80 of  312 - Training Loss:  1.0 - Training Accuracy:  5.305786885401176e-07
idx:  85 of  312 - Training Loss:  1.0 - Training Accuracy:  4.0612556517771736e-07
idx:  90 of  312 - Training Loss:  0.8207412362098694 - Training Accuracy:  0.0344742126762867
idx:  95 of  312 - Training Loss:  0.7463213205337524 - Training Accuracy:  0.19459737837314606
idx:  100 of  312 - Training Loss:  1.0 - Training Accuracy:  4.863646818620282e-09
idx:  105 of  312 - Training Loss:  0.35790306329727173 - Training Accuracy:  0.608722984790802
idx:  110 of  312 - Training Loss:  1.0 - Training Accuracy:  3.3852198821904267e-09
idx:  115 of  312 - Training Loss:  1.0 - Training Accuracy:  1.5268487585373691e-09
idx:  120 of  312 - Training Loss:  1.0 - Training Accuracy:  3.46353523639209e-09
idx:  125 of  312 - Training Loss:  1.0 - Training Accuracy:  2.5878148582347826e-11
idx:  130 of  312 - Training Loss:  1.0 - Training Accuracy:  2.3601216467272756e-11
idx:  135 of  312 - Training Loss:  1.0 - Training Accuracy:  1.1504343033763575e-09
idx:  140 of  312 - Training Loss:  0.4516671299934387 - Training Accuracy:  0.13879922032356262

I dont think the network is learning anything from this..

Now I'm confused, as my problem should not be too complex as I am sure MRI scans have target tensors as well where majority of them signify background.. What am I doing wrong?

Thanks


title: Pre-trained VGG-16 in MATLAB and PyTorch Same?

body: I have been trying to figure this out for some time but i am still a bit unsure. Does the PyTorch pre-trained VGG-16 (torchvision model) have exactly the same weights as the MATLAB pre-trained VGG-16?


title: Update Tensor at index in a tf.while_loop

body:
Ok so I want to replicate this keep[count] = i Pytorch code in Tensorflow, this is happening within a tf.while_loop both count and i are tensors.

keep = tf.Variable(tf.zeros(tf.size(scores), tf.int64))
count = 0
idx = tf.argsort(scores, axis=0)#scores.sort(0)  # sort in ascending order
idx = idx[-top_k:]
...

def loop_body(idx, keep, count, ...)
    i = idx[-1]  # index of current largest val
    keep = tf.scatter_update(keep, count, i)

tf.while_loop(loop_cond, loop_body, loop_vars)

This is what I'm trying to do keep = tf.scatter_update(keep, count, i), this unfortunately raises a AttributeError: 'Tensor' object has no attribute '_lazy_read


title: How to save cuda memory with torch in this case?

body:
Just as the code below, I am trying to get the mean of 47 repeated outputs of the model. But it is always out of memory. If I remove the z_proto_class_list.append(z_proto_class), then it is fine. I guess it is because the memory is freed if I don't append the tensor. I am always trying to generate the 47 output at once but it is obviously more memory consuming than my current choice. Is there a method to solve my current issue? Thank you.

        z_proto_class_list = []
        for support_input_ids, support_input_mask, support_segment_ids in dataloader:
            s_z, s_pooled_output = model(support_input_ids, support_input_mask, support_segment_ids, output_all_encoded_layers=False)
            sz_dim = s_z.size(-1)
            index = torch.LongTensor(support_idx_list).unsqueeze(1).unsqueeze(2).expand(len(support_idx_list),1,sz_dim).cuda()
            z_proto_raw = torch.gather(s_z,1,index)
            z_proto_class = z_proto_raw.view(1,n_support, sz_dim).mean(1)
            z_proto_class_list.append(z_proto_class)
            torch.cuda.empty_cache()
        z_proto = torch.cat(z_proto_class_list, 0)


title: TypeError when adding cuda device

body:
I'm running a simple demo of Pytorch 1.0, and get stuck when trying cuda settings.(vscode 1.33.1, Python 3.6)

My pytorch code is as followed.

import torch
from torch import cuda
if cuda.is_available():
    devic=cuda.device(0)
    layer=torch.rand([5,3,2],requires_grad=True)

Everything worked fine...But when I tried to add cuda device

layer=torch.rand([5,3,2],requires_grad=True,device=devic)

There raised a TypeError

Traceback (most recent call last):
  File "c:\Users\H\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\H\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\lib\python\ptvsd\__main__.py", line 410, in main
    run()
  File "c:\Users\H\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\lib\python\ptvsd\__main__.py", line 291, in run_file
    runpy.run_path(target, run_name='__main__')
  File "D:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "D:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "D:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\Users\H\Desktop\pth_test\tutorial1.py", line 25, in <module>
    layer1=torch.rand([5,3,2],requires_grad=True,device=devic)
TypeError: rand() received an invalid combination of arguments - got (list, requires_grad=bool, device=device), but expected one of:
 * (tuple of ints size, torch.Generator generator, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool requires_grad)
 * (tuple of ints size, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool requires_grad)

Changing rand() to randn() affected nothing, While empty() and zeros() raised another TypeError

Traceback (most recent call last):
  File "c:\Users\H\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\H\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\lib\python\ptvsd\__main__.py", line 410, in main
    run()
  File "c:\Users\H\.vscode\extensions\ms-python.python-2019.4.11987\pythonFiles\lib\python\ptvsd\__main__.py", line 291, in run_file
    runpy.run_path(target, run_name='__main__')
  File "D:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "D:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "D:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "c:\Users\H\Desktop\pth_test\tutorial1.py", line 25, in <module>
    layer1=torch.empty([5,3,2],requires_grad=True,device=devic)
TypeError: empty(): argument 'device' must be torch.device, not device

Things are out of control :( Any help will be appreciated


title: Is there a way to create a graph comparing hyper-parameters vs model accuracy with TRAINS python package?

body:
I would like to run multiple experiments, then report model accuracy per experiment.

I'm training a toy MNIST example with pytorch (v1.1.0), but the goal is, once I can compare performance for the toy problem, to have it integrated with the actual code base.

As I understand the TRAINS python package, with the "two lines of code" all my hyper-parameters are already logged (Command line argparse in my case).

What do I need to do in order to report a final scalar and then be able to sort through all the different training experiments (w/ hyper-parameters) in order to find the best one.

What I'd like to get, is a graph/s where on the X-axis I have hyper-parameter values and on the Y-axis I have the validation accuracy.


title: Python (Pytorch) - Looping and logging

body:
I am very new to Python and would need some help on looping ANN with different number of epoch.

The code starts with

import sys
from tqdm import tqdm
import numpy as np

import torch
import torch.nn.functional as F

try:

and the epoch part

for t in range(2000):
prediction = net(x)

May I ask what should I add (before and after the current code) in order to loop the programme for 3 times, then add 1000 epoch and run again till epoch reaches 10000?

Thanks very much.

Also, I would like to know after looping, how to keep the log of the results?

loss_func = torch.nn.MSELoss()


title: Why I am getting this error when I train the data?

body:
Why I am getting this error when I run train data?
This is my train code and I am facing the error in loss = criterion(output, labels)
I don't know why I am facing this error

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.01, momentum=0.9)

train_accuracies = []
train_loss = []
predictions = []
for epoch in range(5):
    iterations = 0
    running_loss = 0
    for i,(inputs,labels) in enumerate(train_loader):

        iterations+=1

        inputs = inputs.float()
        labels = labels.long()

        # Feed Forward
        output = net(inputs)
        # Loss Calculation
        loss = criterion(output, labels)

        running_loss = running_loss + loss.item()
        #running_loss = running_loss + loss.tolist()
        _, prd = torch.max(output, dim = 1)
        predictions.append(prd.item())
        #predictions.extend(prd.tolist())
        accuracy = (prd == labels).float().mean()
        train_accuracies.append(accuracy.item())
        #train_accuracies.append(accuracy.tolist())
        train_loss.append(running_loss / iterations)

        #i = i.view(i.shape[0], -1)

        # Clear the gradient buffer (we don't want to accumulate gradients)
        optimizer.zero_grad()
        # Backpropagation
        loss.backward()
        # Weight Update: w <-- w - lr * gradient
        optimizer.step()

        #print("Epoch [{}][{}/{}], Loss: {:.3f}".format(epoch, i, len(train_loader), running_loss / iterations))
        print("Epoch [{}][{}/{}], Loss: {:.3f}".format(epoch ,i , len(train_loader), running_loss))

And the error showing me is :

RuntimeError                              Traceback (most recent call last)
<ipython-input-76-4f34dec75c72> in <module>
     15         output = net(inputs)
     16         # Loss Calculation
---> 17         loss = criterion(output, labels)
     18
     19         running_loss = running_loss + loss.item()

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at C:\w\1\s\tmp_conda_3.7_055457\conda\conda-bld\pytorch_1565416617654\work\aten\src\THNN/generic/ClassNLLCriterion.c:94

  Any idea about this?


title: pytorch acess of weights and biases from a spcecific neuron

body:
e.g:

input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))

i want to acess all the weights and the bias of the N-TH neuron in a specific layer. I know that model.layer[1].weight gives the acess to all weights in a layer, but i also want to know of what neuron this weight is.


title: restore the hidden state of a LSTM network in pytorch

body:
I am using a LSTM in my model and my forward function is defined as:

self.rnn = nn.LSTM(...)

def forward(self, input, hidden):
   #  dropout, linear embedding
    emb = self.drop(self.encoder(input.contiguous().view(-1, self.num_in_features)))
    emb = emb.view(-1, input.size(1), self.rnn_hid_size)

    output, hidden = nn.LSTM(emb, hidden)
    output = self.drop(output)
    # [(seq_len x batch_size) * feature_size]
    decoded = self.decoder(output.view(output.size(0) * output.size(1), output.size(2)))
    # [ seq_len, batch_size, feature_size]
    decoded = decoded.view(output.size(0), output.size(1), decoded.size(1))

    return decoded, hidden, output

I am doing some inference as follows:

model.eval()
hidden = model.init_hidden(1)

with torch.no_grad():
    for i in range(end_point):
        if i >= start_point:
            out, hidden, _ = model.forward(out, hidden)
       else:
            out, hidden, _ = model.forward(my_input_seq[i].unsqueeze(0), hidden)

Now after I run this statement

out, hidden, _ = model.forward(out, hidden)

After I get the output, I want to undo this statement i.e. restore the LSTM state to before the call. I am guessing this would mean somehow undoing or restoring the hidden state to before the call. What would be a fast (and hopefully easy) way to achieve this in pytorch?


title: when I train my network ,It got an errror, Expected target size (4, 224), got torch.Size([4, 224, 224])

body:
In my train.py

criteon = nn.CrossEntropyLoss()
loss = criteon(binary_output_c1,labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()

both binary_output_c1,labels size are [4, 224,224], 4 means batchsize, 224 means h and w.
and it got error like this

Traceback (most recent call last):
  File "D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-78553e2886de>", line 1, in <module>
    runfile('F:/experiment_code/U-net/train.py', wdir='F:/experiment_code/U-net')
  File "D:\pycharm\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "D:\pycharm\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "F:/experiment_code/U-net/train.py", line 77, in <module>
    loss = criteon(binary_output_c1,labels)
  File "D:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 942, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "D:\Anaconda3\lib\site-packages\torch\nn\functional.py", line 2056, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "D:\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1881, in nll_loss
    out_size, target.size()))
ValueError: Expected target size (4, 224), got torch.Size([4, 224, 224])

I don't know if it is possible to use 3D tensor in crossentropyloss,this network use to Semantic segmentation

I make my label'size to [4,256,224,224] where 256 is the number of classes. code is here

model.train()
outputs = model(imgs)   # output  B * C * H *W
output_c1 = outputs[:,1,:,:] # 2 channels ,I choose the second channel
Rounding_output_c1 = torch.round(output_c1)
labelss =  torch.stack([(labels == i).long() for i in range(256)])
labelss = labelss.permute(1,0,2,3)
Rounding_output_c11 = torch.stack([(Rounding_output_c1 == i).float() for i in range(256)])
Rounding_output_c11 = Rounding_output_c11.permute(1,0,2,3)
loss = criteon(Rounding_output_c11,labelss)
optimizer.zero_grad()
loss.backward()

It get error too

Traceback (most recent call last):
  File "F:/experiment_code/U-net/train_2.py", line 76, in <module>
    loss = criteon(Rounding_output_c11,labelss)
  File "D:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 942, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "D:\Anaconda3\lib\site-packages\torch\nn\functional.py", line 2056, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "D:\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1873, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [4, 256, 224, 224]


title: I want to change the torch nn.Conv2D to the tensorflow 2.0 Conv2D

body:
# Localization prediction convolutions (predict offsets w.r.t prior-boxes)
        self.loc_conv4_3 = nn.Conv2d(512, n_boxes['conv4_3'] * 4, kernel_size=3, padding=1) # 38 x 38 x 512
        self.loc_conv7 = nn.Conv2d(1024, n_boxes['conv7'] * 4, kernel_size=3, padding=1)# 19 x 19 x 1024
        self.loc_conv8_2 = nn.Conv2d(512, n_boxes['conv8_2'] * 4, kernel_size=3, padding=1)# 10 x 10 x 512
        self.loc_conv9_2 = nn.Conv2d(256, n_boxes['conv9_2'] * 4, kernel_size=3, padding=1) # 5 x 5 x 256
        self.loc_conv10_2 = nn.Conv2d(256, n_boxes['conv10_2'] * 4, kernel_size=3, padding=1)# 3 x 3 x 256
        self.loc_conv11_2 = nn.Conv2d(256, n_boxes['conv11_2'] * 4, kernel_size=3, padding=1)# 1 x 1 x 256

here we have the #38,x 38 x 512 in pytorch
I want to change this convolution to the tensorflow > 2.0

what I tried is here

  # Localization prediction convolution
        self.loc_conv4_3 = Conv2D(512, self.n_boxes['conv4_3'] * 4, kernel_size=3, padding=1)
        self.loc_conv7 = Conv2D(1024, self.n_boxes['conv7'] * 4, kernel_size=3, padding=1)
        self.loc_conv8_2 = Conv2D(512, self.n_boxes['conv8_2'] * 4, kernel_size=3, padding=1)
        self.loc_conv9_2 = Conv2D(256, self.n_boxes['conv9_2'] * 4, kernel_size=3, padding=1)
        self.loc_conv10_2 = Conv2D(256, self.n_boxes['conv10_2'] * 4, kernel_size=3, padding=1)
        self.loc_conv11_2 = Conv2D(256, self.n_boxes['conv11_2'] * 4, kernel_size=3, padding=1)

but get the following error

self.pred_convs = PredictionConvolutions(n_classes)   File "/home/jake/Gits/ssd_tensorflow/model.py", line 142, in __init__
self.loc_conv4_3 = Conv2D(512, self.n_boxes['conv4_3'] * 4, kernel_size=3, padding=1) TypeError: __init__() got multiple values

  for argument 'kernel_size'


title: resnet50.to() function on a non-NVIDIA GPU

body:
I am trying to convert the pretrained ResNet50 model to be used on GPU using the Pytorch function resnet50.to().

The problem is that I am using an Intel Iris Plus Graphics 655 1536 MB GPU on Mac, and I don't know what argument to pass to the function as I only found the one for NVIDIA GPUs (resnet50.to('cuda:0')).


title: How to divide Resnet50 model to two parts and then train data in order in pytorch?

body:
I tried to have get an intermediate feature map of a pre-trained Resnet50 model in pytorch. So i did like this:

resnet50_model = torchvision.models.resnet50(pretrained=True)

x = torch.ones(16, 3, 224, 224)

m0 = nn.Sequential(*list(resnet50_model.children())[:5])
m1 = nn.Sequential(*list(resnet50_model.children())[5:])
x1 = m0(x)
x2 = m1(x1)
# Do something with x1

But then the program complained that:

RuntimeError: size mismatch, m1: [32768 x 1], m2: [2048 x 1000] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:41

The error seems because the shape of x1 is incompatible with m1(). But in Resnet50 they are connected each other and there is no gap between them. Can anyone help me with that? How should I correct the code?


title: AttributeError when selecting 0s and 1s from MNIST dataset with PyTorch

body:
I am following a guide to build a model to classify between only 0s and 1s from the MNIST dataset. However, what they suggest to do to select 0/1 values is not working for my and raises an error.
This is the code I am using:
from torch.utils.data import DataLoader

mnist_train = datasets.MNIST("./data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = datasets.MNIST("./data", train=False, download=True, transform=transforms.ToTensor())

train_idx = mnist_train.train_labels <= 1
mnist_train.train_data = mnist_train.train_data[train_idx]
mnist_train.train_labels = mnist_train.train_labels[train_idx]

test_idx = mnist_test.test_labels <= 1
mnist_test.test_data = mnist_test.test_data[test_idx]
mnist_test.test_labels = mnist_test.test_labels[test_idx]

train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size = 100, shuffle=False)

And this is the output I get when I run it:
  File "<ipython-input-2-aa7f63047cd9>", line 8, in <module>
    mnist_train.train_data = mnist_train.train_data[train_idx]

AttributeError: can't set attribute

I also tried changing:
mnist_train.train_data = mnist_train.train_data[train_idx]

by:
try:
    mnist_train.train_data = mnist_train.train_data[train_idx]
except AttributeError:
    mnist_train._train_data = mnist_train.train_data[train_idx]

And adding that extra "_", in every statement of that kind seemed to solve the issue, but later on, when trying to train the model, I realized it had not selected only 0s and 1s. Any suggestions?


title: How can I use apex AMP (Automatic Mixed Precision) with model parallelism on Pytorch?

body:
My model has a few LSTMs which run out of Cuda memory when run on large sequences with one GPU. So I shifted a few components of the model to another GPU. I tried 2 things with Apex AMP:

Move the model components to another GPU before invoking amp.initialize. In this case, I get NaNs soon after first backpropagation.

First invoke amp.initialize, and then move the model components to another GPU. In this case, its like the model backpropagation runs on a single GPU. It runs out of Cuda memory.

The model training runs fine without Apex, so I suppose I am missing some step where the loss is backpropagated on both GPUs. I looked through the documentations of Apex, however, it only talks about DataParallelism, and not ModelParallelism.
Any ideas?


title: Using AWS Sagemaker for model performance without creating endpoint

body:
I've been using Amazon Sagemaker Notebooks to build a pytorch model for an NLP task.
I know you can use Sagemaker to train, deploy, hyper parameter tuning, and model monitoring.
However, it looks like you have to create an inference endpoint in order to monitor the model's inference performance.
I already have a EC2 instance setup to perform inference tasks on our model, which is currently on a development box and rather not use an endpoint to make
Is it possible to use Sagemaker to train, run hyperparam tuning and model eval without creating an endpoint.


title: Char RNN classification with batch size

body:
I'm replicating this example for a classification with a Pytorch char-rnn.
for iter in range(1, n_iters + 1):
    category, line, category_tensor, line_tensor = randomTrainingExample()
    output, loss = train(category_tensor, line_tensor)
    current_loss += loss

I see that every epoch only 1 example is taken and random. I would like that each epoch all the dataset is taken with a specific batch size of examples. I can adjust the code to do this myself but I was wondering if some flags already exist.
Thank you


title: Pytorch listed by conda but cannot import

body:
I am well aware similar questions have been asked
at least twice, but none of the answers seams to solve the issue at hand
My configuration
Windows 10.0.18363,
Anaconda 4.8.5,
Cuda 10.1.243
conda env create -n torch -y python 3.7
conda activate torch
conda install conda -y
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch -y

Here's the bug
python -c "import torch"

Traceback (most recent call last):   File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'torch'

What I tried
Verifying python and conda
where python

C:\ProgramData\Anaconda3\envs\torch\python.exe
C:\ProgramData\Anaconda3\python.exe
C:\csvn\Python25\python.exe

python -c "import site; print(site.getsitepackages())"

['C:\ProgramData\Anaconda3',
'C:\ProgramData\Anaconda3\lib\site-packages']

conda update -n base -y conda
conda update --all -y
conda init

...
No action taken.

Verifying the torch installation
conda list | findstr torch

_pytorch_select           0.1                       cpu_0
pytorch                   1.6.0           cpu_py37h538a6d7_0
torchvision               0.7.0                py37_cu102    pytorch


title: in Torchaudio=0.7.0, where is torchaudio.functional.istft?

body:
I used torchaudio.functional.isftf when the version torchaudio=0.4.0
But now, the version of torchaudio has been upgraded to 0.7.0, so I installed it, but torchaudio.functional.istft is gone.
I need to learn a deep learning model using torchaudio.functional.istft.  How do I do it??


title: Pytorch: How to unflatten/get back the network from flattened network?

body:
I am using the following function to flatten the network:
#############################################################################
# Flattening the NET
#############################################################################
def flattenNetwork(net):
    flatNet = []
    shapes = []
    for param in net.parameters():
        #if its WEIGHTS
        curr_shape = param.cpu().data.numpy().shape
        shapes.append(curr_shape)
        if len(curr_shape) == 2:
            param = param.cpu().data.numpy().reshape(curr_shape[0]*curr_shape[1])
            flatNet.append(param)
        elif len(curr_shape) == 4:
            param = param.cpu().data.numpy().reshape(curr_shape[0]*curr_shape[1]*curr_shape[2]*curr_shape[3])
            flatNet.append(param)
        else:
            param = param.cpu().data.numpy().reshape(curr_shape[0])
            flatNet.append(param)
    finalNet = []
    for obj in flatNet:
        for x in obj:
            finalNet.append(x)
    finalNet = np.array(finalNet)
    return finalNet,shapes

The above function returns all the weights as a numpy column vector finalNet and shapes (list) of the network. I want to see the effect of weight modifications on the prediction accuracy. So, I change the weights. How can I copy this modified weight vector back to the original network? Please help. Thank you.


title: Converted from pytorch to coreml model is too large

body:
I am trying to convert my pytorch(.pth) model to coreml(.mlmodel) using the coremltools. The size of my initial pth file is about 218mb and I use the following code for conversion.
model = model.UNet_ConvLSTM(config.img_channel, config.class_num).to(device)
model.load_state_dict(torch.load(config.pretrained_path))
model.eval()
input = rand_tensor = torch.rand((1, 5, 3, 128, 256), device=device)
print(input.shape)
trace = torch.jit.trace(model, input)
mlmodel = ct.convert(
    trace,
    inputs=[ct.TensorType(name="input", shape=input.shape)],
)
mlmodel.save("unetlstm.mlmodel")

The conversions is successful but the output file is about 900mb large. Why could that happen and what could I do to decrease the size of the model?


title: AllenNLP DatasetReader: only loads a single instance, instead of iterating over all instances in the training dataset

body:
I am using AllenNLP to train a hierarchical attention network model. My training dataset consists of a list of JSON objects (eg, each object in the list is a JSON object with keys := ["text", "label"]. The value associated with the text key is a list of lists, eg:
[{"text":[["i", "feel", "sad"], ["not", "sure", "i", "guess", "the", "weather"]], "label":0} ... {"text":[[str]], "label":int}]

My DatasetReader class looks like:
@DatasetReader.register("my_reader")
class TranscriptDataReader(DatasetReader):
    def __init__(self,
                 token_indexers: Optional[Dict[str, TokenIndexer]] = None,
                 lazy: bool = True) -> None:
        super().__init__(lazy)
        self._token_indexers = token_indexers or {'tokens': SingleIdTokenIndexer()}

    def _read(self, file_path: str) -> Iterator[Instance]:
        with open(file_path, 'r') as f:
            data = json.loads(f.read())
            for _,data_json in enumerate(data):
                sent_list = []
                for segment in data_json["text"]:
                    sent_list.append(self.get_text_field(segment))
                yield self.create_instance(sent_list, str(data_json["label"]))

    def get_text_field(self, segment):
        return TextField([Token(token.lower()) for token in segment],self._token_indexers)

    def create_instance(self, sent_list, label):
        label_field = LabelField(label, skip_indexing=False)
        fields = {'tokens': ListField(sent_list), 'label': label_field}
        return Instance(fields)

and in my config file, I have:
{
  dataset_reader: {
    type: 'my_reader',
  },

  train_data_path: 'data/train.json',
  validation_data_path: 'data/dev.json',

 data_loader: {
    batch_sampler: {
      type: 'bucket',
      batch_size: 10
    }
 },

I have tried (alternatively) setting the lazy param for the dataset reader to True and False.

When set to True, the model is able to train, however, I observe that only one train and one dev instance actually get loaded, when my dataset contains ~100.
When set to False, I've modified the yield line in _read to be return; however, this causes a type error in the base vocabulary class. I've also tried keeping the yield as is when set to False; in this case, no instances get loaded at all, and since the set of instances is empty, the vocabulary does not get instantiated, and the embedding class throws an error.

Would appreciate pointers, and/or tips for debugging.


title: PyTorch installation issues on MacOS through Anaconda

body:
I am trying to install PyTorch on my Macbook Pro. I had no issues installing NumPy or Matplotlib using the following commands:
conda install numpy
conda install matplotlib

When I then import those into Python console, they work correctly. However, when I try to import PyTorch I get the following error:
(myenv) $ % python
Python 3.9.1 (default, Dec 11 2020, 06:28:49)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/__init__.py", line 189, in <module>
    _load_global_deps()
  File "/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/__init__.py", line 142, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.dylib, 10): Library not loaded: @rpath/libomp.dylib
  Referenced from: /Users/jeasl/opt/anaconda3/envs/myenv/lib/python3.9/site-packages/torch/lib/libtorch_global_deps.dylib
  Reason: image not found

I have absolutely no idea what is causing this, even after looking through several forums for answers. When I got to try and reinstall PyTorch, I get this:
(myenv) $ % conda install pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

So it seems like it is all downloaded correctly - I just can't import it when in the Python console.
Any idea how to get this working correctly?


I am trying to load and train again a HuggingFace BERT model. But, I keep getting 'BertForPreTraining' object has no attribute 'train_model' error.
The model and the other files are downloaded from here: https://huggingface.co/bert-base-uncased/tree/main
I downloaded and renamed the model file to pytorch_model.bin
Here is the simple code to load and train the model.
    BERT_PATH = './input/pretrained_models/'
    config = BertConfig.from_pretrained(BERT_PATH + 'config.json')
    model = BertForPreTraining.from_pretrained(BERT_PATH, config=config)
    model.train_model(train_df)


title: Issue while using transformers package inside the docker image

body:
I am using transformers pipeline to perform sentiment analysis on sample texts from 6 different languages. I tested the code in my local Jupyterhub and it worked fine. But when I wrap it in a flask application and create a docker image out of it, the execution is hanging at the pipeline inference line and its taking forever to return the sentiment scores.

mac os catalina 10.15.7 (no GPU)
Python version : 3.8
Transformers package : 4.4.2
torch version : 1.6.0

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
classifier = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)
results = classifier(["We are very happy to show you the Transformers library.", "We hope you don't hate it."])
print([i['score'] for i in results])

The above code works fine in Jupyter notebook and it has provided me the expected result
[0.7495927810668945,0.2365245819091797]

So now if I create a docker image with flask wrapper its getting stuck at the results = classifier([input_data]) line and the execution is running forever.
My folder structure is as follows:
- src
    |-- app
         |--main.py
    |-- Dockerfile
    |-- requirements.txt

I used the below Dockerfile to create the image
FROM tiangolo/uwsgi-nginx-flask:python3.8
COPY ./requirements.txt /requirements.txt
COPY ./app /app
WORKDIR /app
RUN pip install -r /requirements.txt
RUN echo "uwsgi_read_timeout 1200s;" > /etc/nginx/conf.d/custom_timeout.conf

And my requirements.txt file is as follows:
pandas==1.1.5
transformers==4.4.2
torch==1.6.0

My main.py script look like this :
from flask import Flask, json, request, jsonify
import traceback
import pandas as pd
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

app = Flask(__name__)
app.config["JSON_SORT_KEYS"] = False

model_name = 'nlptown/bert-base-multilingual-uncased-sentiment'
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
nlp = pipeline('sentiment-analysis', model=model_path, tokenizer=model_path)

@app.route("/")
def hello():
    return "Model: Sentiment pipeline test"

@app.route("/predict", methods=['POST'])
def predict():

    json_request = request.get_json(silent=True)
    input_list = [i['text'] for i in json_request["input_data"]]

    results = nlp(input_list)         ##########  Getting stuck here
    for result in results:
        print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
    score_list = [round(i['score'], 4) for i in results]

    return jsonify(score_list)

if __name__ == "__main__":
    app.run(host='0.0.0.0', debug=False, port=80)

My input payload is of the form
{"input_data" : [{"text" : "We are very happy to show you the Transformers library."},
                 {"text" : "We hope you don't hate it."}]}

I tried looking into the transformers github issues but couldn't find one. I execution works fine even when using the flask development server but it runs forever when I wrap it and create a docker image. I am not sure if I am missing any additional dependency to be included while creating the docker image.
Thanks.


title: Loss Not Coverging in SGD

body:
I am using  PyTorch Linear Regression loss and my SGD loss is not converging.
Use case -

I am using MNIST dataset and implementing an image classification to classify handwritten digit 0 and 1.
Then using Logistic regression model :
model = nn.Linear(input_size,num_classes)
Created the custom loss function.
Training the model where in I converted the labels from 0,1 to -1,1. Convert labels from 0,1 to -1,1
Determine Loss.

  total_step = len(train_loader)
    for epoch in range(num_epochs):
       for i, (images, labels) in enumerate(train_loader):
            # Reshape images to (batch_size, input_size)
            images = images.reshape(-1, 28*28)
            #Convert labels from 0,1 to -1,1
            labels = Variable(2*(labels.float()-0.5))
            # Forward pass
            outputs = model(images)
            # we need maximum value of two class prediction
            oneout = torch.max(outputs.data,1)[0]
            loss = loss_criteria(oneout, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
           print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
               .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

Loss output: 0.8445,0.6883,0.7976,0.8133,0.8289,0.7195. As you notice the Loss is not converging.
Expected result :
0.8445,0.8289,0.8133,0.7976,0.7195,0.6883 all the way to zero....


title: Determining the result of a convolution operation

body:
Following guide from https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807 I'm attempting to calculate the number of output features using code below :

The output of :

%reset -f

import torch
import torch.nn as nn

my_tensor = torch.randn((1, 16, 12, 12), requires_grad=False)
print(my_tensor.shape)

update_1 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
print(update_1(my_tensor).shape)

is :

torch.Size([1, 16, 12, 12])
torch.Size([1, 16, 6, 6])

How is torch.Size([1, 16, 6, 6]) calculated in the context of applying formula :

(taken from https://medium.com/mlreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-e0f514068807)

Attempting to calculate the number of output features manually by applying the formula :

stride = 2
padding = 1
kernel_size = 3

# 2304 as n_in = 1 * 16 * 16 * 12

n_out = ((2304 + (2 * padding) - kernel_size) / stride) + 1

print(n_out)

prints : 1152.5

But the produced number of output features is print(1 * 16 * 6 *6) = 576. I've taken the product of 1,16,6,6 as this is the shape of the result of print(update_1(my_tensor).shape)

Update :

Based on questions below I've updated code to :

%reset -f

import torch
import torch.nn as nn
from math import floor

stride_value = 2
padding_value = 1
kernel_size_value = 3

number_channels = 3
width = 10
height = 12

my_tensor = torch.randn((1, number_channels, width, height), requires_grad=False)
print(my_tensor.shape)

update_1 = nn.Conv2d(in_channels=number_channels,
                     out_channels=16,
                     kernel_size=kernel_size_value,
                     stride=stride_value,
                     padding=padding_value)

print(update_1(my_tensor).shape)

n_out = floor((number_channels + (2 * padding_value) - kernel_size_value) / stride_value) + 1
print(n_out)

print(my_tensor.shape) produces :
torch.Size([1, 3, 10, 12])

print(update_1(my_tensor).shape) produces :
torch.Size([1, 16, 5, 6])

print(update_1(n_out).shape) produces :
2

2 does not match the number of output features in each dimension. Have I implemented the calculation correctly ?

As the number of horizontal features produces is 5 and number of vertical features produces is 6 is this formula not applicable where the number of features differ as for an image it does not make sense to have differing x and y axis values length ?


title: What does a.sub_(lr*a.grad) actually do?

body:
I am doing the course of fast-ai, SGD and I can not understand.....

This subtracts the coefficients by (learning rate * gradient)...

But why is it necessary to subtract?

here is the code:

def update():
  y_hat = x@a
  loss = mse(y_hat, y)
  if t % 10 == 0: print (loss)
  loss.backward()
  with torch.no_grad():
    a.sub_(lr * a.grad)


title: Receptive fields for feed forward network

body: I am pretty new to artificial intelligence and neural networks. I have implemented a feed-forward neural network in PyTorch for classification on the MNIST data set. Now I want to visualize the receptive fields of (a subset of) hidden neurons. But I am having some problems with understanding the concept of receptive fields and when I google it all results are about CNNs. So can anyone help me with how I could do this in PyTorch and how to interpret the results?


title: CNN to first make feature key-points prediction then classify image on the base of these keypoints using Pytorch or TensorFlow

body:
I have already trained an Image Classifier using MobileNet in Pytorch to classify between close eyes image and open eyes image and also deployed it to mobile using Tensorflow

But problem is that the dataset is not big enough and it also doesn't work when the face object is far away or zoomed out.
I want to classify face with predefined key points like following

I want to make a CNN to first make feature key-points prediction and then classify image on the base of these keypoints.

Please guide me to any research paper or guide to predict feature keypoints using CNN and classify keypoints to two classes using Deep Learning techniques. The more Deep Learning used is better

I have already read about unsupervised machine learning but it is not working for me. I want to used deeplearning and pytorch or tensorflow


title: Pytorch gradient computation

body:
I am trying to figure out how exactly the function grad works.
This is my code :

A = torch.Tensor(2, 3).uniform_(-1, 1).requires_grad_()
B = torch.Tensor(3, 1).uniform_(-1, 1).requires_grad_()
o = torch.matmul(A,B)
print("A : ", A)
print("B : ", B)
do_dinput = torch.autograd.grad(o, A, grad_outputs=torch.ones(2, 1))
print('Size do/dA :', (do_dinput[0].size()))

I was expecting torch.Size([1, 3]) to be printed, because the derivative of AB w.r.t A is B^T. However, I got torch.Size([2, 3]).

Is there something wrong with my code, or am I missing something ?


title: Pytorch, custom layer works in Sequential but not in Functional

body:
I'm using convGRU from here and it works OK when I use it in the Sequential mode but it does not with the Functional. When I say it does not work, I mean that I'm getting black predictions from the Functional, while from the Sequential the outputs are similar to the inputs. Everything else in the code remains the same.
Below is an example of what I consistently get with one and another (being the first row the target and the second the prediction)

class MyModule(nn.Module):
    def __init__(self):
        super(MyModule, self).__init__()
        self.rnn1 = ConvGRU(
            input_size=(64, 64),
            input_dim=1,
            hidden_dim=1,
            kernel_size=(3, 3),
            num_layers=1,
            dtype=dtype,
            batch_first=True,
            bias=True,
            return_all_layers=False,
        )

    def forward(self, x):
        x = self.rnn1(x)
        return x

versus

model = nn.Sequential(
    ConvGRU(
        input_size=(64, 64),
        input_dim=1,
        hidden_dim=1,
        kernel_size=(3, 3),
        num_layers=1,
        dtype=dtype,
        batch_first=True,
        bias=True,
        return_all_layers=False,
    )
)

Any idea if when programming custom layers there should be different considerations for when it is intended to use in Functional or Sequential?

Thanks!


title: question with fast.ai course lesson 8 g attribute

body:
In the course fast.ai 2019 lesson 8, there is a weird g attribute used in back propagation, which i check for torch.Tensor this attribute doesn't exist. I tried to print the value of inp.g/out.g in call method but i got AttributeError: 'Tensor' object has no attribute 'g', but i am able to obtain the inp.g/out.g value before the assignment in backward, how does this g attribute works?

class Linear():
    def __init__(self, w, b):
        self.w, self.b = w, b

    def __call__(self, inp):
        print('in lin call')
        self.inp = inp
        self.out = inp@self.w + self.b
        try:
            print('out.g', self.out.g)
        except Exception as e:
            print('out.g dne yet')
        return self.out

    def backward(self):
        print('out.g', self.out.g)
        self.inp.g = self.out.g @ self.w.t()
        self.w.g = (self.inp.unsqueeze(-1) * self.out.g.unsqueeze(1)).sum(0)
        self.b.g = self.out.g.sum(0)

link to full code from the course

-update-

i am able to figure out the self.out.g value is exact same as the cost function MSE self.inp.g but still unable to figure out how the value is passed into the last linear layer.

class MSE():
    def __call__(self, inp, targ):
        self.inp = inp
        self.targ = targ
        self.out = (inp.squeeze() - targ).pow(2).mean()
        return self.out

    def backward(self):
        self.inp.g = 2. * (self.inp.squeeze() - self.targ).unsqueeze(-1) \
                        / self.targ.shape[0]
        print('in mse backward', self.inp.g)

class Model():
    def __init__(self, w1, b1, w2, b2):
        self.layers = [Lin(w1, b1), Relu(), Lin(w2, b2)]
        self.loss = Mse()

    def __call__(self, x, targ):
        for l in self.layers:
            x = l(x)
        return self.loss(x, targ)

    def backward(self):
        self.loss.backward()
        for l in reversed(self.layers):
            l.backward()


title: Combine two tensors of same dimension to get a final output tensor using trainable weights

body:
While working on a problem related to question-answering(MRC), I have implemented two different architectures that independently give two tensors (probability distribution over the tokens). Both the tensors are of dimension (batch_size,512). I wish to obtain the final output of the form (batch_size,512). How can I combine the two tensors using trainable weights and then train the model on the final prediction?
Edit (Additional Information):
So in the forward function of my NN model, I have used BERT model to encode the 512 tokens. These encodings are 768 dimensional. These are then passed to a Linear layer nn.Linear(768,1) to output a tensor of shape (batch_size,512,1). Apart from this I have another model built on top of the BERT encodings that also yields a tensor of shape (batch_size, 512, 1). I wish to combine these two tensors to finally get a tensor of shape (batch_size, 512, 1) which can be trained against the output logits of the same shape using CrossEntropyLoss.
Please share the PyTorch code snippet if possible.


title: K means implementation with Pytorch

body:
I am trying to implement a k-means algorithm for a CNN that first of all calculate the centroids of the k-means.
I have a tensor of dims [80,1000] with the features of one layer of the CNN.
Then i randomly create a tensor of the same dims.
I calculate the euclidean dist. and take the minimum of this tensor. Then, i assign the new centroids corresponding to the index of the tensor with the features.
My problem is that i want to loop this until the centroids dont change their position but i am not very sure how i could achieve so and how to go updating the values received for the centroids.
the code is the following:

def k_means_torch(dictionary, model):
    centroids = torch.randn(len(dictionary), 1000).cuda()
    dist_centroids = torch.cdist(dictionary,centroids, p=2.0)
    (values, indices) = torch.min(dist_centroids, dim=1)
    centroids_new = dictionary[indices]
    x = False
    while (x != True) :
      print("Itera")
      dist_centroids_loop = torch.cdist(dictionary,centroids_new, p=2.0)
      (values_, indices_) = torch.min(dist_centroids_loop, dim=1)
      #idx = centroids_new == dictionary[indices_]
      #idx_ = idx.sum(1)
      if (torch.equal(centroids_new, dictionary[indices_])):
        print("yuhu se sale")
        x = True
      centroids_new = dictionary[indices_]
    return centroids_new

Any ideas are more than welcome


title: Is there a way for inference with torchtext using from_pretrained() transformers method?

body:
BertTokenizer.save_pretrained("OUTPUT_DIR")

saves vocab.txt, special_tokens_map.json and tokenizer_config.json to my output directory,
the available train model is stored as pytorch_model.bin and the config is also there.
How is it possible to use these for inference, preferably with torchtext?


title: What does PyTorch classifier output?

body:
So i am new to deep learning and started learning PyTorch. I created a classifier model with following structure.
class model(nn.Module):
def __init__(self):
    super(model, self).__init__()
    resnet = models.resnet34(pretrained=True)
    layers = list(resnet.children())[:8]
    self.features1 = nn.Sequential(*layers[:6])
    self.features2 = nn.Sequential(*layers[6:])
    self.classifier = nn.Sequential(nn.BatchNorm1d(512), nn.Linear(512, 3))

def forward(self, x):
    x = self.features1(x)
    x = self.features2(x)
    x = F.relu(x)
    x = nn.AdaptiveAvgPool2d((1,1))(x)
    x = x.view(x.shape[0], -1)
    return self.classifier(x)

So basically I wanted to classify among three things {0,1,2}. While evaluating, I passed the image it returned a Tensor with three values like below
    (tensor([[-0.1526,  1.3511, -1.0384]], device='cuda:0', grad_fn=<AddmmBackward>)

So my question is what are these three numbers? Are they probability ?
P.S. Please pardon me If I asked something too silly.


title: Arrays selection in python

body:
What is the most efficient way in python to select rows from a matrix with indices divisible by certain number (using numpy, torch or any)?
Example 1:
mat=[[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15]]
num=3
output=[[1,2,3],[10,11,12]] #Rows 0,3

Example 2:
mat=[[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15]]
num=2
output=[[1,2,3],[7,8,9],[13,14,15]] #Rows 0,2,4


title: Convert 5D tensor to 4D tensor in PyTorch

body:
In PyTorch I have a 5D tensor X of dimensions B x 9 x C x H x W. I want to convert it into a 4D tensor Y with dimensions B x 9C x H x W such that concatenation happens channel wise.
To illustrate let,
a = X[1,0,:,:,:]
b = X[1,1,:,:,:]
c = X[1,2,:,:,:]
...
i = X[1,8,:,:,:]

Then in the tensor Y, a to i should be channel wise concatenated.


title: when I run my model.py , it said OSError: no file with expected extension , but i have put the .so file under the same folder

body:
This is my code:
dll = np.ctypeslib.load_library('render_balls_so', '.')

And I get this error:

Traceback (most recent call last):   File "/home/name/Documents/PointNet/show_3d_ball.py", line 22, in
dll = np.ctypeslib.load_library('render_balls_so', '.')   File "/home/name/.local/lib/python3.6/site-packages/numpy/ctypeslib.py", line 155, in load_library
raise OSError("no file with expected extension") OSError: no file with expected extension


title: Trying to use Distributed data parallel on GANs but getting runtime error about an inplace operation

body:
I am trying to train a GAN a machine with 3GPUs using distributed data parallel.
before wrapping my model in the DDP everything works fine but when I wrap it, it givers me the following Runtime Error

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128]] is at version 5; expected version 4 instead.

I cloned every related tensor to the gradient to solve the inplace operation (if it is any) but I could not find it.
the part of code with the problem is as follow:
Tensor = torch.cuda.FloatTensor

# ----------
#  Training
# ----------

def train_gan(rank, world_size, opt):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)

    if rank == 0:
        get_dataloader(rank, opt)
    dist.barrier()
    print(f"Rank {rank}/{world_size} training process passed data download barrier.\n")

    dataloader = get_dataloader(rank, opt)

    # Loss function
    adversarial_loss = torch.nn.BCELoss()
    # Initialize generator and discriminator
    generator = Generator()
    discriminator = Discriminator()
    # Initialize weights
    generator.apply(weights_init_normal)
    discriminator.apply(weights_init_normal)

    generator.to(rank)
    discriminator.to(rank)

    generator_d = DDP(generator, device_ids=[rank])
    discriminator_d = DDP(discriminator, device_ids=[rank])

    # Optimizers
    # Since we are computing the average of several batches at once (an effective batch size of
    # world_size * batch_size) we scale the learning rate to match.
    optimizer_G = torch.optim.Adam(generator_d.parameters(), lr=opt.lr * opt.world_size, betas=(opt.b1, opt.b2))
    optimizer_D = torch.optim.Adam(discriminator_d.parameters(), lr=opt.lr * opt.world_size, betas=(opt.b1, opt.b2))

    losses = []

    for epoch in range(opt.n_epochs):
        for i, (imgs, _) in enumerate(dataloader):

            # Adversarial ground truths
            valid = Variable(Tensor(imgs.shape[0], 1).fill_(1.0), requires_grad=False).to(rank)
            fake = Variable(Tensor(imgs.shape[0], 1).fill_(0.0), requires_grad=False).to(rank)

            # Configure input
            real_imgs = Variable(imgs.type(Tensor)).to(rank)

            # -----------------
            #  Train Generator
            # -----------------

            optimizer_G.zero_grad()

            # Sample noise as generator input
            z = Variable(Tensor(np.random.normal(0, 1, (imgs.shape[0], opt.latent_dim)))).to(rank)

            # Generate a batch of images
            gen_imgs = generator_d(z)

            # Loss measures generator's ability to fool the discriminator
            g_loss = adversarial_loss(discriminator_d(gen_imgs), valid)

            g_loss.backward()
            optimizer_G.step()

            # ---------------------
            #  Train Discriminator
            # ---------------------

            optimizer_D.zero_grad()

            # Measure discriminator's ability to classify real from generated samples
            real_loss = adversarial_loss(discriminator_d(real_imgs), valid)
            fake_loss = adversarial_loss(discriminator_d(gen_imgs.detach()), fake)
            d_loss = ((real_loss + fake_loss) / 2).to(rank)

            d_loss.backward()
            optimizer_D.step()


title: How to pad a numpy 3D array (or torch tensor) with values from surrounding 3D arrays

body:
I have a 3D numpy array of shape 3,3,3 to which I want to pad 2 layers of values from arrays surrounding it spatially, so that it becomes a 5,5,5 array.

What I have done so far using torch cat function (which works the same as numpy concat) to pad the y array, is the following:
x = torch.from_numpy(np.arange(1,28,1).reshape(3,3,3))
y = torch.from_numpy(np.arange(28,55,1).reshape(3,3,3))
z = torch.from_numpy(np.arange(55,82,1).reshape(3,3,3))

torch.cat((y,z[:,:2,:]), dim=1) #To concat z+ with 2 pads
torch.cat((x[:,1:,:],y), dim=1) #To concat z- with 2 pads

torch.cat((y,z[:,:,:2]), dim=2) #To concat x+ with 2 pads
torch.cat((x[:,1:,:],y), dim=1) #To concat x- with 2 pads

torch.cat((x,z[:2,:,:]), dim=2) #To concat y+ with 2 pads
torch.cat((x[1:,:,:],y), dim=1) #To concat y- with 2 pads

But it does not give me the right values. How can I acheive this?


title: How to run LSTM on very long sequence using Truncated Backpropagation in Pytorch (lightning)?

body:
I have a very long time series I want to feed into an LSTM for classification per-frame.
My data is labeled per frame, and I know some rare events happen that influence the classification heavily ever since they occur.
Thus, I have to feed the entire sequence to get meaningful predictions.
It is known that just feeding very long sequences into LSTM is sub-optimal, since the gradients vanish or explode just like normal RNNs.

I wanted to use a simple technique of cutting the sequence to shorter (say, 100-long) sequences, and run the LSTM on each, then pass the final LSTM hidden and cell states as the start hidden and cell state of the next forward pass.
Here is an example I found of someone who did just that. There it is called "Truncated Back propagation through time". I was not able to make the same work for me.

My attempt in Pytorch lightning (stripped of irrelevant parts):
def __init__(self, config, n_classes, datamodule):
    ...
    self._criterion = nn.CrossEntropyLoss(
        reduction='mean',
    )

    num_layers = 1
    hidden_size = 50
    batch_size=1

    self._lstm1 = nn.LSTM(input_size=len(self._in_features), hidden_size=hidden_size, num_layers=num_layers, batch_first=True)
    self._log_probs = nn.Linear(hidden_size, self._n_predicted_classes)
    self._last_h_n = torch.zeros((num_layers, batch_size, hidden_size), device='cuda', dtype=torch.double, requires_grad=False)
    self._last_c_n = torch.zeros((num_layers, batch_size, hidden_size), device='cuda', dtype=torch.double, requires_grad=False)

def training_step(self, batch, batch_index):
    orig_batch, label_batch = batch
    n_labels_in_batch = np.prod(label_batch.shape)
    lstm_out, (self._last_h_n, self._last_c_n) = self._lstm1(orig_batch, (self._last_h_n, self._last_c_n))
    log_probs = self._log_probs(lstm_out)
    loss = self._criterion(log_probs.view(n_labels_in_batch, -1), label_batch.view(n_labels_in_batch))

    return loss

Running this code gives the following error:

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

The same happens if I add
def on_after_backward(self) -> None:
    self._last_h_n.detach()
    self._last_c_n.detach()

The error does not happen if I use
lstm_out, (self._last_h_n, self._last_c_n) = self._lstm1(orig_batch,)

But obviously this is useless, as the output from the current frame-batch is not forwarded to the next one.

What is causing this error? I thought detaching the output h_n and c_n should be enough.
How do I pass the output of a previous frame-batch to the next one and have torch back propagate each frame batch separately?


title: BERT tokenizer and model without transformers library

body:
I'm trying to use both BertTokenizer and BertForTokenClassification offline. The goal of the project is to use these pretrained models and apply them to my own dataset for an NLP task. Due to network and security limitations, I am not able to install the transformers library from HuggingFace. I am only able to use PyTorch.
At this point, I have downloaded and saved the following bert-base-uncased files from the HuggingFace website to a local directory:

config.json
python_model.bin
vocab.txt

I've used the transformers library before, so I'm familiar with initializing the models from local files using something like BertTokenizer.from_pretrained('/path/to/local'). However, since I'm not able to install the package and call the model classes, I don't know how to use these downloaded local files in a similar manner. How do I use these local files to use BertTokenizer and BertForTokenClassification?
I've been instructed to use the following link to implement this: https://pytorch.org/tutorials/beginner/saving_loading_models.html


title: Python loop faster than Pandas

body:
The below code will show that using python loop is faster than using Pandas. My understanding before I tested it was different. So I'm wondering am I using pandas wrongly for this operation?
The below code shows that Pandas solution is about 7 times slower:

Pandas time  0.0008931159973144531
Loop time    0.0001239776611328125

Code:

import pandas as pd
import numpy as np
import time
import torch

batch_size = 5
classes = 4
raw_target = torch.from_numpy(np.array([1, 0, 3, 2, 0]))
rows = np.array(range(batch_size))

t0 = time.time()
zeros = pd.DataFrame(0, index=range(batch_size), columns=range(classes))
zeros.iloc[[rows, raw_target.numpy()]] = 1
t1 = time.time()

print("Pandas time ", t1-t0)
t0 = time.time()

target = raw_target.numpy()
zeros = np.zeros((batch_size, classes), dtype=np.float64)
for zero, target in zip(zeros, target):
    zero[target] = 1

t1 = time.time()
print("Loop  time  ", t1-t0)

The code uses PyTorch because the actual code where the problem exists uses PyTorch.
What could be better/optimal solution to this example? The resulting matrix is:

[[0. 1. 0. 0.]
 [1. 0. 0. 0.]
 [0. 0. 0. 1.]
 [0. 0. 1. 0.]
 [1. 0. 0. 0.]]


title: Tying all values along the diagonal of a matrix in PyTorch

body:
The model I am implementing has a set of parameters a1, ..., aH that correspond to weighing previous outputs. It's realized through multiplying a matrix that looks like this:

a1  0  0  0  0 ...
a2 a1  0  0  0 ...
a3 a2 a1  0  0 ...
 :  :  :  :  :

In the current implementation, the a's are saved in a one-dimensional nn.parameter.Parameter with H entries, from which the matrix is constructed during each forward pass. The gradient of the matrix automatically propagates to the parameters via autograd.

However, this requires constructing the matrix anew every forward pass. Is there a way to have the matrix itself be the parameter but tie the weights along the main diagonal and lower subdiagonals so that it is equivalent to constructing it from the parameter vector?


I'm completely new to cntk. I recently installed cntk 2.7 (GPU version) on my pc (windows 10,i5-7200U CPU) with GeForce 940MX GPU. I'm trying to set up cntk and the faster rcnn object-detection example provided in the link below
https://docs.microsoft.com/en-us/cognitive-toolkit/object-detection-using-faster-r-cnn

I'm trying to run the toy example.

After running the install_data_and_model.py in Examples/Image/Detection/FastRCNN folder

I run the run_faster_rcnn.py Examples/Image/Detection/FasterRCNN folder

I get the following error:

Selected GPU[0] GeForce 940MX as the process wide default device.

About to throw exception 'Failed to parse Dictionary from the input stream.'
Traceback (most recent call last):
File "run_faster_rcnn.py", line 34, in
trained_model = train_faster_rcnn(cfg)

File "C:\Users\HP-PC\Anaconda3\Lib\site-packages\cntk\Examples\Image\Detection\FasterRCNN\FasterRCNN_train.py", line 291, in train_faster_rcnn
eval_model = train_faster_rcnn_e2e(cfg)

File "C:\Users\HP-PC\Anaconda3\Lib\site-packages\cntk\Examples\Image\Detection\FasterRCNN\FasterRCNN_train.py", line 314, in train_faster_rcnn_e2e
loss, pred_error = create_faster_rcnn_model(image_input, roi_input, dims_node, cfg)

File "C:\Users\HP-PC\Anaconda3\Lib\site-packages\cntk\Examples\Image\Detection\FasterRCNN\FasterRCNN_train.py", line 177, in create_faster_rcnn_model
base_model = load_model(cfg['BASE_MODEL_PATH'])

File "C:\Users\HP-PC\Anaconda3\lib\site-packages\cntk\internal\swig_helper.py", line 69, in wrapper
result = f(*args, **kwds)

File "C:\Users\HP-PC\Anaconda3\lib\site-packages\cntk\ops\functions.py", line 1721, in load_model
return Function.load(model, device, format)

File "C:\Users\HP-PC\Anaconda3\lib\site-packages\cntk\internal\swig_helper.py", line 69, in wrapper
result = f(*args, **kwds)

File "C:\Users\HP-PC\Anaconda3\lib\site-packages\cntk\ops\functions.py", line 1635, in load
return cntk_py.Function.load(str(model), device, format.value)

RuntimeError: Failed to parse Dictionary from the input stream.

[CALL STACK]
> CNTK::Internal:: UseSparseGradientAggregationInDataParallelSGD

- CNTK::operator>>

- CNTK::Function:: Load

- PyInit__cntk_py (x2)

- PyCFunction_Call

- PyEval_GetFuncDesc

- PyEval_EvalFrameEx (x2)

- PyFunction_SetAnnotations

- PyObject_Call

- PyEval_GetFuncDesc

- PyEval_EvalFrameEx (x2)

- PyEval_GetFuncDesc (x2)

Can someone help me with what the issue is all about?


title: CNN architecture for binary classification problem (Infrared images)

body:
I try to train a model for a binary classification problem with the images being infrared (temperatures) with one-channel. After converting them to three channels (by replicate the 3rd channel) I tried two CNN architecture, VGG-11, and VGG-16, but didn't manage to get a stable training(low accuracy, and after 2-10 epochs(depending on learning rate adjustment) loss freezes in some value.

Standard VGG architecture is used except from AdaptiveAvgPool2d()  which is first used in order to alleviate inputs with an arbitrary size. The input size of images is 340x340.

CrossEntropyLoss()  is used with labels [0,1] output from the aforementioned network given. Also, the model is trained from scratch(because of the data's nature).
Any idea for improving my architecture in the needs of my problem?
I haven't found many works on infrared-image classification, so any help would be highly appreciated.


title: IndexError: index 87 is out of bounds for dimension 0 with size 39

body:
I am making an image classifier and this error occurs, when I use Vgg16, DenseNet, etc. I have seen the model, after printing the model using this model = models.densenet169(pretrained=True).

Now, this is actual error -

output = model.forward(images)
---> 90     conf_matrix = confusion_matrix(output, labels, conf_matrix)
     91     p = torch.nn.functional.softmax(output, dim=1)
     92     prediction = torch.argmax(p, dim=1)

<ipython-input-11-04b29a81a4e9> in confusion_matrix(preds, labels, conf_matrix, title, cmap)
     50     preds = torch.argmax(preds, 1)
     51     for p, t in zip(preds, labels):
---> 52         conf_matrix[p, t] += 1
     53
     54     #print(conf_matrix)

This is how I implemented my model,

model = models.densenet169(pretrained=True)

for param in model.parameters():
    param.requires_grad = True

model.fc = nn.Sequential(nn.Linear(1664, 512),
                                 nn.ReLU(),
                                 nn.Dropout(0.4),
                                 nn.Linear(512,128),
                                 nn.ReLU(),
                                 nn.Dropout(0.4),
                                 nn.Linear(128,39),
                                 nn.LogSoftmax(dim=1))

and this is my confusion matrix part -

def confusion_matrix(preds, labels, conf_matrix, title='Confusion matrix', cmap=plt.cm.Blues):
    preds = torch.argmax(preds, 1)
    for p, t in zip(preds, labels):
        conf_matrix[p, t] += 1

    #print(conf_matrix)
    #plt.imshow(conf_matrix)
    TP = conf_matrix.diag()
    for c in range(n_classes):
        idx = torch.ones(n_classes).byte()
        idx[c] = 0
        TN = conf_matrix[idx.nonzero()[:,None], idx.nonzero()].sum()
        FP = conf_matrix[c, idx].sum()
        FN = conf_matrix[idx, c].sum()

        Recall = (TP[c] / (TP[c]+FN))
        precision = (TP[c] / (TP[c]+FP))
        f1 = (2 * ((precision * Recall)/(precision + Recall)))

        #print('Class {}\nTP {}, TN {}, FP {}, FN {}'.format(c, TP[c], TN, FP, FN))
        #print('Sensitivity = {}'.format(sensitivity))
        #print('Specificity = {}'.format(specificity))

    return conf_matrix

Can anyone tell me what is the issue? I am clueless about what is happening because, for ResNet, it worked fine. Other than that there should not be any errors. Thank you.


title: How do I predict using my type  upon my set of images? Python / Torchvision / PyTorch

body:
I've trained my images upon my training and validation set, but now I wish to apply my model to my testing set (which I also have the classifications stored). The only way I've seen of even processing the testing set is via the exact same mechanisms of the validation set, but this makes it an extension of the validation set rather than the testing set? There is no prediction.

model_ft, hist = train_model(model_ft, dataloaders_dict, criterion, optimizer_ft, num_epochs=num_epochs)


The code successfully works while executing using CMD , but when it is binded to *.exe file using Pyinstaller, It throws an error and stops executing (the Message is given below). I tried all possible solutions by linking the path library and other stuff too. Any tries failed, although the file caffe2_detectron_ops.dll was included in the right folder.
OS system: win10
Python 3.7
Pyinstaller3.6
Pytorch1.5.0
PyQt5

Traceback (most recent call last):
  File "site-packages\PyInstaller\loader\pyiboot01_bootstrap.py", line 151, in __init__
  File "ctypes\__init__.py", line 364, in __init__
OSError: [WinError 126] 找不到指定的模块。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "MolPy\molpy.py", line 21, in <module>
    from dgljtnn.molecule_generation import generate
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "D:\Program\PyMOL-2.3.4_121-Win64-portable-py37\PyMOL\envs\molpy\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "MolPy\dgljtnn\molecule_generation.py", line 4, in <module>
    import torch
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "D:\Program\PyMOL-2.3.4_121-Win64-portable-py37\PyMOL\envs\molpy\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\__init__.py", line 81, in <module>
  File "site-packages\PyInstaller\loader\pyiboot01_bootstrap.py", line 153, in __init__
__main__.PyInstallerImportError: Failed to load dynlib/dll 'D:\\Users\\16886\\Desktop\\molpy\\caffe2_detectron_ops.dll'. Most probably this dynlib/dll was not found when the application was frozen.
[20848] Failed to execute script molpy

Who can help me? Thanks very much.


title: pytorch versus autograd.numpy

body:
What are the big differences between pytorch and numpy, in particular, the autograd.numpy package? ( since both of them can compute the gradient automatically for you.)
I know that pytorch can move tensors to GPU, but is this the only reason for choosing pytorch over numpy?
While pytorch is well known for deep learning, obviously it can be used for almost any machine learning algorithm, its nn.Module structure is very flexible and we don't have to confine to the neural networks. (although I've never seen any neural network model written in numpy)
So I'm wondering what's the biggest difference underlying pytorch and numpy.


title: PyTorch: batching from multiple datasets

body:
I have multiple datasets that I want to use in the training. I want each batch to be from one dataset but have batches from (possibly) all of the datasets in each epoch.
Merging the datasets into one simple Dataset object and using the default Dataloader leads to having samples from different datasets in one batch.
My own guess is to have a separate Dataset object for each dataset and override the Dataloader or the sampler, but I don't know how to do it.


title: AttributeError on using multi gpu even on using ddp

body:
I am getting attribute not found error on using multi gpu. The code works fine on using a single gpu. I am also using ddp as suggested. I am using pytorch lightning.Any suggestions?
Traceback:
Traceback (most recent call last): File "/home/sohigre/STL/stl_bert_trial_lightning.py", line 245, in <module> trainer.fit(model) File "/home/sohigre/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 952, in fit self.ddp_train(task, model) File "/home/sohigre/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_data_parallel.py", line 500, in ddp_train self.optimizers, self.lr_schedulers, self.optimizer_frequencies = self.init_optimizers(model) File "/home/sohigre/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/optimizers.py", line 18, in init_optimizers optim_conf = model.configure_optimizers() File "/home/sohigre/STL/stl_bert_trial_lightning.py", line 178, in configure_optimizers total_steps = len(self.train_dataloader()) * self.max_epochs File "/home/sohigre/STL/stl_bert_trial_lightning.py", line 109, in train_dataloader return DataLoader(self.ds_train, batch_size=self.batch_size,num_workers=4) File "/home/sohigre/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 594, in __getattr__ type(self).__name__, name)) AttributeError: 'Abuse_lightning' object has no attribute 'ds_train'
Code snippet:
    # download only (not called on every GPU, just the root GPU per node)
    df = pd.read_csv(self.filename)
    self.df_train, self.df_test = train_test_split(df, test_size=0.2, random_state=self.RANDOM_SEED)
    self.df_val, self.df_test = train_test_split(self.df_test, test_size=0.5, random_state=self.RANDOM_SEED)
    self.tokenizer = BertTokenizer.from_pretrained(self.PRE_TRAINED_MODEL_NAME)
    self.ds_train = AbuseDataset(reviews=self.df_train.comment.to_numpy(), targets=self.df_train.Score.to_numpy(),
                          tokenizer=self.tokenizer,max_len=self.max_len)
    self.ds_val = AbuseDataset(reviews=self.df_val.comment.to_numpy(), targets=self.df_val.Score.to_numpy(),
                          tokenizer=self.tokenizer,max_len=self.max_len)
    self.ds_test = AbuseDataset(reviews=self.df_test.comment.to_numpy(), targets=self.df_test.Score.to_numpy(),
                          tokenizer=self.tokenizer,max_len=self.max_len)

  @pl.data_loader
  def train_dataloader(self):

    return DataLoader(self.ds_train, batch_size=self.batch_size,num_workers=4)```