title: When I run train.py with YOLACT, I get the error KeyError: 0

body:
I'm new to machine learning and program.
Now I'm trying to develop YOLACT AI using my own data.
However, when I run train.py, I get the following error and cannot learn.
What can I do to overcome this error?`
(yolact) tmori@tmori-Lenovo-Legion-Y740-15IRHg:~/yolact$ python train.py --config=can_config  --save_interval=2000

    loading annotations into memory...
    Done (t=0.00s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.00s)
    creating index...
    index created!
    Initializing weights...
    Begin training!

    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    [  0]       0 || B: 4.840 | C: 16.249 | M: 4.682 | S: 2.749 | T: 28.521 || ETA: 9:18:44 || timer: 3.352
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    [  1]      10 || B: 4.535 | C: 9.228 | M: 4.379 | S: 1.867 | T: 20.008 || ETA: 3:25:24 || timer: 0.864
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)
    /home/tmori/yolact/utils/augmentations.py:309: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      mode = random.choice(self.sample_options)

    Computing validation mAP (this may take a while)...

    Traceback (most recent call last):
      File "train.py", line 504, in <module>
        train()
      File "train.py", line 371, in train
        compute_validation_map(epoch, iteration, yolact_net, val_dataset, log if args.log else None)
      File "train.py", line 492, in compute_validation_map
        val_info = eval_script.evaluate(yolact_net, dataset, train_mode=True)
      File "/home/tmori/yolact/eval.py", line 956, in evaluate
        prep_metrics(ap_data, preds, img, gt, gt_masks, h, w, num_crowd, dataset.ids[image_idx], detections)
      File "/home/tmori/yolact/eval.py", line 427, in prep_metrics
        detections.add_bbox(image_id, classes[i], boxes[i,:],   box_scores[i])
      File "/home/tmori/yolact/eval.py", line 315, in add_bbox
        'category_id': get_coco_cat(int(category_id)),
      File "/home/tmori/yolact/eval.py", line 293, in get_coco_cat
        return coco_cats[transformed_cat_id]
    KeyError: 0

I'm trying to develop an AI that finds cans and segments them.
First I annotated only one item of "can" with labelme, and then created a COCO format json file with labelme2coco.py.
After that, I modified config.py according to "Custom Datasets" on YOLACT's GitHub and ran train.py.
My development environment is as follows.
OS:Ubuntu20.04LTS
Anaconda:4.8.3
Python: 3.6.12
Pytorch: 1.4.0
CUDA Toolkit: 10.1
cuDNN: 7.6.5


title: Replacing adaptive average 2d pooling with regular average 2d pooling

body:
I have a PyTorch model (PoolNet) that uses an adaptive average pooling layer in the following manner:
ppms = []
for ii in [1, 3, 5]:
    ppms.append(nn.Sequential(
                nn.AdaptiveAvgPool2d(ii),
                nn.Conv2d(512, 512, 1, 1, bias=False),
                nn.ReLU(inplace=True)))

self.ppms = nn.ModuleList(ppms)

I want to adapt this model for a fixed input size in order to later convert it to ONNX format. I assume an input tensor shape of 3 x 1600 x 1000 (Channels x Width x Height). So, I replaced nn.AdaptiveAvgPool2d(ii) with
nn.AvgPool2d((1000//ii, 1600//ii)).
I have also verified that the input shape to the average pooling is the input shape of the entire model - (3x1600x100)
In addition, I've verified (elsewhere) that the output tensor shapes after pooling for both versions are identical.
However, when I run torch.onnx.export, the second version crashes with an exception:

File "C:\REPOSITORIES\poolnet2TF\venv\lib\site-packages\torch\nn\modules\pooling.py", line 599, in forward
self.padding, self.ceil_mode, self.count_include_pad, self.divisor_override)
RuntimeError: Given input size: (512x63x101). Calculated output size: (512x0x0). Output size is too small

which means that contrary to my assumption, the output tensors after pooling for both versions are not the same. What's going on?


title: Pytorch Vectorize Pairwise Gaussian/Normal Density Evaluations

body:
In Pytorch, I have three tensors A, mu, and sigma, all with shape B x N x M.

I want a new tensor D of shape B x N x M x B, where every entry is the normal density of an entry of A with respect to a pair of parameters in mu and sigma. More concretely:

D[i,j,k,l] = univariate_normal_density(A[i,j,k] | mu[l,j,k], sigma[l,j,k])

In other words, each entry of mu and sigma is a mean and variance of a distribution. It doesn't matter whether D holds probabilities or log-probabilities, incidentally. Also, note that one can unfold the second and third dimensions, I think I am missing something there.

  What is the most efficient way to do this?

The most straightforward way seems to be:

Create B*N*M objects of type torch.distributions.normal, so that
N[l,j,k]=N(mu[l,j,k],sigma[l,j,k])
For each one, evaluate log_prob on A like D[i,j,k,l] = N[l,j,k].log_prob(A[i,j,k])

I can speed this up via: D[:,j,k,l] = N[l,j,k].log_prob(A[:,j,k]), but this still seems undervectorized. I cannot seem to find any built in methods that would do better though.


title: CUDA IPC Ops Not Supported On Windows? Is there a workaround?

body:
When trying to train a neural network with a custom DataLoader:

train_dataset = datasets.VectorizedDataset(x_train,y_train)
train_loader = datasets.DataLoader(shuffle=True,num_workers=4)

This immediately errors out on data loading with:

RuntimeError: cuda runtime error (71) : operation not supported at c:\a\w\1\s\tmp_conda_3.6_090826\conda\conda-bld\pytorch_1550394668685\work\torch\csrc\generic\StorageSharing.cpp:232

And this (CUDA IPC ops) is documented here:

  They are not supported on Windows. Something like doing
  multiprocessing on CUDA tensors cannot succeed, there are two
  alternatives for this.

  Don’t use multiprocessing. Set the num_worker of DataLoader to zero.
  Share CPU tensors instead. Make sure your custom DataSet returns CPU tensors.

I'm using a GTX 1070 GPU on Windows 10 via PyTorch 1.0.1 and CUDA 10.0.

Is it literally not possible to offload CUDA tensor data loading to multiple CPUs on Windows machines in Pytorch? Is there a workaround? Do we have to do something more complicated like write our own multithreading? Or are we forced to keep all data loading on the main process?

Thanks for the help.


title: Changing a layer inside torch.nn.Sequential wrapper

body:
I was working with a transfer learning task.

Inadvertently, to make things easier, I put all the last layers in a torch.nn.Sequential wrapper like this-

self.fc=nn.Sequential(
            nn.Linear(24*24*64,80),
            nn.ReLU(True),
            nn.Linear(80,964),
        )

Now what I wanted to do is to change the last 80-unit linear layer with an identity mapping. I had trained this network and saved the weights, and I don't want to train it again (time-consuming :( ).

Is there any way I can replace the Linear layer inside ?

I know the usual outer layer replacement with model.fc1=nn.Identity(), but I just do not think this can happen here, as the last fc individual layers are not single objects, and wrapped in the Sequential layer object.

Perhaps, another hack around ? I have all the time amidst the coronavirus crisis :), any other solution would do ?


title: Deploy a customized, trained pytorch Bert model that runs on gpu for inference

body:
Thank you for taking the time to read the questions.
I want to seek for some advice to deploy a customized, trained pytorch Bert model that runs on gpu for inference (no need to train, model has saved as .pt file).
I am searching different documentations on AWS and found links like below:
https://github.com/aws/amazon-sagemaker-examples/tree/master/advanced_functionality/scikit_bring_your_own
https://github.com/aws/amazon-sagemaker-examples/tree/master/advanced_functionality/pytorch_extending_our_containers
https://github.com/aws-samples/amazon-sagemaker-bert-pytorch
First I don’t know if it’s necessary to create my container if I want to schedule daily automatic batch inference. Because for the last link i put it there, they didn't even create a container.
If yes – I’ve tried to follow the tutorials to create a container file that has below structures:
Container
Dockerfile
build_and_push.sh
classification

predictor.py
serve
wsgi.py
nginx.conf
other python scripts to support the predictor.py
model (the folder that contains the save .pt file)

However, currently I am confused about several things:

There are a lot of Dockerfile examples online, some were using python, some were using ubuntu, and also some were using pytorch_training from AWS account. I picked the one from huggingface, pytorch-gpu: nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
Question: does it matter what I use as base image? Do I still need to write things like:
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/program
ENV SAGEMAKER_PROGRAM review-classification/serve
(Assuming these only works when using pytorch_training image?
I used the build_and_push.sh file to created an image on my ECR, but how do I know if it’s properly setup?
Does the serve code matter? Right now I got the serve code from one of the first link. It says you usually don't need to modify anything in the serve. However, i can tell it's setting parameters for CPU. Do I need to modify it for GPU. If so, how?
What should be my next steps?

My current Dockerfile looks like this:
# https://hub.docker.com/r/huggingface/transformers-pytorch-gpu/dockerfile
FROM nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04
# FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:1.5.0-gpu-py36-cu101-ubuntu16.04
# FROM 785573368785.dkr.ecr.us-east-1.amazonaws.com/sagemaker-inference-pytorch:1.5.0f-gpu-py3

LABEL maintainer="bqge@amazon.com"
LABEL project="product-review-models"

RUN apt update && \
    apt install -y bash \
                   build-essential \
                   git \
                   curl \
                   wget \
                   nginx \
                   ca-certificates \
                   python3 \
                   python3-pip && \
    rm -rf /var/lib/apt/lists

# Here we get all python packages.
RUN python3 -m pip install --no-cache-dir --upgrade pip && \
    python3 -m pip install --no-cache-dir \
        mkl \
        torch==1.5.0 \
        transformers==2.11.0 \
        path \
        sklearn \
        xlrd \
        spacy==2.1.0 \
        flask \
        gevent \
        gunicorn \
        pandas \
        ipython \
        spacy==2.1.0 \
        neuralcoref==4.0 && \
    python3 -m spacy download en_core_web_md

# RUN rm -f /usr/bin/python && ln -s /usr/bin/python /usr/bin/python3
# Set some environment variables. PYTHONUNBUFFERED keeps Python from buffering our standard
# output stream, which means that logs can be delivered to the user quickly. PYTHONDONTWRITEBYTECODE
# keeps Python from writing the .pyc files which are unnecessary in this case. We also update
# PATH so that the train and serve programs are found when the container is invoked.

ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV PATH="/opt/program:${PATH}"

# Set up the program in the image
# /opt/ml and all subdirectories are utilized by SageMaker, we use the /code subdirectory to store our user code.
COPY /review-classification /opt/program
WORKDIR /opt/program

# this environment variable is used by the SageMaker PyTorch container to determine our user code directory.
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/program

# this environment variable is used by the SageMaker PyTorch container to determine our program entry point
# for training and serving.
ENV SAGEMAKER_PROGRAM review-classification/serve

ENTRYPOINT ["/usr/bin/python3", "/opt/program/serve"]

And my current build_and_push.sh looks like this
#!/usr/bin/env bash

# This script shows how to build the Docker image and push it to ECR to be ready for use
# by SageMaker.

# The name of our algorithm
algorithm_name=product-review-repo

# parameters
PY_VERSION="py36"

account=$(aws sts get-caller-identity --query Account --output text)

if [ $? -ne 0 ]
then
    exit 255
fi

cd SageMaker/container

chmod +x review-classification/serve

# Get the region defined in the current configuration (default to us-west-2 if none defined)
region=$(aws configure get region)
region=${region:-us-east-1}

TAG="gpu-${PY_VERSION}"

fullname="${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:${TAG}"

# If the repository doesn't exist in ECR, create it.
aws ecr describe-repositories --repository-names ${algorithm_name} || aws ecr create-repository --repository-name ${algorithm_name}

if [ $? -ne 0 ]
then
    aws ecr create-repository --repository-name "${algorithm_name}" > /dev/null
fi

echo "---> repository done.."
# Get the login command from ECR and execute it directly
aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin ${fullname}

echo "---> logged in to account ecr.."

echo "Building image with arch=gpu, region=${region}"

# Build the docker image locally with the image name and then push it to ECR
# with the full name.

docker build  -t ${algorithm_name} .
docker tag ${algorithm_name} ${fullname}

docker push ${fullname}

And below is my serve code:
#!/usr/bin/env python

# This file implements the scoring service shell. You don't necessarily need to modify it for various
# algorithms. It starts nginx and gunicorn with the correct configurations and then simply waits until
# gunicorn exits.
#
# The flask server is specified to be the app object in wsgi.py
#
# We set the following parameters:
#
# Parameter                Environment Variable              Default Value
# ---------                --------------------              -------------
# number of workers        MODEL_SERVER_WORKERS              the number of CPU cores
# timeout                  MODEL_SERVER_TIMEOUT              60 seconds

from __future__ import print_function
import multiprocessing
import os
import signal
import subprocess
import sys

cpu_count = multiprocessing.cpu_count()

model_server_timeout = os.environ.get('MODEL_SERVER_TIMEOUT', 60)
model_server_workers = int(os.environ.get('MODEL_SERVER_WORKERS', cpu_count))

def sigterm_handler(nginx_pid, gunicorn_pid):
    try:
        os.kill(nginx_pid, signal.SIGQUIT)
    except OSError:
        pass
    try:
        os.kill(gunicorn_pid, signal.SIGTERM)
    except OSError:
        pass

    sys.exit(0)

def start_server():
    print('Starting the inference server with {} workers.'.format(model_server_workers))

    # link the log streams to stdout/err so they will be logged to the container logs
    subprocess.check_call(['ln', '-sf', '/dev/stdout', '/var/log/nginx/access.log'])
    subprocess.check_call(['ln', '-sf', '/dev/stderr', '/var/log/nginx/error.log'])

    nginx = subprocess.Popen(['nginx', '-c', '/opt/program/nginx.conf'])
    gunicorn = subprocess.Popen(['gunicorn',
                                 '--timeout', str(model_server_timeout),
                                 '-k', 'gevent',
                                 '-b', 'unix:/tmp/gunicorn.sock',
                                 '-w', str(model_server_workers),
                                 'wsgi:app'])

    signal.signal(signal.SIGTERM, lambda a, b: sigterm_handler(nginx.pid, gunicorn.pid))

    # If either subprocess exits, so do we.
    pids = set([nginx.pid, gunicorn.pid])
    while True:
        pid, _ = os.wait()
        if pid in pids:
            break

    sigterm_handler(nginx.pid, gunicorn.pid)
    print('Inference server exiting')

# The main routine just invokes the start function.

if __name__ == '__main__':
    start_server()

Thank you so much!


title: Is there an analogy for Python's array slicing in C++ (libtorch)?

body:
In Python with PyTorch, if you have an array:
torch.linspace(0, 10, 10)
you can use e.g. only the first three elements by saying
reduced_tensor = torch.linspace(0, 10, 10)[:4].
Is there an analog to the [:] array slicing in C++/libtorch? If not, how can I achieve this easily?


title: Using PyTorch for scientific computation

body:
I would like to use PyTorch as a scientific computation package. It has much to recommend it in that respect - its Tensors are basically GPU-accelerated numpy arrays, and its autograd mechanism is potentially useful for a lot of things besides neural networks.

However, the available tutorials and documentation seem strongly geared towards quickly getting people up and running using it for machine learning. Although there is lots of good information available on the Tensor and Variable classes (and I understand that material reasonably well), the nn and optim packages always seem to be introduced by example rather than by explaining the API, which makes it hard to figure out exactly what's going on.

My main question at this point is whether I can use the optim package without also using the nn package, and if so how to do so. Of course I can always implement my simulations as subclasses of nn.Module even though they are not neural networks, but I would like to understand what happens under the hood when I do this, and what benefits/drawbacks it would give for my particular application.

More broadly, I would appreciate pointers to any resource that gives more of a logical overview of the API (for nn and optim specifically), rather than just presenting examples.


title: Loading custom dataset in pytorch

body:
Normally, when we are loading data in pytorch, we do the followings

for x, y in dataloaders:
    # Do something

However, in this dataset called MusicNet, they declare their own dataset and dataloader like this

train_set = musicnet.MusicNet(root=root, train=True, download=True, window=window)#, pitch_shift=5, jitter=.1)
test_set = musicnet.MusicNet(root=root, train=False, window=window, epoch_size=50000)

train_loader = torch.utils.data.DataLoader(dataset=train_set,batch_size=batch_size,**kwargs)
test_loader = torch.utils.data.DataLoader(dataset=test_set,batch_size=batch_size,**kwargs)

Then they load the data like this

with train_set, test_set:
    for i, (x, y) in enumerate(train_loader):
        # Do something

Question 1

I don't understand why the code doesn't work without the line with train_set, test_set.

Question 2

Also, how do I access the data?

I tried

train_set.access(2560,0)

and

with train_set, test_set:
    x, y = train_set.access(2560,0)

They either give me an error message like

  KeyError Traceback (most recent call last) in
  ----> 1 train_set.access(2560,0)

  /workspace/raven_data/AMT/MusicNet/pytorch_musicnet/musicnet.py in
  access(self, rec_id, s, shift, jitter) 106 107 if self.mmap:
  --> 108 x = np.frombuffer(self.records[rec_id][0][ssz_float:int(s+scaleself.window)*sz_float],
  dtype=np.float32).copy() 109 else: 110 fid,_ = self.records[rec_id]

  KeyError: 2560

or giving me an empty x and y


title: Pytorch double image input : broken pipe with dataloader

body:
I am developing a neural network in python with pytorch in order to classify a dataset of pairs of images. So I wanted to return as an output the 2 images and the ground truth but whenever I try to use the data loader I get the error "Broken pipe".

I want it to work as a classifier so I'm using this link (pytorch classifier CIFAR10)  as an example

Here is my code :

###Defining class
class continuousImgDataset(Dataset):
  def __init__(self, tabDataset, transform=None):
        """
        Args:
            tabDataset : Contains the imported data like : Item 1 = [im1,im2,ground-truth]

            transform (callable, optional): Optional transform to be applied
                on a sample.

        """
        self.tabPairImg = tabDataset
        self.transform = transform

    def __len__(self):
        return len(self.tabPairImg)

    def __getitem__(self, idx):
        img1 = self.tabPairImg[idx][0]
        img2 = self.tabPairImg[idx][1]
        label = self.tabPairImg[idx][2]
        if self.transform:
            img1 = self.transform(img1)
            img2 = self.transform(img2)
        return img1, img2,label

transform = transforms.Compose(
                   [transforms.Scale((32,32)),
                    transforms.ToTensor(),
                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

###Make the training and testing dataset

custom_dataset = continuousImgDataset(pairImg,transform)
train_dataset = continuousImgDataset(pairImg[: int(len(custom_dataset) * .90)],transform)
test_dataset =  continuousImgDataset(pairImg[int(len(custom_dataset) * .90) : int(len(custom_dataset))],transform)

print(len(custom_dataset.tabPairImg)) #Output :22620
print(len(train_dataset.tabPairImg)) #Output : 20358
print(len(test_dataset.tabPairImg))  #Output : 2262

### Loaders

trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=4,
                                          shuffle=True, num_workers=2)

testloader = torch.utils.data.DataLoader(test_dataset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('joint','disjoint')

### Image show and Error

"""From the official website but a little modified to have 2 imgs"""
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
dataiter = iter(trainloader)              #ERROR HERE : Broken pipe
images1,images2, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images1))
imshow(torchvision.utils.make_grid(images2))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(2)))

I may not have fully understand what the dataloader was expecting in order to iterate on it. Any help would be appreciated :)

EDIT : Here is the full error :

BrokenPipeError                           Traceback (most recent call last)
<ipython-input-58-314e85cc8fbb> in <module>
      7
      8 # get some random training images
----> 9 dataiter = iter(trainloader)
     10 images1,images2, labels = dataiter.next()
     11

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __iter__(self)
    191
    192     def __iter__(self):
--> 193         return _DataLoaderIter(self)
    194
    195     def __len__(self):

C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __init__(self, loader)
    467                 #     before it starts, and __del__ tries to join but will get:
    468                 #     AssertionError: can only join a started process.
--> 469                 w.start()
    470                 self.index_queues.append(index_queue)
    471                 self.workers.append(w)

C:\ProgramData\Anaconda3\lib\multiprocessing\process.py in start(self)
    110                'daemonic processes are not allowed to have children'
    111         _cleanup()
--> 112         self._popen = self._Popen(self)
    113         self._sentinel = self._popen.sentinel
    114         # Avoid a refcycle if the target function holds an indirect

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    221     @staticmethod
    222     def _Popen(process_obj):
--> 223         return _default_context.get_context().Process._Popen(process_obj)
    224
    225 class DefaultContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    320         def _Popen(process_obj):
    321             from .popen_spawn_win32 import Popen
--> 322             return Popen(process_obj)
    323
    324     class SpawnContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj)
     87             try:
     88                 reduction.dump(prep_data, to_child)
---> 89                 reduction.dump(process_obj, to_child)
     90             finally:
     91                 set_spawning_popen(None)

C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)
     61
     62 #

BrokenPipeError: [Errno 32] Broken pipe


title: How do I merge 2D Convolutions in PyTorch?

body:
From linear algebra we know that linear operators are associative.

In the deep learning world, this concept is used to justify the introduction of non-linearities between NN layers, a phenomenon colloquially known as linear lasagna, (reference).

In signal processing this is a well known trick to optimize memory and/or runtime requirements (reference).

So merging convolutions is a very useful tool from different perspectives. How to implement it with PyTorch?


title: How to improve conditional tensor operation

body:
I have conditional operation of tensors, such as

for i:
   if check(rate[i]):
      rate[i] = reset(rate[i])

right now, i implemented as following,

rate = torch.where(check(rate), reset(rate), rate)

It turns out that even for tensor where check(rate) is false, reset(rate) is still operated, kind of like

new_rate = reset(rate)
rate = torch.where(check(rate), new_rate, rate)

I am wondering anyway i could improve the performance here, if reset(rate) is very expensive

Updated

I've tried mask method, like following,

import torch
import time
import copy

def reset(t):
    return 1.0 / torch.log(1.0 + t)

source1 = torch.FloatTensor(10000, 10000)
source1.uniform_()
source2 = copy.deepcopy(source1)

start = time.time()
mask = source1 < 0.5
source1[mask] = reset(source1[mask])
print('test1 takes: ', time.time() - start)

start = time.time()
mask = source2 < 0.5
source2 = torch.where(mask, reset(source2), source2)
print('test2 takes: ', time.time() - start)

On my machine, it shows,

test1 takes:  3.455298900604248

test2 takes:  1.8106980323791504

Any way i could improve it?


In torch.distributed, how to average gradients on different GPUs correctly?

Modified from https://github.com/seba-1511/dist_tuto.pth/blob/gh-pages/train_dist.py, the codes below can successfully make use of both GPUs (can be checked with nvidia-smi).

But one thing difficult to understand is whether the 'average_gradients' below is indeed the correct way of averaging gradients on the two models on the two GPUs. Like the codes below,  the two 'model = Net()' run with two processes are two models on two different GPUs, but and the line 'average_gradients(model)' just 'averages' gradients of the model on one GPU, not two models on the two GPUs.

The question is that is the codes below indeed a correct way for averaging gradients on the two GPU? If true, how to read, how to understand the codes? If not, what is the correct way of averaging gradients on the two models below?

import os
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from math import ceil
from random import Random
from torch.multiprocessing import Process
from torchvision import datasets, transforms
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"

class Partition(object):
    """ Dataset-like object, but only access a subset of it. """

    def __init__(self, data, index):
        self.data = data
        self.index = index
    def __len__(self):
        return len(self.index)

    def __getitem__(self, index):
        data_idx = self.index[index]
        return self.data[data_idx]

class DataPartitioner(object):
    """ Partitions a dataset into different chuncks. """
    def __init__(self, data, sizes=[0.7, 0.2, 0.1], seed=1234):
        self.data = data
        self.partitions = []
        rng = Random()
        rng.seed(seed)
        data_len = len(data)
        indexes = [x for x in range(0, data_len)]
        rng.shuffle(indexes)
        for frac in sizes:
            part_len = int(frac * data_len)
            self.partitions.append(indexes[0:part_len])
            indexes = indexes[part_len:]

    def use(self, partition):
        return Partition(self.data, self.partitions[partition])

class Net(nn.Module):
    """ Network architecture. """

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

def partition_dataset():
    """ Partitioning MNIST """
    dataset = datasets.MNIST(
        './data',
        train=True,
        download=True,
        transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307, ), (0.3081, ))
        ]))
    size = dist.get_world_size()
    bsz = int(256 / float(size))
    partition_sizes = [1.0 / size for _ in range(size)]
    partition = DataPartitioner(dataset, partition_sizes)
    partition = partition.use(dist.get_rank())
    train_set = torch.utils.data.DataLoader(
        partition, batch_size=bsz, shuffle=True)
    return train_set, bsz

def average_gradients(model):
    """ Gradient averaging. """
    size = float(dist.get_world_size())
    for param in model.parameters():
        dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM)
        param.grad.data /= size

def run(rank, size):
    """ Distributed Synchronous SGD Example """
    # print("107 size = ", size)
    # print("dist.get_world_size() = ", dist.get_world_size()) ## 2

    torch.manual_seed(1234)
    train_set, bsz = partition_dataset()
    device = torch.device("cuda:{}".format(rank))

    model = Net()
    model = model.to(device)
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

    num_batches = ceil(len(train_set.dataset) / float(bsz))
    for epoch in range(10):
        epoch_loss = 0.0
        for data, target in train_set:
            # data, target = Variable(data), Variable(target)
           # data, target = Variable(data.cuda(rank)), Variable(target.cuda(rank))
            data, target = data.to(device), target.to(device)

            optimizer.zero_grad()
            output = model(data)
            loss = F.nll_loss(output, target)
            epoch_loss += loss.item()
            loss.backward()
            average_gradients(model)
            optimizer.step()
        print('Rank ',
              dist.get_rank(), ', epoch ', epoch, ': ',
              epoch_loss / num_batches)
        # if epoch == 4:
            # from utils import module_utils
            # module_utils.save_model()

def init_processes(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_processes, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()


title: Face alignment in pytorch

body:
I am trying to do face alignment on 300W dataset. I am using ResNet50 and L1 loss for training. My code looks like this.

batch_size = 10
image_size = 128

net = torchvision.models.resnet50(pretrained=True)
num_ftrs = net.fc.in_features
net.fc = nn.Linear(num_ftrs, 136) # 136 because 68 points with 2 dim. so 136= 68*2

def train():
    device = torch.device("cuda:0" if torch.cuda.is_available() else
           "cpu")

    optimiser = optim.Adam(net.parameters(), lr=0.001,
            weight_decay=0.0005)

    criterion = L1Loss(reduction='sum')

    for epoch in range(int(0), 200000):
        for batch, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            optimiser.zero_grad()

            outputs = net(inputs).reshape(-1, 68, 2)

            loss = criterion(outputs, labels)
            loss.backward()
            optimiser.step()
            running_loss += loss.item()

            sys.stdout.write(
            '\rTrain Epoch: {} Batch {} avg_Loss_per_batch: {:.2f}
            '.format(epoch, batch, running_loss/(batch+1)))
            sys.stdout.flush()

The trainloader is with images and points. The ground-truths are shaped as (batch, 68, 2). We have 68 points on the face on 2 dimensional space.

The papers suggests that the ResNet50 should get an error of 10 (metric: pixel) for a 256*256 image with L1 loss. I am getting error around 500-800 on validation set even after 5000 epoch.

I am training images with 256*256 resolution with ground truth of 68 points such as ((x1,y1),(x2,y2)....(x68,y68)) and I have trained over 5000 epoch with many learning rates. My validation code looks like this,

def validater(load_weights=False):
    device = torch.device("cuda:0" if torch.cuda.is_available() else
          "cpu")
    net.eval()
    net.to(device)

    with torch.no_grad():
        for batch, data in enumerate(testloader, 0):
            inputs, labels = data
            inputs, labels  = inputs.to(device), labels.to(device)

            outputs = net(inputs).reshape(-1, 68, 2)

            loss = criterion(outputs, labels)

            loss2 = np.linalg.norm(labels.to('cpu') - outputs.to('cpu'))

            sys.stdout.write('\rTest Epoch: {} Batch {} total_L1_Loss:
                {:.2f} avg_L1_Loss_per_img: {:.2f} total_norm_loss:
                 {:.2f}'.format(
                0, batch, avg_loss, avg_loss/batch/batch_size,
                avg_loss2))
            sys.stdout.flush()

    print()

What is wrong with my code ?

PS: I normalise the imgs with the following code

    img = cv2.normalize(img, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)

After 4000 epoch I get outputs like this where yellow dots are ground truth and blue ones are predicted


title: Error while inferencing a LSTM model with the help of onnx-runtime . Invalid Argument Error

body:
I have exported a LSTM model from pytorch to onnx . The model takes sequences of length 200. It has hidden state size 256 , number of layers = 2.The forward function takes input size of (batches , sequencelength) as input along with a tuple consisting of hidden state and cell state. I am getting an error while inferencing the model with onnx runtime. hidden state and cell state dimensions are same.
ioio1 = np.random.rand(1,200)
ioio2 = np.zeros((2,1,256),dtype = np.float)
pred = runtime_session.run([output_name],{runtime_session.get_inputs()[0].name:ioio1,
                                          runtime_session.get_inputs()[1].name :ioio2,
                                          runtime_session.get_inputs()[2].name : ioio2})

InvalidArgument                           Traceback (most recent call last)
<ipython-input-204-3928823f661e> in <module>()
      1 pred = runtime_session.run([output_name],{runtime_session.get_inputs()[0].name:ioio1,
      2                                           runtime_session.get_inputs()[1].name :ioio2,
----> 3                                           runtime_session.get_inputs()[2].name : ioio2})

/usr/local/lib/python3.6/dist-packages/onnxruntime/capi/session.py in run(self, output_names, input_feed, run_options)
    109             output_names = [output.name for output in self._outputs_meta]
    110         try:
--> 111             return self._sess.run(output_names, input_feed, run_options)
    112         except C.EPFail as err:
    113             if self._enable_fallback:

InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (N11onnxruntime17PrimitiveDataTypeIdEE) , expected: (N11onnxruntime17PrimitiveDataTypeIlEE)


title: PyTorch DatasetLoader tensor should be a torch tensor. Got

body:
data_dir="D:\ML-ComputerVision\Datasets"
train_transforms=transforms.Compose([transforms.RandomRotation(30),
                                     transforms.RandomResizedCrop(100),
                                     transforms.RandomHorizontalFlip(),
                                    transforms.Normalize([0.5,],[0.5,]),
                                    transforms.ToTensor()])

test_transforms=transforms.Compose([transforms.Normalize([0.5,],[0.5,]),
                                    transforms.ToTensor()])

train_data=datasets.ImageFolder(data_dir + "/Train",transform=train_transforms)

test_data=datasets.ImageFolder(data_dir + "/Test",transform=test_transforms)

trainloader=torch.utils.data.DataLoader(train_data,batch_size=32,shuffle=True)

testloader=torch.utils.data.DataLoader(test_data,batch_size=32,shuffle=False)

images, labels = next(iter(trainloader)) # <-- Error line

I am getting  tensor should be a torch tensor. Got <class 'PIL.Image.Image'> error even though I have implemented transforms.ToTensor(). Any ideas how it can be fixed?


title: Codes worked fine one week ago, but keep getting error since yesterday: Fine-tuning Bert model training via PyTorch on Colab

body:
I am new to Bert. Two weeks ago I successfully ran a fine-tuning Bert model on a nlp classification task though the outcome was not brilliant. Yesterday, however, when I tried to run the same code and data, an AttributeError was always there, which says: 'str' object has no attribute 'dim'. Please know everything is on Colab and via PyTorch Transformers.
What should I do to fix it?
Here is one thing I tried when I installed transformers but turned out it did not work:
instead of
!pip install transformers ,
I tried to use previous transformers version:
!pip install --target lib --upgrade transformers==3.5.0
Any feedback will be greatly appreciated!
Please see the code and the error message as below:
Code:

train definition

# function to train the model
def train():

  model.train()

  total_loss, total_accuracy = 0, 0

  # empty list to save model predictions
  total_preds=[]

  # iterate over batches
  for step,batch in enumerate(train_dataloader):

    # progress update after every 50 batches.
    if step % 200 == 0 and not step == 0:
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(train_dataloader)))

    # push the batch to gpu
    batch = [r.to(device) for r in batch]

    sent_id, mask, labels = batch

    # clear previously calculated gradients
    model.zero_grad()

    # get model predictions for the current batch
    preds = model(sent_id, mask)

    # compute the loss between actual and predicted values
    loss = cross_entropy(preds, labels)

    # add on to the total loss
    total_loss = total_loss + loss.item()

    # backward pass to calculate the gradients
    loss.backward()

    # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    # update parameters
    optimizer.step()

    # update learning rate schedule
    # scheduler.step()

    # model predictions are stored on GPU. So, push it to CPU
    preds=preds.detach().cpu().numpy()

    # append the model predictions
    total_preds.append(preds)

  # compute the training loss of the epoch
  avg_loss = total_loss / len(train_dataloader)

  # predictions are in the form of (no. of batches, size of batch, no. of classes).
  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  #returns the loss and predictions
  return avg_loss, total_preds

training process

# set initial loss to infinite
best_valid_loss = float('inf')

# empty lists to store training and validation loss of each epoch
train_losses=[]
valid_losses=[]

#for each epoch
for epoch in range(epochs):

    print('\n Epoch {:} / {:}'.format(epoch + 1, epochs))

    #train model
    train_loss, _ = train()

    #evaluate model
    valid_loss, _ = evaluate()

    #save the best model
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'saved_weights.pt')

    # append training and validation loss
    train_losses.append(train_loss)
    valid_losses.append(valid_loss)

    print(f'\nTraining Loss: {train_loss:.3f}')
    print(f'Validation Loss: {valid_loss:.3f}')

Error message:

 Epoch 1 / 10
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-c5138ddf6b25> in <module>()
     12
     13     #train model
---> 14     train_loss, _ = train()
     15
     16     #evaluate model

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1686         if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
   1687             return handle_torch_function(linear, tens_ops, input, weight, bias=bias)
-> 1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
   1690         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'str' object has no attribute 'dim'


title: PackagesNotFoundError: The following packages are not available from current channels: pytorch

body:
I am trying to install Pytorch Library on My Windows 10, having Python Version 3.6.9 and using the following command taken from this website :https://pytorch.org/get-started/locally/#windows-package-manager

conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c
pytorch

The installation ended with the following message:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

cuda80
torchvision==0.2.1
pytorch==1.0.0

Current channels:

https://conda.anaconda.org/pytorch/win-64
https://conda.anaconda.org/pytorch/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org

and use the search bar at the top of the page.
Please require solution of the  above problem. Thank you in advance


title: Pytorch (0.4.1) can not find GPU (Nvidia V100)

body:
I have installed Pytorch(0.4.1) with CUDA 9.0 via anaconda.
I also have installed Nvidia driver (390.30).
Through nvidia-smi, I can see all GPUs(V100) with their situation.
But I always get False when calling torch.cuda.is_avaliable().
I also tried to downgrade the CUDA version from 9.0 to 8.0, but the situation was just the same.
Tried to install Pytorch again but nothing changed. Could anyone give me a hand, please?


title: Loss does not decrease for pytorch LSTM

body:
I am new to pytorch and seeking your help with the lstm implementation. I have a single layer LSTM followed by a fully connected layer and sigmoid (implementing Deep Knowledge Tracing).

The example input output pairs are as follow,
input =

[[1, 0, 0, 0],

[0, 1, 0, 0],

[0, 0, 1, 0],

[0, 0, 0, 1]]

output=
[[1, 0],

[0, 0],

[0, 1],

[-1, -1]]

I have the following code for the LSTM and expect to compute the binary cross entropy as loss. However, the loss does not decrease over epochs. I have followed several blogs to implement this and I think it is right. Is there anything wrong with the code that I have? It would be great if you could spend a couple of minutes looking at the code and help suggest if anything's wrong with it.

    class BuildModel(nn.Module):
        on_gpu = False
        def __init__(self, num_skills, batch_size = 2, lstm_units = 200):
            super(BuildModel, self).__init__()
            self.lstm_units = lstm_units
            self.batch_size = batch_size
            self.output_dim = num_skills
            self.input_dim = num_skills * 2
            self.seq_len = seq_len
            self.__build_model()

        def __build_model(self):
            self.lstm = nn.LSTM(
                    input_size = self.input_dim,
                    hidden_size = self.lstm_units,
                    num_layers = 1,
                    batch_first = True,
                    )
            self.hidden_to_skills = nn.Linear(self.lstm_units, self.output_dim)

        def init_hidden(self):
            hidden_a = torch.randn(1, self.batch_size, self.lstm_units)
            hidden_b = torch.randn(1, self.batch_size, self.lstm_units)

            if self.on_gpu:
                hidden_a = hidden_a.cuda()
                hidden_b = hidden_b.cuda()

            hidden_a = Variable(hidden_a, requires_grad=True)
            hidden_b = Variable(hidden_b, requires_grad=True)
            return (hidden_a, hidden_b)

        def forward(self, X, X_lengths):
            self.hidden = self.init_hidden()

            batch_size, seq_len, _ = X.size()

            ## X is of shape batch_size, sequence_length, num_skills*2
            X = torch.nn.utils.rnn.pack_padded_sequence(X, X_lengths, batch_first=True, enforce_sorted=False)

            X, self.hidden = self.lstm(X, self.hidden)

            X, _ = torch.nn.utils.rnn.pad_packed_sequence(X, batch_first=True)

            # Transfer data from (batch_size, seq+len, lstm_units) --> (batch_size * seq_len, lstm_units)
            X = X.contiguous()
            X = X.view(-1, X.shape[2])

            X = self.hidden_to_skills(X)

            X = torch.nn.functional.sigmoid(X)
            # return the predictions
            return X

        def loss(self, Y_hat, Y, threshold, seq_len):
            #flatten labels
            Y = Y.view(-1)

            Y_hat = Y_hat.view(-1, seq_len * self.output_dim * self.batch_size)

            mask = (Y > -1).float()
            mask_long = (Y > -1).long()
            nb_tokens = int(torch.sum(mask).item())

            Y_hat = Y_hat[range(Y_hat.shape[0])] * mask
            Y_hat = Y_hat.reshape(-1)
            Y = Y[range(Y.shape[0])] * mask_long
            Y = Y.float()
            Y_hat = (Y_hat > 0.0).float() * 1.0
            loss = torch.nn.BCELoss()
            ce_loss = loss(Y_hat, Y)

            return Variable(ce_loss, requires_grad = True)

### For training

learning_rate = .1
total_step = len(loader)
model = BuildModel(num_skills, 2, 200)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
      overall_loss = 0.0
      for i, (X, y, lengths, seq_len) in enumerate(loader):
          X = torch.from_numpy(X).float()
          y = torch.from_numpy(y).long()

          outputs = model(X, lengths)
          loss = model.loss(outputs, y, 0.5, seq_len)
          optimizer.zero_grad()
          loss.backward()
          optimizer.step()


title: training by batches leads to more over-fitting

body:
I'm training a sequence to sequence  (seq2seq) model and I have different values to train on for the input_sequence_length.

For values 10 and 15, I get acceptable results but when I try to train with 20, I get memory errors so I switched the training to train by batches but the model over-fit and the validation loss explodes, and even with the accumulated gradient I get the same behavior, so I'm looking for hints and leads to more accurate ways to do the update.

Here is my training function (only with batch section) :

    if batch_size is not None:
        k=len(list(np.arange(0,(X_train_tensor_1.size()[0]//batch_size-1), batch_size )))
        for epoch in range(num_epochs):
            optimizer.zero_grad()
            epoch_loss=0
            for i in list(np.arange(0,(X_train_tensor_1.size()[0]//batch_size-1), batch_size )): # by using equidistant batch till the last one it becomes much faster than using the X.size()[0] directly
                sequence = X_train_tensor[i:i+batch_size,:,:].reshape(-1, sequence_length, input_size).to(device)
                labels = y_train_tensor[i:i+batch_size,:,:].reshape(-1, sequence_length, output_size).to(device)
                # Forward pass
                outputs = model(sequence)
                loss = criterion(outputs, labels)
                epoch_loss+=loss.item()
                # Backward and optimize
                loss.backward()

            optimizer.step()
            epoch_loss=epoch_loss/k
            model.eval
            validation_loss,_= evaluate(model,X_test_hard_tensor_1,y_test_hard_tensor_1)
            model.train()
            training_loss_log.append(epoch_loss)
            print ('Epoch [{}/{}], Train MSELoss: {}, Validation : {} {}'.format(epoch+1, num_epochs,epoch_loss,validation_loss))

EDIT:
here are the parameters that I'm training with :

batch_size = 1024
num_epochs = 25000
learning_rate = 10e-04

optimizer=torch.optim.Adam(model.parameters(), lr=learning_rate)
criterion = nn.MSELoss(reduction='mean')


title: Why Pytorch autograd need another vector to backward instead of computing Jacobian?

body:
To perform backward in Pytorch, we can use an optional parameter y.backward(v) to compute the Jacobian matrix multiplied by v:
x = torch.randn(3, requires_grad=True)
y = x * 2

v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(v)

print(x.grad)

I think that costs the same to compute the Jacobian matrix, because each node in the AD graph which is necessary to compute the Jacobian matrix is still computed. So why not Pytorch doesn't want to give us the Jacobian matrix?


I am not sure why do I get this error:

python train.py --batch-size 20 --rnn_type GRU --cuda --gpu 1 --lr 0.0001 --mdl RNN --clip_norm 1 --opt Adam
/scratch/sjn-p2/anaconda/anaconda2/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
There are 2 CUDA devices
Setting torch GPU to 1
Using device:1
Stored Environment:['term_len', 'word_index', 'glove', 'max_len', 'train', 'dev', 'test', 'index_word']
Loaded environment
Creating Model...
Setting Pretrained Embeddings
Initialized GRU model
Starting training
Namespace(aggregation='mean', attention_width=5, batch_size=20, clip_norm=1, cuda=True, dataset='Restaurants', dev=1, dropout_prob=0.5, embedding_size=300, epochs=50, eval=1, gpu=1, hidden_layer_size=300, l2_reg=0.0, learn_rate=0.0001, log=1, maxlen=0, mode='term', model_type='RNN', opt='Adam', pretrained=1, rnn_direction='uni', rnn_layers=1, rnn_size=300, rnn_type='GRU', seed=1111, term_model='mean', toy=False, trainable=1)
========================================================================
/scratch2/debate_tweets/sentiment/pytorch_sentiment_rnn/models/rnn.py:51: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  decoded = self.softmax(decoded)
Traceback (most recent call last):
  File "train.py", line 343, in <module>
    exp.train()
  File "train.py", line 326, in train
    loss = self.train_batch(i)
  File "train.py", line 303, in train_batch
    coeff = clip_gradient(self.mdl, self.args.clip_norm)
  File "train.py", line 35, in clip_gradient
    modulenorm = p.grad.data.norm()
AttributeError: 'NoneType' object has no attribute 'data'
[jalal@goku pytorch_sentiment_rnn]$

for train.py file in https://github.com/vanzytay/pytorch_sentiment_rnn
I have followed all the steps in the readme up to here.
What do you think should be fixed?

OS: CentOS Linux release 7.4.1708 (Core)
PyTorch version: 0.3.1.post2
How you installed PyTorch (conda, pip, source): conda install -c pytorch pytorch
Python version: Python 2.7.14 |Anaconda custom (64-bit)| (default, Dec 7 2017, 17:05:42)
CUDA/cuDNN version: CUDA Version 8.0.61
GPU models and configuration: GP102 [GeForce GTX 1080 Ti], driver=nvidia latency=0
GCC version (if compiling from source): [GCC 7.2.0] on linux2


title: PyTorch UnicodeEncodeError: 'latin-1' codec can't encode character '\u22f1'

body:
I'm trying to use the PyTorch Dataset. It works fine on my laptop. But when i run it on the Server, it cannot iterate anything. When I tried to print the data in the getitem, it shows that

Traceback (most recent call last):
File "test.py", line 98, in <module>
print(fileDataSet[0])
File "/home/cjunjie/NLP/DocSummarization/dataset.py", line 32, in __getitem__
  print(abstracts)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u22f1' in position 273: ordinal not in range(256)

What I did is to read data from a file that only contains integer and comma in a PyTorch Dataset:

def __getitem__(self, index):
    abstracts = np.genfromtxt(os.path.join(self.abstract_path, self.abstract_list[index]),
                              delimiter=',').astype(int)
    abstracts_length = abstracts[:, 0].flatten()

    print(abstracts) // It works

    abstracts = torch.LongTensor(abstracts[:, 1:])

    // Error
    print(abstracts)

    articles = np.genfromtxt(os.path.join(self.article_path, self.article_list[index]),
                             delimiter=',').astype(int)
    articles_length = articles[:, 0].flatten()
    articles = torch.LongTensor(articles[:, 1:])

    return {'abstracts_data': abstracts, 'abstracts_length': abstracts_length, 'articles_data': articles,
            'articles_length': articles_length}

So it seems that there is something wrong with the LongTensor, but I don't know why it is wrong.


title: Considerations of model definitions when moving from Tensorflow to PyTorch

body:
I've just recently switched to PyTorch after getting frustrated in debugging tf and understand that it is equivalent to coding in numpy almost completely. My question is what are the permitted python aspects we can use in a PyTorch model (to be put completely on GPU) eg. if-else has to be implemented as follows in tensorflow

a = tf.Variable([1,2,3,4,5], dtype=tf.float32)
b = tf.Variable([6,7,8,9,10], dtype=tf.float32)
p = tf.placeholder(dtype=tf.float32)
ps = tf.placeholder(dtype=tf.bool)

li = [None]*5
li_switch = [True, False, False, True, True]

for i in range(5):
    li[i] = tf.Variable(tf.random.normal([5]))

sess = tf.Session()
sess.run(tf.global_variables_initializer())

def func_0():
    return tf.add(a, p)
def func_1():
    return tf.subtract(b, p)

with tf.device('GPU:0'):
    my_op = tf.cond(ps, func_1, func_0)

for i in range(5):
    print(sess.run(my_op, feed_dict={p:li[i], ps:li_switch[i]}))

How would the structure change in pytorch for the above code? How to place the variables and ops above on GPU and parallelize the list inputs to our graph in pytorch?


title: Feeding Torch objects from csv to learn

body:
I am trying to give PyTorch the input to build a very simple Neural Network. Here is my problem:
I have all the data I want to use in an csv and I am using Panda to read it.
Here is my code:

data = pd.read_csv("../myFile.csv")
input = [x for x in data]
input = np.asarray(input)
input = torch.from_numpy(input)

This returns the error:

line 42, in <module>
input = torch.from_numpy(input)
TypeError: can't convert np.ndarray of type numpy.str_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, and uint8.

I understand what the error means. The main problem is, that there are a few columns in my csv that can not be casted to int or float as they are basically Strings, for example the customer-ID would be something like: AAABBBCCC; and i can not cast that to float or int. Do you have any idea what I can do?

EDIT: Here is my new updated code with the proposed answer by Anubhav:

data = pd.read_csv("myFile.csv", names=col_names)
data = data.drop(["Customer-ID", "name"], axis=1)

for column in list(data):
    # one hot encode of Object Columns
    one_hot = pd.get_dummies(data[column])
    # drop encoded columns
    data = data.drop(column, axis=1)
    # join the encoded data
    data = data.join(one_hot)

print(data.dtypes)

inp = [x for x in data]
inp = np.asarray(inp, dtype=np.float32)
inp = torch.from_numpy(inp)

But I still get the following error:

line 52, in <module>
    inp = np.asarray(inp, dtype=np.float32)
  File "C:\Users\Paul\anaconda3\lib\site-packages\numpy\core\numeric.py", line 538, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: could not convert string to float: '01139_Lichtenau'

Looking up this value from print(data.dtypes) it clearly says:

01139_Lichtenau  uint8

Did the encoding fail for some reason?


title: Forward vs reverse mode differentiation - Pytorch

body:
In the first example of Learning PyTorch with Examples, the author demonstrates how to create a neural network with numpy. Their code is pasted below for convenience:

# from: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html
# -*- coding: utf-8 -*-
import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

What is confusing to me is why gradients of w1 and w2 are computed with respect to loss (2nd to last code block).

Normally the opposite computation happens: the gradients of loss is computed with respect to the weights, as quoted here:

"When training neural networks, we think of the cost (a value describing how bad a neural network performs) as a function of the parameters (numbers describing how the network behaves). We want to calculate the derivatives of the cost with respect to all the parameters, for use in gradient descent. Now, there’s often millions, or even tens of millions of parameters in a neural network. So, reverse-mode differentiation, called backpropagation in the context of neural networks, gives us a massive speed up!" (Colah's blog).

So my question is: why is the derivation computation in the example above in reverse order as compared to normal back propagation computations?


title: why is torch::nonzero() very slow?

body:
I executed the code below.But nonzero() function is very slow (take 0.1 sec). what is wrong my code?

torch::Tensor a = torch::randn({30000, 80});
torch::Tensor b = a.select(1, 4) > 0.2;
const torch::Tensor c = torch::nonzero(b);
.
.
.


title: using torch.multiprocessing on multiple GPUs

body:
I am trying to run inference on images in parallel using torch multiprocessing.  What's odd is I see only one GPU being used in nvtop and performance is terrible.

class Model:
   def __init__(self, nets):
       self._nets = nets or []
       self.to(device or (torch.device("cuda" if torch.cuda.is_available() else "cpu")))

   def set_device(self, ndx):
      #ndx = 'cuda:'+str(ndx) #didnt help
       #os.environ['CUDA_VISIBLE_DEVICES'] = str(ndx) #didnt help
      self._nets.to(str(ndx))

   def process_image(self, frame)
     ...

if __name__ == "__main__":
    model = Model(net)
    mp.set_start_method('spawn')
    pool = mp.Pool(2)
    pool.map(model.set_device, [0,1]) #set GPUs on processed

    pool.starmap(model.process_img,zip(frames))

The result is in nvtop I one of the processes on GPU1 with the others on 0.  However GPU 0 is doing all the work.  What is also odd is both GPUs show memory allocated but GPU 0 is twice that of GPU 1.  Any thoughts on what I have wrong here?  Thanks.


title: torch.einsum 'RuntimeError: dimension mismatch for operand 0: equation 4 tensor 2'

body:
I'm trying to manually calculate a gradient of a matrix and I can do it by using numpy but I don't know to do the same thing in pytorch.
the equation in NumPy is

def grad(A, W0, W1, X):
    dim = A.shape
    assert len(dim) == 2
    A_rows = dim[0]
    A_cols = dim[1]
    gradient = (np.einsum('ik, jl', np.eye(A_cols, A_rows), (((A).dot(X)).dot(W0)).dot(W1).T) + np.einsum('ik, jl', A, ((X).dot(W0)).dot(W1).T))
    return gradient

I wrote a function in pytorch but it's giving me an error saying 'RuntimeError: dimension mismatch for operand 0: equation 4 tensor 2'

The function I wrote using pytorch is

def torch_grad(A, W0, W1, X):
    dim = A.shape
    A_rows = dim[0]
    A_cols = dim[1]
    W0W1 = torch.mm(W0, W1)
    AX = torch.mm(A, X)
    AXW0W1 = torch.mm(AX, W0W1)
    XW0W1 = torch.mm(X, W0W1)
    print(torch.eye(A_cols, A_rows).shape, torch.t(AXW0W1).shape)
    e1 = torch.einsum('ik jl', torch.eye(A_cols, A_rows), torch.t(AXW0W1))
    e2 = torch.einsum('ik, jl', A, torch.t(XW0W1))
    return e1 + e2

I would appreciate if someone can show me how to implement the numpy code in pytorch.
Thanks!


title: How to use autograd find MIN/MAX point

body: Say we have a simple function y=sin(x**2), how do I use autograd find all X:s with the first order derivative value of 0?


title: Anaconda always want to replace my GPU Pytorch version to CPU Pytorch version when updating

body:
I have a newly installed Anaconda3 (version 2020.02) environment, and I have installed Pytorch GPU version by the command conda install pytorch torchvision cudatoolkit=10.2 -c pytorch. I have verified that my Pytorch indeed runs fine on GPU.
However, whenever I update Anaconda by conda update --all, the following messages always shows:
The following packages will be SUPERSEDED by a higher-priority channel:

  pytorch            pytorch::pytorch-1.5.0-py3.7_cuda102_~ --> pkgs/main::pytorch-1.5.0-cpu_py37h9f948e0_0

In other words, it always want to replace my GPU version Pytorch to CPU version. I have tried that if continue the update, it will install the CPU version Pytorch and my previous Pytorch code on GPU could not run anymore. I have also tried the command conda update --all --no-channel-priority but the message still shows.
To my knowledge I have never modified Anaconda channels or add custom channels. How can I get rid of this message?


title: Can I speed up inference in PyTorch using autocast (automatic mixed precision)?

body: The docs (see also this) for autocast in PyTorch only discuss training.  Does it speed things up if I also use autocast for inference?


title: Keras to Pytorch model translation and input size

body:
I am following a Keras tutorial and want to shadow it in Pytorch, so am translating. I'm not strongly familiar with either and am coming unstuck on the input size parameter especially, but also the final layer - do I need another Linear layer? Can anyone translate the following to a Pytorch sequential definition?

visible = Input(shape=(64,64,1))
conv1 = Conv2D(32, kernel_size=4, activation='relu')(visible)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
conv2 = Conv2D(16, kernel_size=4, activation='relu')(pool1)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
hidden1 = Dense(10, activation='relu')(pool2)
output = Dense(1, activation='sigmoid')(hidden1)
model = Model(inputs=visible, outputs=output)

This is the output of the model:

Layer (type)                 Output Shape              Param #
_________________________________________________________________
input_1 (InputLayer)         (None, 64, 64, 1)         0
conv2d_1 (Conv2D)            (None, 61, 61, 32)        544
max_pooling2d_1 (MaxPooling2 (None, 30, 30, 32)        0
conv2d_2 (Conv2D)            (None, 27, 27, 16)        8208
max_pooling2d_2 (MaxPooling2 (None, 13, 13, 16)        0
dense_1 (Dense)              (None, 13, 13, 10)        170
dense_2 (Dense)              (None, 13, 13, 1)         11

Total params: 8,933
Trainable params: 8,933
Non-trainable params: 0

What I have worked out lacks a specification for the shape of the input, and I am also a bit perplexed at the translation of stride in the specified Keras model as it uses stride 2 in the MaxPooling2D but doesn't specify this elsewhere - it is perhaps a toy example.

model = nn.Sequential(
      nn.Conv2d(1, 32, 4),
      nn.ReLU(),
      nn.MaxPool2d(2, 2),
      nn.Conv2d(1, 16, 4),
      nn.ReLU(),
      nn.MaxPool2d(2, 2),
      nn.Linear(10, 1),
      nn.Sigmoid(),
      )


title: Runtime error while training the model in pytorch

body:
RuntimeError: cuda runtime error (59) :

epochs = 3
print_every = 40
steps = 0

# change to cuda
model.to(device)

for e in range(epochs):
    running_loss = 0
    for ii, (inputs, labels) in enumerate(trainloader):
        steps += 1
        inputs, labels = inputs.to(device), labels.to(device)

        optimizer.zero_grad()

        # Forward and backward passes
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:
            print("Epoch: {}/{}... ".format(e+1, epochs),
                  "Loss: {:.4f}".format(running_loss/print_every))

            running_loss = 0

i am getting the below error on above code.

RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCTensorCopy.c:20

i am using one of the pretrained models from torchvision.models to get the image features. Build and train a new feed-forward classifier using those features.

and i am not sure what i am doing wrong here.
also, i am runing this code on online portal inside jupyter norebook where GPU is enabled.

Thanks in Advance for any help !!


title: Efficiently saving and loading data using h5py (or other methods)

body:
I am testing ways of efficient saving and retrieving data using h5py. But am having trouble with running time while not using up all my memory.

In my first method I simply create a static h5py file

with h5py.File(fileName, 'w') as f:
        f.create_dataset('data_X', data = X, dtype = 'float32')
        f.create_dataset('data_y', data = y, dtype = 'float32')

In the second method, I set parameter maxshape in order to append more training data in the future. (see How to append data to one specific dataset in a hdf5 file with h5py)

with h5py.File(fileName2, 'w') as f:
            f.create_dataset('data_X', data = X, dtype = 'float32',maxshape=(None,4919))
            f.create_dataset('data_y', data = y, dtype = 'float32',maxshape=(None,6))

I am using PyTorch and am set up my data loader as such:

class H5Dataset_all(torch.utils.data.Dataset):
    def __init__(self, h5_path):
        # super(dataset_h5, self).__init__()
        self.h5_path = h5_path
        self._h5_gen = None

    def __getitem__(self, index):
        if self._h5_gen is None:
            self._h5_gen = self._get_generator()
            next(self._h5_gen)
        return self._h5_gen.send(index)

    def _get_generator(self):
        with h5py.File( self.h5_path, 'r') as record:
            index = yield
            while True:
                X = record['data_X'][index]
                y = record['data_y'][index]
                index = yield X, y

    def __len__(self):
        with h5py.File(self.h5_path,'r') as record:
            length = record['data_X'].shape[0]
            return length

loader = Data.DataLoader(
        dataset=H5Dataset_all(filename),
        batch_size=BATCH_SIZE,
        shuffle=True, num_workers=0)

Having saved the same data for each of these methods I would expect them to be similar in running time, however that is not the case. The data I used has size X.shape=(200722,4919) and y.shape=(200772,6). The files are about 3.6 GB each.
I test the running time using:

import time
t0 = time.time()
for i, (X_batch, y_batch) in enumerate(loader):
    # assign a dummy value
    a = 0
t1 = time.time()-t0
print(f'time: {t1}')

For the first method the running time is 83 s and for the second it is 1216 s, which In my mind doesn't make sense. Can anyone help me figure out why?

Additionally I also tried saving/loading it as a torch file using torch.save and torch.load and passing the data to Data.TensorDataset before setting up the loader. This implementation runs significantly faster (about 3.7 s), but has the disadvantage of having to load the files before training, which could quickly be capped by my memory.

Is there a better way in which I can train somewhat fast while not using having to load all of the data before training?


title: pytorch element intersection

body:
When I calculate the Hit Ratio, I need to calculate the number of elements of predict tensor in the target tensor, I wanna calculate the number of elements in their intersection.

For example: [# of classes: 20, # of samples: 2]

target:
tensor([[14, 13,  8, 11, 18, 12,  5,  1,  0, 10],
        [ 8, 10,  2, 10,  7, 17,  6, 12, 13, 14]])

pred_idx: (HR@5):
tensor([[14, 11,  8, 19,  4],
        [ 6,  9,  8, 13, 18]])

now when I do >>> (pred_idx & target).sum((1,2))

RuntimeError: The size of tensor a (5) must match the size of tensor b (10) at non-singleton dimension 1.

But the thing is, the prediction and target have different size, how can I calculate the number of elements in the pred@5 that are also in the target?


title: Creating a stack of convolutional layers using for loop in forward function of a pytoch class for Residual block

body:
I'm defining a residual block in pytorch for ResNet in which you can input how many convolutional layers you want to have and not necessarily two. This is done through a parameter named nc (number of Convs). The first layer gets ni as the number of input nf number of filters. But from second layer on I put them in a for loop. Here's my code:

class ResBlock(nn.Module):
    def __init__(self, ni, nf,nc=2):
        super().__init__()
        self.conv1 = nn.Conv2d(ni,nf, kernel_size=3, stride=2, padding=0)
        self.conv2 = nn.Conv2d(nf,nf, kernel_size=3, stride=1, padding=0)
        self.conv1x1 = nn.Conv2d(ni, nf, kernel_size=1, stride=1, padding=0)
        self.nc = nc
    def forward(self, x):
        y = self.conv1(x)
        for i in range(self.nc-1):
            y = self.conv2(y)
            print(torch.mean(y))
        return self.conv1x1(x) + y

But no matter what value I give to nc, it always returns 2 convs with kernel size 3. I'm not sure if for loop can really do this job in pytorch but it was working when I used functional API in Keras. Could anyone help me understand what's going on?


title: How to implement a location specific convolutional filters in Tensorflow or Pytorch?

body:
I want to implement a convolutional layer, with a different convolutional filter for each output location. Specifically, think of the case when the output is of 16*16*128 (W * H * C). Instead of having a 3*3*128 filter we have 16*16 filters; each with size 3*3*128. This would lead to huge amount of parameters, but it can the case be that each of the 3*3*128 filter may be the same except scaled by a different constant, and the constants can be learned through a side network. In this way the number of parameters won't be too much.

The similar idea is briefly in Dynamic Filter Networks, but I cannot find an implementation of location specific filters. My question is, if we want a location specific convolutional filter how do I implement it in Tensorflow or Pytorch? Do I need to write my own operation or there is some smart way to use the functions provided? If I have to write an OP is there any trick that can easily achieve this idea? Any help is appreciated!


title: vgg probability doesn't add up to 1, pytorch

body:
I've trained a vgg16 model to predict 102 classes of flowers.
It works however now that I'm trying to understand one of it's predictions I feel it's not acting normally.

model layout

# Imports here
import os
import numpy as np
import torch

import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import json
from pprint import pprint
from scipy import misc

%matplotlib inline

data_dir = 'flower_data'
train_dir = data_dir + '/train'
test_dir = data_dir + '/valid'

json_data=open('cat_to_name.json').read()
main_classes = json.loads(json_data)
main_classes = {int(k):v for k,v in classes.items()}

train_transform_2 = transforms.Compose([transforms.RandomResizedCrop(224),
                                    transforms.RandomRotation(30),
                                    transforms.RandomHorizontalFlip(),
                                    transforms.ToTensor()])

test_transform_2= transforms.Compose([transforms.RandomResizedCrop(224),
                                    transforms.ToTensor()])

# TODO: Load the datasets with ImageFolder
train_data = datasets.ImageFolder(train_dir, transform=train_transform_2)
test_data = datasets.ImageFolder(test_dir, transform=test_transform_2)

# define dataloader parameters
batch_size = 20
num_workers=0

# TODO: Using the image datasets and the trainforms, define the dataloaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
                                        num_workers=num_workers, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,
                                        num_workers=num_workers, shuffle=True)

vgg16 = models.vgg16(pretrained=True)

# Freeze training for all "features" layers
for param in vgg16.features.parameters():
    param.requires_grad = False

import torch.nn as nn

n_inputs = vgg16.classifier[6].in_features

# add last linear layer (n_inputs -> 102 flower classes)
# new layers automatically have requires_grad = True
last_layer = nn.Linear(n_inputs, len(classes))

vgg16.classifier[6] = last_layer

import torch.optim as optim

# specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()

# specify optimizer (stochastic gradient descent) and learning rate = 0.001
optimizer = optim.SGD(vgg16.classifier.parameters(), lr=0.001)

pre_trained_model=torch.load("model.pt")
new=list(pre_trained_model.items())

my_model_kvpair=vgg16.state_dict()
count=0

for key,value in my_model_kvpair.items():
    layer_name, weights = new[count]
    my_model_kvpair[key] = weights
    count+=1

# number of epochs to train the model
n_epochs = 6

# initialize tracker for minimum validation loss
valid_loss_min = np.Inf # set initial "min" to infinity

for epoch in range(1, n_epochs+1):

    # keep track of training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    ###################
    # train the model #
    ###################
    # model by default is set to train
    vgg16.train()

    for batch_i, (data, target) in enumerate(train_loader):
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = vgg16(data)
        # calculate the batch loss
        loss = criterion(output, target)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
        # update training loss
        train_loss += loss.item()

        if batch_i % 20 == 19:    # print training loss every specified number of mini-batches
            print('Epoch %d, Batch %d loss: %.16f' %
                (epoch, batch_i + 1, train_loss / 20))
            train_loss = 0.0

    ######################
    # validate the model #
    ######################
    vgg16.eval() # prep model for evaluation
    for data, target in test_loader:
        # forward pass: compute predicted outputs by passing inputs to the model
        output = vgg16(data)
        # calculate the loss
        loss = criterion(output, target)
        # update running validation loss
        valid_loss += loss.item()

    # print training/validation statistics
    # calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.dataset)
    valid_loss = valid_loss/len(test_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch+1,
        train_loss,
        valid_loss
        ))

    # save model if validation loss has decreased
    if valid_loss <= valid_loss_min:
        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
        valid_loss_min,
        valid_loss))
        torch.save(vgg16.state_dict(), 'model.pt')
        valid_loss_min = valid_loss

testing on a single image

tensor = torch.from_numpy(test_image)
reshaped = tensor.permute(2, 0, 1).unsqueeze(0)
floatified = reshaped.to(torch.float32) / 255
vgg16(floatified)
>>>
tensor([[ 2.5686, -1.1964, -0.0872, -1.7010, -1.6669, -1.0638,  0.4515,  0.1124,
          0.0166,  0.3156,  1.1699,  1.5374,  1.8720,  2.5184,  2.9046, -0.8241,
         -1.1949, -0.5700,  0.8692, -1.0485,  0.0390, -1.3783, -3.4632, -0.0143,
          1.0986,  0.2667, -1.1127, -0.8515,  0.7759, -0.7528,  1.6366, -0.1170,
         -0.4983, -2.6970,  0.7545,  0.0188,  0.1094,  0.5002,  0.8838, -0.0006,
         -1.7993, -1.3706,  0.4964, -0.3251, -1.7313,  1.8731,  2.4963,  1.1713,
         -1.5726,  1.5476,  3.9576,  0.7388,  0.0228,  0.3947, -1.7237, -1.8350,
         -2.0297,  1.4088, -1.3469,  1.6128, -1.0851,  2.0257,  0.5881,  0.7498,
          0.0738,  2.0592,  1.8034, -0.5468,  1.9512,  0.4534,  0.7746, -1.0465,
         -0.7254,  0.3333, -1.6506, -0.4242,  1.9529, -0.4542,  0.2396, -1.6804,
         -2.7987, -0.6367, -0.3599,  1.0102,  2.6319,  0.8305, -1.4333,  3.3043,
         -0.4021, -0.4877,  0.9125,  0.0607, -1.0326,  1.3186, -2.5861,  0.1211,
         -2.3177, -1.5040,  1.0416,  1.4008,  1.4225, -2.7291]],
       grad_fn=<ThAddmmBackward>)

sum([ 2.5686, -1.1964, -0.0872, -1.7010, -1.6669, -1.0638,  0.4515,  0.1124,
          0.0166,  0.3156,  1.1699,  1.5374,  1.8720,  2.5184,  2.9046, -0.8241,
         -1.1949, -0.5700,  0.8692, -1.0485,  0.0390, -1.3783, -3.4632, -0.0143,
          1.0986,  0.2667, -1.1127, -0.8515,  0.7759, -0.7528,  1.6366, -0.1170,
         -0.4983, -2.6970,  0.7545,  0.0188,  0.1094,  0.5002,  0.8838, -0.0006,
         -1.7993, -1.3706,  0.4964, -0.3251, -1.7313,  1.8731,  2.4963,  1.1713,
         -1.5726,  1.5476,  3.9576,  0.7388,  0.0228,  0.3947, -1.7237, -1.8350,
         -2.0297,  1.4088, -1.3469,  1.6128, -1.0851,  2.0257,  0.5881,  0.7498,
          0.0738,  2.0592,  1.8034, -0.5468,  1.9512,  0.4534,  0.7746, -1.0465,
         -0.7254,  0.3333, -1.6506, -0.4242,  1.9529, -0.4542,  0.2396, -1.6804,
         -2.7987, -0.6367, -0.3599,  1.0102,  2.6319,  0.8305, -1.4333,  3.3043,
         -0.4021, -0.4877,  0.9125,  0.0607, -1.0326,  1.3186, -2.5861,  0.1211,
         -2.3177, -1.5040,  1.0416,  1.4008,  1.4225, -2.7291])
>>>
5.325799999999998

given this as how I test it on a single image (and the model as usual is trained and tested on batches it returns a prediction matrix that doesn't seem to be normalized or add up to 1.

Is this normal?


title: Sampling point pairs from a grid in Pytorch

body:
I need sample point pairs from a grid in PyTorch.

I have a tensor of size (1 x 500 x 1000). I also have a mask tensor of size(1 x 500 x 1000), denoting if a point is valid or not. I want to sample 200k point pairs from this grid. In other words, I want to get coordinates of sampled point pairs as a tensor of size (200k x 4), denoting (x1, y1, x2, y2) for all 200k point pairs. All points in the pairs should be valid points.

This will be repeated many times, so I need to have an efficient way of performing this procedure. What is an elegant way to implement this in PyTorch?


title: How to properly implement padding for Seq2Seq LSTM in PyTorch?

body:
In this PyTorch tutorial, they did not use padding despite having variable length input and output. It seems to work fine, but how? Shouldn't we implement padding if input is of variable length? What is the proper way to go about this?

Thank you.


title: Torch.sort and argsort sorting randomly in case of same element

body:
When same elements are encountered, torch.sort and argsort sort the tensor in random manner.
This is not the case in numpy.
I have a list of elements already sorted according to the second column and now i want to sort it using the first column but preserve the earlier sort in case of tie in the new sorting.

import torch

a = torch.tensor(
        [[ 0., 3.],
        [ 2., 3.],
        [ 2., 2.],
        [10., 2.],
        [ 0., 2.],
        [ 6., 2.],
        [10., 1.],
        [ 2., 1.],
        [ 0., 1.],
        [ 6., 1.],
        [10., 0.],
        [12., 0.]]
)
print(a[torch.argsort(a[:, 0])])

Output:

tensor([[ 0.,  3.],
        [ 0.,  2.],
        [ 0.,  1.],
        [ 2.,  1.],
        [ 2.,  2.],
        [ 2.,  3.],
        [ 6.,  1.],
        [ 6.,  2.],
        [10.,  1.],
        [10.,  2.],
        [10.,  0.],
        [12.,  0.]])

Numpy:

import numpy as np

a = np.array(
        [[ 0., 3.],
        [ 2., 3.],
        [ 2., 2.],
        [10., 2.],
        [ 0., 2.],
        [ 6., 2.],
        [10., 1.],
        [ 2., 1.],
        [ 0., 1.],
        [ 6., 1.],
        [10., 0.],
        [12., 0.]]
)
print(a[np.argsort(a[:, 0])])

Output:

[[ 0.  3.]
 [ 0.  2.]
 [ 0.  1.]
 [ 2.  3.]
 [ 2.  2.]
 [ 2.  1.]
 [ 6.  2.]
 [ 6.  1.]
 [10.  2.]
 [10.  1.]
 [10.  0.]
 [12.  0.]]

What could be the reason for this? And what can I do to avoid it?


title: Load Tensor From Buffer using PyTorch and Tensorflow

body:
Given a file io binary buffer I would like to construct a tensor from the raw bytes, in some shape and in some offset without loading it into numpy, but instead directly into torch and tensorflow.

In numpy, my method would be:

def unpack_numpy(s: struct.Struct, shape: Tuple, buffer: bytes, read_offset: int):
   return np.ndarray(shape, s.format, buffer, read_offset)

Is there a way to implement this for pytorch?

def unpack_torch(s: struct.Struct, shape: Tuple, buffer: bytes, read_offset: int):
   # ???

There exists torch.as_tensor, but it can't set the shape, or the read offset from the buffer.

Same question for tensorflow - Is there a way to implement this for tensorflow?

def unpack_tensorflow(s: struct.Struct, shape: Tuple, buffer: bytes, read_offset: int):
   # ???

There exists tf.io.decode_raw, but it can't set the shape, or the read offset.


title: Gradient error when calculating - pytorch

body:
I am learning to use pytorch (0.4.0) to automate the gradient calculation, however I did not quite understand how to use the backward () and grad, as I'm doing an exercise I need to calculate df / dw using pytorch and
 making the derivative analytically, returning respectively auto_grad, user_grad, but I did not quite understand the use of automatic differentiation, in the code I made f.backward () and did w.grad to find df / dw, in addition the two calculations are not corresponding, if I even erred the derivative, it follows the graph that I am using and the code that I am trying to do:

import numpy as np
import torch
import torch.nn.functional as F

def graph2(W_np, x_np, b_np):
    W = torch.Tensor(W_np)
    W.requires_grad = True
    x = torch.Tensor(x_np)
    b = torch.Tensor(b_np)
    u = torch.matmul(W, x) + b
    g = F.sigmoid(u)
    f = torch.sum(g)
    user_grad = (sigmoid(W_np*x_np + b_np)*(1 - sigmoid(W_np*x_np + b_np))).T*x_np
    f.backward(retain_graph=True)
    auto_grad = W.grad

    print(auto_grad)
    print(user_grad)

 #   raise NotImplementedError("falta completar a função graph2")
    # END YOUR CODE
    return f, auto_grad, user_grad

test:

iterations = 1000
sizes = np.random.randint(2,10, size=(iterations))
for i in range(iterations):
    size = sizes[i]
    W_np = np.random.rand(size, size)
    x_np = np.random.rand(size, 1)
    b_np = np.random.rand(size, 1)
    f, auto_grad, user_grad = graph2(W_np, x_np, b_np)
    manual_f = np.sum(sigmoid(np.matmul(W_np, x_np) + b_np))
    assert np.isclose(f.data.numpy(), manual_f, atol=1e-4), "f not correct"
    assert np.allclose(auto_grad.numpy(), user_grad), "Gradient not correct"


title: pytorch/cpp_extension.py path error

body:
During the installing of pytorch/audio an external library that builds of off pyTorch it is necessary to run this setup.py. The program was run with the argument "install" from a fully updated Anacond5.2 with python3.6 terminal on windows10:

The following error output was generated:

Traceback (most recent call last):
  File "setup.py", line 4, in <module>
    from torch.utils.cpp_extension import BuildExtension, CppExtension
  File "C:\Users\jason\Anaconda3\lib\site-packages\torch\utils\cpp_extension.py", line 59, in <module>
    CUDA_HOME = _find_cuda_home() if torch.cuda.is_available() else None
  File "C:\Users\jason\Anaconda3\lib\site-packages\torch\utils\cpp_extension.py", line 30, in _find_cuda_home
    if not os.path.exists(cuda_home):
  File "C:\Users\jason\Anaconda3\lib\genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not list

NOTE: path is user specific and usually not needed information, however in this case the error MAY be path related so it is included.


title: My neural network takes too much time to train one epoch

body:
I am training a neural network which tries to classify a traffic signs, but it takes too much time to train only one epoch, maybe 30+ mins for just one epoch, I have set the batch size to 64 and the learning rate to be 0.002, the input is 20x20 pixels with 3 channels, and the model summary shows that it is training 173,931 parameters, is that too much or good?

Here is the network architecture

import torch.nn as nn
import torch.nn.functional as F
from torchsummary import summary

class Network(nn.Module):
  def __init__(self):
    super(Network,self).__init__()
    #Convolutional Layers
    self.conv1 = nn.Conv2d(3,16,3,padding=1)
    self.conv2 = nn.Conv2d(16,32,3,padding=1)

    #Max Pooling Layers
    self.pool = nn.MaxPool2d(2,2)

    #Linear Fully connected layers
    self.fc1 = nn.Linear(32*5*5,200)
    self.fc2 = nn.Linear(200,43)

    #Dropout
    self.dropout = nn.Dropout(p=0.25)

  def forward(self,x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))

    x = x.view(-1,32*5*5)
    x = self.dropout(x)
    x = F.relu(self.fc1(x))
    x = self.dropout(x)
    x = self.fc2(x)

    return x

Here is the optimizer instance

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optim = optim.SGD(model.parameters(),lr = 0.002)

Here is the training code

epochs = 20
valid_loss_min = np.Inf
print("Training the network")

for epoch in range (1,epochs+1):
  train_loss = 0
  valid_loss = 0
  model.train()

  for data,target in train_data:
    if gpu_available:
      data,target = data.cuda(),target.cuda()

    optim.zero_grad()
    output = model(data)
    loss = criterion(output,target)
    loss.backward()
    optim.step()
    train_loss += loss.item()*data.size(0)

  #########################
  ###### Validate #########
  model.eval()
  for data,target in valid_data:
    if gpu_available:
      data,target = data.cuda(),target.cuda()

    output = model(data)
    loss = criterion(output,target)
    valid_loss += loss.item()*data.size(0)

  train_loss = train_loss/len(train_data.dataset)
  valid_loss = train/len(valid_data.dataset)

  print("Epoch {}.....Train Loss = {:.6f}....Valid Loss = {:.6f}".format(epoch,train_loss,valid_loss))

  if valid_loss < valid_loss_min:
    torch.save(model.state_dict(), 'model_traffic.pt')
    print("Valid Loss min {:.6f} >>> {:.6f}".format(valid_loss_min, valid_loss))

I am using GPU through google colab