title: PyTorch Convolution `in_channels` and `out_channels` meaning?

body:
From the PyTorch documentation for Convolution, I see the function torch.nn.Conv1d requires users to pass the parameters in_channels and out_channels.
I know these refer to "input channels" and "output channels", but I am not sure what they mean in the context of a convolution. My guess is that in_channels is equivalent to the input features and out_channels is equivalent to the output features, but I am not sure.
Could someone explain what thse arguments refer to?


title: Why are Pytorch and Keras implementations giving vastly different results?

body:
I am trying to train a 1-D ConvNet for time series classification as shown in this paper (refer to FCN om Fig. 1b) https://arxiv.org/pdf/1611.06455.pdf

The Keras implementation is giving me vastly superior performance. Could someone explain why is that the case?

The code for Pytorch is as follow:

class Net(torch.nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv1d(x_train.shape[1], 128, 8)
    self.bnorm1 = nn.BatchNorm1d(128)
    self.conv2 = nn.Conv1d(128, 256, 5)
    self.bnorm2 = nn.BatchNorm1d(256)
    self.conv3 = nn.Conv1d(256, 128, 3)
    self.bnorm3 = nn.BatchNorm1d(128)
    self.dense = nn.Linear(128, nb_classes)

def forward(self, x):
   c1=self.conv1(x)
   b1 = F.relu(self.bnorm1(c1))
   c2=self.conv2(b1)
   b2 = F.relu(self.bnorm2(c2))
   c3=self.conv3(b2)
   b3 = F.relu(self.bnorm3(c3))
   output = torch.mean(b3, 2)
   dense1=self.dense(output)
   return F.softmax(dense1)

 model = Net()
 criterion = nn.CrossEntropyLoss()
 optimizer = torch.optim.SGD(model.parameters(), lr=0.5, momentum=0.99)
 losses=[]
 for t in range(1000):
     y_pred_1= model(x_train.float())
     loss_1 = criterion(y_pred_1, y_train.long())
     print(t, loss_1.item())
     optimizer.zero_grad()
     loss_1.backward()
     optimizer.step()

For comparison, I use the following code for Keras:

x = keras.layers.Input(x_train.shape[1:])
conv1 = keras.layers.Conv1D(128, 8, padding='valid')(x)
conv1 = keras.layers.BatchNormalization()(conv1)
conv1 = keras.layers.Activation('relu')(conv1)
conv2 = keras.layers.Conv1D(256, 5, padding='valid')(conv1)
conv2 = keras.layers.BatchNormalization()(conv2)
conv2 = keras.layers.Activation('relu')(conv2)
conv3 = keras.layers.Conv1D(128, 3, padding='valid')(conv2)
conv3 = keras.layers.BatchNormalization()(conv3)
conv3 = keras.layers.Activation('relu')(conv3)
full = keras.layers.GlobalAveragePooling1D()(conv3)
out = keras.layers.Dense(nb_classes, activation='softmax')(full)

model = keras.models.Model(inputs=x, outputs=out)
optimizer = keras.optimizers.SGD(lr=0.5, decay=0.0, momentum=0.99)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)
hist = model.fit(x_train, Y_train, batch_size=x_train.shape[0], nb_epoch=2000)

The only difference I see between the two is the initialization but however, the results are just vastly different. For reference, I use the same preprocessing as follows for both the datasets, with a subtle difference in input shapes, for Pytorch (Batch_Size, Channels, Length) and for Keras: (Batch_Size, Length, Channels).


title: Torch Network load does not processed properly

body:
I am trying to make a network using 3x64x64 image at pytorch environment, and it seems that I succeeded in training my network and save it. The network looks like :

class LC_small(nn.Module):
    def __init__(self,c_in,c_out = 256):
    super(LC_small,self).__init__()
        self.conv1 = conv(c_in,64,k=3,stride=1,pad=1)
        self.conv2 = conv(64, 128, k=3, stride=2, pad=1)
        self.conv3 = conv(128, 128, k=3, stride=1, pad=1)
        self.conv4 = conv(128, 128, k=3, stride=2, pad=1)
        self.conv5 = conv(128, 128, k=3, stride=1, pad=1)
        self.conv6 = conv(128, 256, k=3, stride=2, pad=1)
        self.conv7 = conv(256, 256, k=3, stride=1, pad=1)# int(h/8 x w/8 x 256)
        self.flat = dense(int(w_rsz/8)*int(h_rsz/8)*256,256)
        self.dense1 = dense(256,128,False)
        self.dense2 = dense(128,3,False)
    def forward(self, input):
        out = self.conv1(input)
        out = self.conv2(out)
        out = self.conv3(out)
        out = self.conv4(out)
        out = self.conv5(out)
        out = self.conv6(out)
        out = self.conv7(out)
        out = out.view(out.size(0),-1)
        out = self.flat(out)
        out = self.dense1(out)
        out = self.dense2(out)
         # print(out.shape)
        normal = torch.nn.functional.normalize(out, 2, 1)

        return normal

And I saved my model while Training :

for epoch in range(10):
#  continue    # 현재 Training 됐다고 가정하고
    total_loss = 0
    route_param = open(route_diffuse+'/netparam.txt','w')
    for param in lcnet.state_dict():
    route_param.write(str(param)+'\t'+str(lcnet.state_dict()[param].size())+'\n')
    for i,data in enumerate(load_LC,0):
    input, gtval = data[0].to(dev),data[1].to(dev)
    opt.zero_grad()

    output = lcnet(input)
    loss = crit(output,gtval)
    loss.backward()
    opt.step()
    total_loss +=loss.item()
    if i%10 == 9:
         print(epoch,i,total_loss/10)
         torch.save(lcnet,route_save)
         total_loss = 0

However, by the time I try to load the network I made I saw a error message as following :

Traceback (most recent call last):

File "E:/DLPrj/venv/torch_practice.py", line 324, in <module>

ipl,npl = getseqi_np(sq_t,lcnet)   #  data : 8 x 6 x w x h

File "E:/DLPrj/venv/torch_practice.py", line 133, in getseqi_np

l1 = net_lc(torch.from_numpy(i1r))

File "E:\DLPrj\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__

result = self.forward(*input, **kwargs)

File "E:/DLPrj/venv/torch_practice.py", line 216, in forward

out = self.conv1(input)

File "E:\DLPrj\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__

result = self.forward(*input, **kwargs)

File "E:\DLPrj\venv\lib\site-packages\torch\nn\modules\container.py", line 92, in forward

input = module(input)

File "E:\DLPrj\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__

result = self.forward(*input, **kwargs)

File "E:\DLPrj\venv\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward

return self.conv2d_forward(input, self.weight)

File "E:\DLPrj\venv\lib\site-packages\torch\nn\modules\conv.py", line 342, in conv2d_forward

self.padding, self.dilation, self.groups)

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 3 3 3, but got 3-dimensional input of size [64, 64, 3] instead

After this error pycharm freezes and I cannot re-run this code until I restart the pycharm.

When I train my network, I also get some warning messages :

E:\DLPrj\venv\lib\site-packages\torch\serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LC_small. It won't be checked for correctness upon loading.

 "type " + obj.__name__ + ". It won't be checked "

E:\DLPrj\venv\lib\site-packages\torch\serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Sequential. It won't be checked for correctness upon loading.

 "type " + obj.__name__ + ". It won't be checked "

E:\DLPrj\venv\lib\site-packages\torch\serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Conv2d. It won't be checked for correctness upon loading.

 "type " + obj.__name__ + ". It won't be checked "

E:\DLPrj\venv\lib\site-packages\torch\serialization.py:292: UserWarning: Couldn't retrieve source code for container of type BatchNorm2d. It won't be checked for correctness upon loading.

 "type " + obj.__name__ + ". It won't be checked "

E:\DLPrj\venv\lib\site-packages\torch\serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LeakyReLU. It won't be checked for correctness upon loading.

 "type " + obj.__name__ + ". It won't be checked "

E:\DLPrj\venv\lib\site-packages\torch\serialization.py:292: UserWarning: Couldn't retrieve source code for container of type Linear. It won't be checked for correctness upon loading.

 "type " + obj.__name__ + ". It won't be checked "

I cannot understand why the input size the network takes suddenly changes, or why it saved my network wrong. Please check my problem and thank you very much.


title: Fetching data from remote server in pytorch dataloader

body:
I have a large hd5 file (~100GB) containing image features from resnet. This file is located on my local machine (laptop). My model is trained on cluster node that has storage limit of 25GB.
Right now, I am using torch.distributed.rpc for tranferring data from my local machine to cluster.
I am exposing a server on my local machine in the following way,

num_worker = 4
utils.WORLD_SIZE = num_worker + 1

import os
import torch
import utils
import torch.distributed.rpc as rpc

def run_worker(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '8888'

    rpc.init_rpc(utils.SERVER_NAME,
                 rank = rank,
                 world_size = world_size)
    print("Server Initialized", flush=True)
    rpc.shutdown()

if __name__ == "__main__":
    rank = 0
    world_size = utils.WORLD_SIZE
    run_worker(rank, world_size)

This server sends data from local machine to cluster. (other classes are omitted)
Now for requesting data from cluster, I am initializing rpc for each worker using worker_init_fn for dataloader,
def worker_init_fn(worker_id):
    rpc.init_rpc(utils.CLIENT_NAME.format(worker_id+1),
                rank=worker_id+1, world_size=utils.WORLD_SIZE)

    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset
    worker_id = worker_info.id
    server_info = rpc.get_worker_info(utils.SERVER_NAME)
    dataset.server_ref = rpc.remote(server_info, utils.Server)

Now, when I run my code, the training loop completes one iteration of the dataset and hangs after that and I get the following error on cluster side,
Traceback (most recent call last):
  File "custom_datasets.py", line 134, in <module>
    main()
  File "custom_datasets.py", line 110, in main
    for i, (images, labels) in enumerate(mn_dataset_loader):
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 135, in _worker_loop
    init_fn(worker_id)
  File "custom_datasets.py", line 78, in worker_init_fn
    rank=worker_id+1, world_size=utils.WORLD_SIZE)
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/distributed/rpc/__init__.py", line 67, in init_rpc
    store, _, _ = next(rendezvous_iterator)
  File "/home/kanishk/.local/lib/python3.7/site-packages/torch/distributed/rendezvous.py", line 168, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: connect() timed out.

Above problem doesn't occur when I set num_worker = 0, but the cluster code is very slow. I think the error is because of multi-threading but I am not sure how to resolve this. Please help me resolving the issue.


title: Is there any way to use both a GPU accelerator and Torch in google cloud AI platform for model deployment?

body:
I already have a torch model (BERT), and I'd like to use the ai-platform service to get online predictions using a GPU, but I can't figure out how to do it.

The following command, without an accelerator, works:

gcloud alpha ai-platform versions create {VERSION} --model {MODEL_NAME} --origin=gs://{BUCKET}/models/ --python-version=3.5 --runtime-version=1.14 --package-uris=gs://{BUCKET}/packages/my-torch-package-0.1.tar.gz,gs://cloud-ai-pytorch/torch-1.0.0-cp35-cp35m-linux_x86_64.whl --machine-type=mls1-c4-m4 --prediction-class=predictor.CustomModelPrediction

However, if I try to add the accelerator parameter:

--accelerator=^:^count=1:type=nvidia-tesla-k80

I get the following error message:

ERROR: (gcloud.alpha.ai-platform.versions.create) INVALID_ARGUMENT: Field: version.machine_type Error: GPU accelerators are not supported on the requested machine type: mls1-c4-m4
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: 'GPU accelerators are not supported on the requested machine type:
      mls1-c4-m4'
    field: version.machine_type

But if I use a different machine type, that I know I can use with an accelerator, I get the following error:

ERROR: (gcloud.alpha.ai-platform.versions.create) FAILED_PRECONDITION: Field: framework Error: Machine type n1-highcpu-4 does not support CUSTOM_CLASS.
- '@type': type.googleapis.com/google.rpc.BadRequest
  fieldViolations:
  - description: Machine type n1-highcpu-4 does not support CUSTOM_CLASS.
    field: framework

It's like any machine that supports GPU accelerators doesn't support custom classes (required AFAIK to use Torch), and any machine that supports custom classes doesn't support GPU accelerators.

Any way to make it work?

There are a bunch of tutorials on how to use ai-platform with Torch, but I can't see the point of using gcloud to train and predict if you have to do everything on the CPU so that feels very odd to me.


title: Send Pytorch's Tensor as message in gRPC

body:
I am trying to make a gRPC call to split deep learning between layers in PyTorch. For this, I need to send a Tensor to my cloud for processing using gRPC call. I am currently trying to use the following as message:

message Tensor {
    google.protobuf.Any out = 1;
}

and in my pyTorch code,

from google.protobuf.any_pb2 import Any
...
request = Any()
...
request.Pack(out)
send_grpc_msg(request)

Here, type(out) is <class 'torch.Tensor'>. However, this code is giving the following error:

Traceback (most recent call last):
  File "vgg_inference.py", line 95, in <module>
    out = detect_images(Image.open("images/dog.jpg"))
  File "vgg_inference.py", line 88, in detect_images
    request.Pack(out)
  File "/home/rohit/.local/lib/python3.6/site-packages/google/protobuf/internal/well_known_types.py", line 78, in Pack
    self.type_url = '%s%s' % (type_url_prefix, msg.DESCRIPTOR.full_name)
AttributeError: 'Tensor' object has no attribute 'DESCRIPTOR'

I am also unable to find a good resource of using ANY in python.

Thanks


title: Training using GradientTape not working for Regression fits

body:
I have been trying to build a Neural Network based regressor that can model a nonlinear function. I am recreating the issue with a dummy case here.

Initially I have established a function u=f(x,t)=sin(x/2)+e^(-t) and built up a dataset by doing a parameter scan along x and t.

N = 5000
x = np.linspace(-np.pi, np.pi, N)
t = np.linspace(0, 2, N)

lb = np.asarray([x.min(), t.min()])
ub = np.asarray([x.max(), t.max()])

def func_u(x, t): # Function that we are interested in modelling.
    return np.sin(x/2) + np.exp(-t)

u_actual = func_u(x, t)

X = np.vstack((x, t)).T

After having built a labelled dataset that maps the function from input to output, tried fitting using a keras approach.

def tf_model(): # Creating a two layer Neural Network.
    act_func = 'relu'
    model_tf = keras.Sequential()
    model_tf.add(keras.layers.Input(shape=(2,)))
    model_tf.add(keras.layers.Dense(100, activation=act_func))
    model_tf.add(keras.layers.Dense(100, activation=act_func))
    model_tf.add(keras.layers.Dense(1, activation='linear'))

    return model_tf

model = tf_model()
model.compile(optimizer='adam', loss='mse')

model.fit(X, u_actual, # Fitting using the Keras API
          batch_size=500,
          epochs=1000,
          verbose=1)

u_model_fit = model(X).numpy()

Which does an amazing job of fitting to the data.

However, when I try and do that with a custom training implementation using GradientTape, it all goes awfully wrong:

def shuffle_and_batch(X, Y, num_batches=50): # Shuffle and group the input data inot various datasizes.
    indices = tf.range(start=0, limit=tf.shape(X)[0], dtype=tf.int32)
    shuffled_indices = tf.random.shuffle(indices)

    X = tf.gather(X, shuffled_indices)
    Y = tf.gather(Y, shuffled_indices)

    X = tf.split(X, num_batches)
    Y = tf.split(Y, num_batches)
    return X, Y

X_tf = tf.Variable(X, tf.float64) #Creating Tensors to treat as the inputs.
Y_tf = tf.Variable(u_actual, tf.float64)

X_tf, Y_tf = shuffle_and_batch(X_tf, Y_tf)

def loss(model, X, Y): # mean squared reconstruction error
    return tf.reduce_mean(tf.square(model(X, training=True) - Y))

def loss_and_gradients(model, X, Y):
    with tf.GradientTape() as tape:
        loss_tf = loss(model, X, Y)
    grads_tf = tape.gradient(loss_tf, model.trainable_variables)  #Calculating the gradient of each of the loss with respect to the weights and biases.
    return loss_tf, grads_tf

optimizer = tf.keras.optimizers.Adam()
nIter =1000
model = tf_model()

for it in range(nIter):
    for batch_num in range(50):
        loss_tf, grads_tf = loss_and_gradients(model, X_tf[batch_num], Y_tf[batch_num]) #Obtaining the loss and the gradients
        optimizer.apply_gradients(zip(grads_tf, model.trainable_variables)) #Applying the gradients for each step

    tf.print('Iter : {}, loss : {}'.format(it, loss_tf))

u_tf = model(X).numpy()

I tried to do that with Pytorch and pretty much arrive at the same solution as that of using Gradient Tape

model_torch = torch.nn.Sequential(
    torch.nn.Linear(2, 100),
    torch.nn.ReLU(),
    torch.nn.Linear(100, 100),
    torch.nn.ReLU(),
    torch.nn.Linear(100, 1)
    )

X_torch = torch.tensor(X, dtype=torch.float64).float()
Y_torch = torch.tensor(u_actual, dtype=torch.float64).float()

# X_torch, Y_torch = shuffle_and_batch_torch(X_torch, Y_torch)
X_torch, Y_torch = torch.autograd.Variable(X_torch, requires_grad=True), torch.autograd.Variable(Y_torch, requires_grad=True) #Ensuring that tracing occurs.

traindata = torch.utils.data.TensorDataset(X_torch, Y_torch) #Loading, Shuffling and Batching the training data.
dataloader = torch.utils.data.DataLoader(traindata, batch_size=50, shuffle=True)

def lossfunc_torch(model, X, Y): # Calculating the mean squared error,
    y = model_torch(X)
    loss = (y-Y).pow(2).mean()

    return loss

optimizer = torch.optim.Adam(model_torch.parameters(), 0.001)
nIter = 1000

for it in range(nIter):
    for i, (x_torch, y_torch) in enumerate (dataloader):
        optimizer.zero_grad()

        loss_torch = lossfunc_torch(model_torch, x_torch, y_torch)

        loss_torch.backward()
        optimizer.step()

    print("Iter : {}, Loss : {}".format(it, loss_torch.item()))

u_torch = model_torch(X_torch).detach().numpy()


title: Explanation behind the following Pytorch results

body:
I am trying to get a deeper understanding of how Pytorch's autograd works. I am unable to explain the following results:
import torch
def fn(a):
 b = torch.tensor(5,dtype=torch.float32,requires_grad=True)
 return a*b

a  = torch.tensor(10,dtype=torch.float32,requires_grad=True)
output = fn(a)
output.backward()
print(a.grad)

The output is tensor(5.). But my question is that the variable b is created within the function and  so should be removed from memory after the function returns a*b right? So when I call backward how is the value of b still present for allowing this computation?
As far as I understand each operation in Pytorch has a context variable which tracks "which" tensor to use for backward computation and there are also versions present in each tensor, and if the version changes then backward should raise an error right?
Now when I try to run the following code,
import torch
def fn(a):
 b = a**2
 for i in range(5):
   b *= b
 return b

a  = torch.tensor(10,dtype=torch.float32,requires_grad=True)
output = fn(a)
output.backward()
print(a.grad)

I get the following error: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor []], which is output 0 of MulBackward0, is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
But if I run the following code, there is no error:
import torch
def fn(a):
  b = a**2
  for i in range(2):
    b = b*b
  return b

def fn2(a):
  b = a**2
  c = a**2
  for i in range(2):
    c *= b
  return c

a  = torch.tensor(5,dtype=torch.float32,requires_grad=True)
output = fn(a)
output.backward()
print(a.grad)
output2 = fn2(a)
output2.backward()
print(a.grad)

The output for this is :
tensor(625000.)
tensor(643750.)
So for a standard computation graphs with quite a few variables, in the same function, I am able to understand how the computation graph works. But when there is a variable changing before the call of backward function, I am having a lot of trouble understanding the results. Can someone explain?


title: What should be the architecture of the generator and discriminator model of the GAN for generating 300 * 300 * 3 images?

body:
I have usually seen people generating images of 28 * 28 , 64 * 64 etc. For creating this sized image they usually start with the number of filters 512, 256, 128,.. so on in decreasing manner for generators and for discriminators in reverse manner. Usually they keep the same number of layers in discriminator and generators.
My first question is what should be the architecture of discriminator and generator model to create 300 * 300 images.
My second question is ..is it mandatory to have equal number of layers in both the discriminator and generator. What if I have more number of layers in my discriminator than the generator ?
My third question depends upon the second question only, can I use feature extractor part of any famous models like resnet , vgg etc. for making discriminators ?
P.S. if you are writing the architecture code please make it in pytorch or keras .


title: How to use the BitShift operator in Pytorch?

body: Does anyone has an example of how to use the BitShift operator in Pytorch?


title: Why define a backward method for a custom layer in Pytorch?

body:
I am currently constructing a model in Pytorch that requires multiple custom layers. I have only been defining the forward method and thus do not define a backward method. The model seems to run well, and the optimizer is able to update using the gradients from the layers. However, I see many people defining backward methods, and I wonder if I am missing something.
Why might you need to define a backwards pass?


title: mask list/tensor with multiple conditions?

body:
The following code masks fine
mask = targets >= 0
targets = targets[mask]

However, when I try masking with two conditions, it gives an error of RuntimeError: Boolean value of Tensor with more than one value is ambiguous
mask = (targets >= 0 and targets <= 5)
targets = targets[mask]

is there a way to do this?


title: Fixing Python Dependencies The Right Way

body:
I'm just getting my first Python environment setup. All have gone well and it seems to be GPU enabled and all that good stuff.
However, I have one issue and no idea how to fix. After getting the correct install command for torch it informed of this issue:
Installing collected packages: torch
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.8.2 requires torch==1.7.1, but you have torch 1.8.0+cu111 which is incompatible.
Successfully installed torch-1.8.0+cu111

As far as I can tell torchvision 0.8.2 is the latest version.
The environment seems happy at the moment as all these commands return expected things:
import torch
print(torch.__version__)
torch.cuda.get_device_name(0)

I've seen some people talking about "patching requirements files" or updating dependencies. But I'm not sure of the best way to tackle this.


title: Using PyTorch's autograd efficiently with tensors by calculating the Jacobian

body:
In my previous question I found how to use PyTorch's autograd with tensors:
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 4) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()

#input
t = torch.tensor([1.0, 2.0, 3.2], requires_grad = True) #input vector
t = torch.reshape(t, (3,1)) #reshape for batch

#method
dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)
dx = torch.diagonal(torch.diagonal(dx, 0, -1), 0)[0] #first vector
#dx = torch.diagonal(torch.diagonal(dx, 1, -1), 0)[0] #2nd vector
#dx = torch.diagonal(torch.diagonal(dx, 2, -1), 0)[0] #3rd vector
#dx = torch.diagonal(torch.diagonal(dx, 3, -1), 0)[0] #4th vector
dx
>>>
tensor([-0.0142, -0.0517, -0.0634])

The issue is that grad only knows how to propagate gradients from a scalar tensor (which my network's output is not), which is why I had to calculate the Jacobian.
However, this is not very efficient and a bit slow as my matrix is large and calculating the entire Jacobian takes a while (and I'm also not using the entire Jacobian matrix).
Is there a way to calculate only the diagonals of the Jacobian (to get the 4 vectors in this example)?
There appears to be an open feature request but it doesn't appear to have gotten much attention.
Update 1:
I tried what @iacob said about setting torch.autograd.functional.jacobian(vectorize=True).
However, this seems to be slower. To test this I changed my network output from 4 to 400, and my input t to be:
val = 100
t = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(t, (val,1)) #reshape for batch

Without vectorized = True:
Wall time: 10.4 s

With:
Wall time: 14.6 s


title: Is there any difference between using WeightedRandomSampler with a big num_samples or doing more epoch with a num_samples lower?

body:
I don't understand when the sampling is make:

Does the first mini batch will be the same for each epoch?
Or there no difference at all?


I'm building a CNN and training it on hand sign gesture classification for letters A through I (9 classes), each image is RGB with 224x224 size.

Not sure which matrix I need to transpose and how. I have managed to match the inputs and outputs of layers, but that matrix multiplication thing, not really sure how to fix it.

class LargeNet(nn.Module):
    def __init__(self):
        super(LargeNet, self).__init__()
        self.name = "large"
        self.conv1 = nn.Conv2d(3, 5, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(5, 10, 5)
        self.fc1 = nn.Linear(10 * 53 * 53, 32)
        self.fc2 = nn.Linear(32, 9)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        print('x1')
        x = self.pool(F.relu(self.conv2(x)))
        print('x2')
        x = x.view(-1, 10*53*53)
        print('x3')
        x = F.relu(self.fc1(x))
        print('x4')
        x = x.view(-1, 1)
        x = self.fc2(x)
        print('x5')
        x = x.squeeze(1) # Flatten to [batch_size]
        return x

and training code

#Loss and optimizer
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(model2.parameters(), lr=learning_rate, momentum=0.9)

# Train the model
total_step = len(train_loader)
loss_list = []
acc_list = []
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        print(i,images.size(),labels.size())
        # Run the forward pass
        outputs = model2(images)
        labels=labels.unsqueeze(1)
        labels=labels.float()
        loss = criterion(outputs, labels)

The code prints up to x4 and then I get this error RuntimeError: size mismatch, m1: [32 x 1], m2: [32 x 9] at C:\w\1\s\tmp_conda_3.7_055457\conda\conda-bld\pytorch_1565416617654\work\aten\src\TH/generic/THTensorMath.cpp:752

Complete traceback error: https://ibb.co/ykqy5wM


title: How to Initalize the weights of nn.ConvTranspose2d in pytorch?

body:
How can I initialize the weights of nn.ConvTranspose2d by bilinear interpolation? I have found nn.init witch supports initialization of each layer by random numbers from a normal distribution or zero initialization but there was nothing about interpolation. Also, interpolation can be utilized for upsampling in torch.nn.Upsample but in this way, the layer is not trainable.
Any idea on how to initialize a trainable ConvTranspose2d layer with bilinear interpolation?


title: How to build a dataset and build_vocab in pytorch with txt files?

body:
I have txt files for training and validation with letters inside for example

File_1:

hg jk lo ju jk lk

File 2:

sd sg gt gh qe

So I want to build a dataset with these files, (I have 5 labels) so I build a Dataset class

class Dataset(data.Dataset):
    def __init__(self,data_root):
        self.letters=[]
        self.labels=[]
        labels=0

        for lan in os.listdir(data_root):
            lan_folder = os.path.join(data_root,lan)
            labels=labels+1
            for type in os.listdir(lan_folder):
                type_folder = os.path.join(lan_folder,type)

                for name in os.listdir(type_folder):
                    file_names=os.path.join(type_folder,name)
                    phonegrams=open(file_names,encoding='utf-8').read().strip().split('\n')
                    self.letters.append(phonegrams)
                    self.labels.append(labels)

    def __len__(self):
        return len(self.labels)

    def __getitem__(self,index):
        return self.letters[index],self.labels[index]

dataset=Dataset(r'C:\Users\ASUS\Desktop\David\Programas\Sequence to tag 1st version\phon\training')

when I print the first item of dataset I get:

([['ij jA: A:z zy yi i']], [1])

My problem is I want to tokenize and index each file , because these files have little words inside, so I made this:

tokenize = lambda x: x.split()
LANS = Field(tokenize=tokenize,init_token='<sos>',eos_token='<eos>',sequential=True)
LANS.build_vocab(dataset)

and then :

LANS.vocab.freqs.most_common(1)

and I get

[('ij jA: A:z zy ,1)]

is using the complete sentence and not each word, is supposed to get me something like:

ij : 1

ja: 2


title: Amazon sagemaker Lifecycle configuration not working

body:
I have the following Lifecycle configuration file in Amazon sagemaker
#!/bin/bash
sudo -u ec2-user -i <<'EOF'

source activate conda_pytorch_p36

# Replace myPackage with the name of the package you want to install.
conda install -c pytorch torchtext
# You can also perform "conda install" here as well.

source deactivate

EOF

But when I try to import torchtext on a jupyter notebook inside the pytorch_conda_p36 virtual environment I get a module not found error
This is my code
from sagemaker import get_execution_role

role = get_execution_role()
bucket='sagemaker-us-east-2-964130302244'
data_dir = 'dataset'
data_location = 's3://{}/{}'.format(bucket, data_dir)

import torch
import torch.utils.data as tud
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from collections import Counter, defaultdict
import operator
import os, math
import numpy as np
import random
import copy
import s3fs
import torchtext
fs = s3fs.S3FileSystem()

The error I get is the following
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-c9a96a48b0b7> in <module>
     11 import copy
     12 import s3fs
---> 13 import torchtext
     14 fs = s3fs.S3FileSystem()
     15 # from nltk import word_tokenize

ModuleNotFoundError: No module named 'torchtext'


title: Pytorch training loss function throws: "TypeError: 'Tensor' object is not callable"

body:
I use Python 3.x, and pytorch 1.5.0 with a GPU. I am trying to write a simple multinomial logistic regression using mnist data.
My issue is the loss() function throws a TypeError: 'Tensor' object is not callable while looping through the training batches. The thing that baffles me is that the error does not show up in the first iteration of the loop, but for the second batch, I get the full error below:
Traceback (most recent call last):
  File "/snap/pycharm-community/207/plugins/python-ce/helpers/pydev/pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/snap/pycharm-community/207/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/pytorch_tutorial/Pytorch_feed_fwd_310720.py", line 78, in <module>
    loss = loss(preds,ys)
TypeError: 'Tensor' object is not callable

The loss() function here is simply loss = nn.CrossEntropyLoss(). The full code is below. Any pointers would be very welcome.
    for epoch in range(5):

        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            xs, ys = data
            opt.zero_grad()
            preds = net(xs)
            loss = loss(preds,ys)
            loss.backward()
            opt.step()

            # print statistics
            running_loss += loss.item()
            if i % 1000 == 999:  # print every 1000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

        print('epoch {}, loss {}'.format(epoch, loss.item()))

    a=1


title: PyTorch softmax return

body:
I am new to PyTorch and have been following this tutorial for reinforcement learning. My environment is a custom Pacman game that does not use the gym environments. The game loop is taken care of. An object in this Pacman game allows access to the state data. I use this data to send input to my Deep Q Network.
First I change the input from a python list into a tensor so that my Deep Q Network can take it as an input. Here is how I convert my python list into a tensor:
#self.getFeatures() returns a dictionary, so I grab the values and convert it to a list
input = torch.FloatTensor(list(self.getFeatures(gameState, bestAction).values())) \
             .unsqueeze(0) \
             .to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))

Then I pass in this input to my policy Deep Q Network:
test_net = self.policy_net(input).max(1)[1].view(1, 1)

Below is my Deep Q Network:
class DQN(nn.Module):

def __init__(self, feature_size, action_size):
    super(DQN, self).__init__()
    self.input = nn.Linear(feature_size, 12)
    self.hidden1 = nn.Linear(12, 5)
    self.hidden2 = nn.Linear(5, action_size)

# Called with either one element to determine next action, or a batch
# during optimization. Returns tensor([[left0exp,right0exp]...]).
def forward(self, x):
    x = F.relu(self.input(x))
    x = F.relu(self.hidden1(x))
    x = F.softmax(self.hidden2(x), dim=1)
    return x

With input tensor([[0., 1., 1., 0., 1.]]) this test_net returns this tensor([[0]]). I don't know what to get from this. I was under the impression that softmax returned a probability of each action. I have 5 actions available in my action space. I don't know what to do with my output from test_net. I want to get an action selection from this test_net, but am getting an integer.
My questions are, should the input be in a different shape? Am I converting my python input list into a tensor correctly? I have 5 features, which are tensor([[0., 1., 1., 0., 1.]]). Should the output tensor([[0]]) be a float and not a 0?


Thanks to everyone reading this.
Can you tell me how to load the Pytorch model(.pt) when using opencv in c++?
Usually in c++
Net net = readNet(model, config);
Like this, the model file and the config (configuration) file are used as parameters to create a net object.
The model file has a .pt extension, but I don't have any information on how to set the configuration file, so I ask.
Thank you.


title: FFT loss in PyTorch

body:
I want to compute the loss between the GT and the output of my network (called TDN) in the frequency domain by computing 2D FFT. The tensors are of dim batch x channel x height x width
amp_ip, phase_ip = 2DFFT(TDN(ip))
amp_gt, phase_gt = 2DFFT(TDN(gt))
loss = ||amp_ip - amp_gt||

For computing FFT I can use torch.fft(ip, signal_ndim = 2). But the output is in a + j b format i.e rectangular coordinates and NOT decomposed into phase and amplitude. How can I convert a + j b into amp exp(j phase) format in PyTorch? A side concern is also if signal_ndims be kept 2 to compute 2D FFT or something else?
The following description, which describes the loss that I plan to implement, maybe useful.


title: Is there a way to use GPU instead of CPU for BERT tokenization?

body:
I'm using a BERT tokenizer over a large dataset of sentences (2.3M lines, 6.53bn words):
#creating a BERT tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased',
                                          do_lower_case=True)

#encoding the data using our tokenizer
encoded_dict = tokenizer.batch_encode_plus(
    df[df.data_type=='train'].comment.values,
    add_special_tokens=True,
    return_attention_mask=True,
    pad_to_max_length=True,
    max_length=256,
    return_tensors='pt'
)

As-is, it runs on CPU and only on 1 core. I tried to parallelize, but that will only speed up processing by 16x with my 16 cores CPU, which will still make it run for ages if I want to tokenize the full dataset.
Is there any way to make it run on GPU or to speed this up some other way?
EDIT:
I have also tried using a fast tokenizer:
#creating a BERT tokenizer
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased',
                                          do_lower_case=True)

Then passing the output to my batch_encode_plus:
#encoding the data using our tokenizer
encoded_dict = tokenizer.batch_encode_plus(
    df[df.data_type=='train'].comment.values,
    add_special_tokens=True,
    return_attention_mask=True,
    pad_to_max_length=True,
    max_length=256,
    return_tensors='pt'
)

But the batch_encode_plus returns the following error:

TypeError: batch_text_or_text_pairs has to be a list (got <class
'numpy.ndarray'>)


title: How to save a model using DefaultTrainer in Detectron2?

body:
How can I save a checkpoint in Detectron2, using a DefaultTrainer?
This is my setup:
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))

cfg.DATASETS.TRAIN = (DatasetLabels.TRAIN,)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 273  # Number of output classes

cfg.OUTPUT_DIR = "outputs"
os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025#0.00025  # Learning Rate
cfg.SOLVER.MAX_ITER = 10000  # 20000 MAx Iterations
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 128  # Batch Size

trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

# Save the model
from detectron2.checkpoint import DetectionCheckpointer, Checkpointer
checkpointer = DetectionCheckpointer(trainer, save_dir=cfg.OUTPUT_DIR)
checkpointer.save("mymodel_0")

I get the error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-94-c1116902655a> in <module>()

      4 checkpointer = DetectionCheckpointer(trainer, save_dir=cfg.OUTPUT_DIR)
----> 5 checkpointer.save("mymodel_0")

/usr/local/lib/python3.6/dist-packages/fvcore/common/checkpoint.py in save(self, name, **kwargs)
    102
    103         data = {}
--> 104         data["model"] = self.model.state_dict()
    105         for key, obj in self.checkpointables.items():
    106             data[key] = obj.state_dict()

AttributeError: 'DefaultTrainer' object has no attribute 'state_dict'

Docs: https://detectron2.readthedocs.io/en/latest/modules/checkpoint.html


title: TypeError: 'dict' object is not callable while loading and testing PyTorch model

body:
I am trying to load and test my final trained PyTorch model but I am getting the following error. What am I doing wrong?
Original Code:
class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

model = torch.load('Final_model.pth.tar')

normalize = transforms.Normalize(mean=[0.4914, 0.4824, 0.4467],std=[0.2471, 0.2435, 0.2616])
transform = transforms.Compose([transforms.ToTensor(),normalize])
val_set = datasets.CIFAR10('../data', train=False,download=True,transform=transform)

for i in range(48,64):
    plt.subplot(4,4,i+1-48)
    plt.subplots_adjust(hspace=1,wspace=1)
    plt.xticks([])
    plt.yticks([])
    plt.imshow(val_set.data[i])
    out = model(transform(val_set.data[i]).view(1,3,32,32))[0].tolist()
    plt.xlabel(class_names[out.index(max(out))])

Error:
    TypeError                                 Traceback (most recent call last)
<ipython-input-48-8554fc617196> in <module>()
     18     plt.yticks([])
     19     plt.imshow(val_set.data[i])
---> 20     out = model(transform(val_set.data[i]).view(1,3,32,32))[0].tolist()
     21     plt.xlabel(class_names[out.index(max(out))])
     22

TypeError: 'dict' object is not callable


title: How to create a graph neural network dataset? (pytorch geometric)

body:
How can I convert my own dataset to be usable by pytorch geometric for a graph neural network?
All the tutorials use existing dataset already converted to be usable by pytorch. For example if I have my own pointcloud dataset how can i use it to train for classification with graph neural network? What about my own image dataset for classification?


title: Why does the hash value of models change everytime I save it?

body:
I am using torch.save() to save a model file. However, everytime I save it, it changes. Why so?

netG_1 = torch.load('netG.pth')
netG_2 = torch.load('netG.pth')

torch.save(netG_1, 'netG_1.pth')
torch.save(netG_2, 'netG_2.pth')

Using md5sum *.pth:

779f0fefca47d17a0644033f9b65e594  netG_1.pth
476f502ec2d1186c349cdeba14983d09  netG_2.pth
b0ceec8ac886a11b79f73fc04f51c6f9  netG.pth

The model is an instance of this class:

https://github.com/taoxugit/AttnGAN/blob/master/code/model.py#L397


title: Trying to print class names for dog breed but it keeps saying list index out of range

body:
I am using a resnet model to classify dog breeds but when I try to print out an image with the label of dog breed it says list index out of range.
Here is my code:

import torchvision.models as models
import torch.nn as nn

model_transfer = models.resnet18(pretrained=True)

if use_cuda:
    model_transfer = model_transfer.cuda()

model_transfer.fc.out_features = 133

Then I train the model and get over 70% accuracy on the dog breeds.

Then here is my code to classify dog and print the dog breed:

data_transfer = {'train':
 datasets.ImageFolder('/data/dog_images/train',transform=transforms.Compose([transforms.RandomResizedCrop(224),transforms.ToTensor()]))}
class_names[0]
class_names = [item[4:].replace("_", " ") for item in data_transfer['train'].classes]

def predict_breed_transfer(img_path):

    image = Image.open(img_path)

    # large images will slow down processing

    in_transform = transforms.Compose([
                        transforms.CenterCrop(224),
                        transforms.ToTensor(),
                        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                             std=[0.229, 0.224, 0.225])])

    # discard the transparent, alpha channel (that's the :3) and add the batch dimension
    image = in_transform(image)[:3,:,:].unsqueeze(0)

    image = image

    output = model_transfer(image)
    pred = torch.argmax(output)

    return class_names[pred]
    predict_breed_transfer('images/Labrador_retriever_06455.jpg')

The code always predicts the dog wrong for some reason
Then when I try to print out the image and the label:

import matplotlib.pyplot as plt
def run_app(img_path):
    img = Image.open(img_path)
    dog = dog_detector(img_path)
    if not dog:
        print('hello, human!')
        plt.imshow(img)
        print('You look like a ... ')
        print(predict_breed_transfer(img_path))
    if dog:
        print('hello, dog!')
        print('Your predicted breed is ....')
        print(predict_breed_transfer(img_path))
        plt.imshow(img)
    else:
        print('Niether human nor dog')

And run a for loop that calls it on some dog images it will print some of the breeds out then it will say list index out of range and doesn't show any of the images.

The length of class_names is 133
And when I print out the resnet model the output is only 133 nodes does anyone know why it is saying list index out of range or why it is so inaccurate.

`IndexError                                Traceback (most recent
call last)
<ipython-input-26-473a9ba884b5> in <module>()
      5 ## suggested code, below
      6 for file in np.hstack((human_files[:3], dog_files[:3])):
----> 7     run_app(file)
      8
 <ipython-input-25-1d44200e44cc> in run_app(img_path)
      10         plt.show(img)
      11         print('You look like a ... ')
 ---> 12         print(predict_breed_transfer(img_path))
      13     if dog:
      14         print('hello, dog!')

 <ipython-input-20-a51fb205659e> in predict_breed_transfer(img_path)
      26     pred = torch.argmax(output)
      27
 ---> 28     return class_names[pred]
      29
predict_breed_transfer('images/Labrador_retriever_06455.jpg')
      30

IndexError: list index out of range`

Here is the full error


title: How to implement a convolution layer in python?

body:
I'm trying to create a custom convolution layer in python that will be using a 3x3 kernel. What I'm attempting to do is take this 3x3 kernel and move it along a 32x32 image where I take the dot product between the pixel values and the kernel.

What I did was create two nested for loops where I kept cutting out a 3x3 area from the image and then I used torch.matmul and torch.sum to filter the pixel values with stride = 1.

for i in range((x.shape[0]-3) + 1):
    for j in range((x.shape[1]-3) + 1):
        x_out[i][j] = torch.sum(torch.matmul(kernel, x[i:i+3, j:j+3]))

The shape of x is torch.size(32, 32) and the shape of x_out will be torch.size(30,30). The kernel is a 3x3 tensor.

The problem is the use of the nested for loops. If I run this in a neural network, it will be too slow. In order to increase the speed, I need figure out how to do this without for loops. So how can I do that?


title: What is Zero-inflated layer and how to implement it on pytorch?

body:
I am building a variational autoencoder for dimension reduction of single-cell RNA-Seq data. I managed to plot clear clusters with traditional autoencoder, but my VAE is not working. The loss decreases but stops around 5. When I visualize the latent space, I just get a random points scattered, no learned patterns whatsover.

I believe the problem is that I have not implemented the ZI-layer. The reason is that I do not even understand the purpose of it, and the math feels very unintuitive.

VASC is a deep variational autoencoder can capture non-linear variations and automatically learn a hierarchical representation of the input data. One of its purpose is to simplify visualization of scRNA-seq datasets. VASC has three major parts called: (1) the encoder network, (2) the decoder network and (3) the zero-inflated layer. The inventers of the VASC has explained it though here paper

What is the explanation behind ZI-layer?
How do I implement it to be after my decoder component?

class VAE(nn.Module):
    def __init__(self):
        super(VAE, self).__init__()

        self.n_components = 6

        self.fc0 = nn.Dropout(p=0.5)
        # For probabilistic encoder
        self.fc1 = nn.Linear(13714, 128)
        self.fc2 = nn.Linear(128, 32)
        self.fc3 = nn.Linear(32, 16)
        self.fc41 = nn.Linear(16, self.n_components)
        self.fc42 = nn.Linear(16, self.n_components)

        # For probabilistic decoder
        self.fc5 = nn.Linear(self.n_components, 16)
        self.fc6 = nn.Linear(16, 32)
        self.fc7 = nn.Linear(32, 128)
        self.fc8 = nn.Linear(128, 13714)

    def encode(self, x):
        h0 = self.fc0(x) #dropout
        h1 = self.fc1(h0)
        h2 = F.relu(self.fc2(h1))
        h3 = F.relu(self.fc3(h2))

        return self.fc41(h3), F.softplus(self.fc42(h3))

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5*logvar)
        eps = torch.randn_like(std)
        return mu + eps*std

    def decode(self, z):
        h5 = F.relu(self.fc5(z))
        h6 = F.relu(self.fc6(h5))
        h7 = F.relu(self.fc7(h6))
        h8 = torch.sigmoid(self.fc8(h7)) #output layer

        return h8

    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 13714))
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

model = VAE().to(device)
optimizer = optim.RMSprop(model.parameters(), lr=0.001)

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 13714), reduction="mean")
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

    return BCE + 0.1*KLD, BCE, KLD

log_interval = 32

for epoch in range(80):
    #model.train()
    running_loss = 0

    for batch_idx, (data, _) in enumerate(train_loader):
        data = data.to(device)
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data.float())
        loss, bce, kld = loss_function(recon_batch, data.float(), mu, logvar)
        loss.backward()
        running_loss += loss.item()
        optimizer.step()
        if batch_idx % log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}\tBCE: {:.6f}\tKLD: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.item(), bce.item(), kld.item()))

    print('====> Epoch: {} Loss: {:.4f}'.format(
          epoch, running_loss))

As I said, the latent space does not learn anything. I believe the problem is lack of the ZI-layer. If not, please could you help me detect the problem, thanks.


title: Unable to Install Torch or torch vision in pycharm I am running python 3.6

body:
Installing torch from PyCharm interpreter but error occurs. Python 3.6

Collecting torch==0.4.1.post2

  Could not find a version that satisfies the requirement torch==0.4.1.post2 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
No matching distribution found for torch==0.4.1.post2


title: Differences in encoder - decoder models between Keras and Pytorch

body:
There seem to be significant, fundamental differences in construction of encoder-decoder models between keras and pytorch. Here is keras' enc-dec blog and here is pytorch's enc-dec blog.

Some differences I noticed are the following:

Keras' model directly feeds input to LSTM layer. Whereas Pytorch uses an embedding layer for both the encoder and decoder.
Pytorch uses an embedding layer with no activation in the encoder but uses relu activation for the embedding layer in the decoder.

Given these observations, my questions are the following:

My understanding is the following, is it correct? The embedding layer is not strictly required but it helps in finding a better and denser representation of the input. It is optional and you can still build a good model without the embedding layer (dependent on the problem). This is why Keras chose not to use it in this particular example. Is this a sound reason or is there more to the story?
Why use an activation for the embedding layer in the decoder but not the encoder?
Why use 'relu' as the activation instead of 'tanh', etc for the embedding layer? What's the intuition here? I've only seen 'relu' applied to data that has spatial relation, not temporal relation.


title: Torchvision's transforms.RandomRotation() is zeroing out my image tensor

body:
Every time I use torchvision.transforms.RandomRotation with torchvision.transforms.ToTensor, I get a zero'd out image tensor.
For example, if I run this on an image:
import sys
from PIL import Image
from torchvision import  transforms

ROTATION_TRANSFORM = transforms.Compose([
    transforms.RandomRotation(30),
    transforms.ToTensor(),
])

NO_ROTATION_TRANSFORM = transforms.Compose([
    transforms.ToTensor(),
])

image = Image.open(sys.argv[1])
rotated_image_tensor = ROTATION_TRANSFORM(image)
image_tensor = NO_ROTATION_TRANSFORM(image)

print(rotated_image_tensor)
print(image_tensor)

I get this result:
tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]],

        [[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]],

        [[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         ...,
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]]])
tensor([[[0.5255, 0.5294, 0.5333,  ..., 0.4902, 0.4980, 0.4980],
         [0.5255, 0.5255, 0.5333,  ..., 0.4863, 0.4941, 0.4941],
         [0.5216, 0.5255, 0.5294,  ..., 0.4784, 0.4863, 0.4863],
         ...,
         [0.6275, 0.6353, 0.6471,  ..., 0.6275, 0.6431, 0.6510],
         [0.6196, 0.6275, 0.6431,  ..., 0.6549, 0.6471, 0.6392],
         [0.6392, 0.6471, 0.6549,  ..., 0.6667, 0.6431, 0.6275]],

        [[0.5255, 0.5294, 0.5333,  ..., 0.5216, 0.5294, 0.5294],
         [0.5255, 0.5255, 0.5333,  ..., 0.5176, 0.5255, 0.5255],
         [0.5216, 0.5255, 0.5294,  ..., 0.5098, 0.5176, 0.5176],
         ...,
         [0.6039, 0.6118, 0.6235,  ..., 0.6196, 0.6353, 0.6431],
         [0.5961, 0.6039, 0.6196,  ..., 0.6471, 0.6392, 0.6314],
         [0.6157, 0.6235, 0.6314,  ..., 0.6588, 0.6353, 0.6196]],

        [[0.5333, 0.5373, 0.5412,  ..., 0.5333, 0.5412, 0.5412],
         [0.5333, 0.5333, 0.5412,  ..., 0.5294, 0.5373, 0.5373],
         [0.5294, 0.5333, 0.5373,  ..., 0.5216, 0.5294, 0.5294],
         ...,
         [0.5569, 0.5647, 0.5765,  ..., 0.5725, 0.5882, 0.5961],
         [0.5490, 0.5569, 0.5725,  ..., 0.6000, 0.5922, 0.5843],
         [0.5686, 0.5765, 0.5843,  ..., 0.6118, 0.5882, 0.5725]]])

Isn't the image tensor losing information through this rotation? Why is it being zero'd out and how can I prevent that?
I also noticed that the result is last drastic when the amount of random rotation is reduced. For example, when I use RandomRotation(10), I still see zeros, but less of them. I'm trying to use RandomRotation to data augment my training images for a classifier, but I feel like this zeroing out of the image tensors is hurting my data severely.


title: Problem with installing pytorch in python - ctypes.CDLL(dll) - OSError: [WinError 126] The specified module could not be found

body:
I'm trying to run the run_placesCNN_basic.py code from https://github.com/CSAILVision/places365.
I'm running Windows 10 and Python 3.7.9.
I open the command prompt and try to install pytorch:
py -m pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

And i get the following process:
C:\Users\..\Downloads\places365-master>py -m pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.6.0+cpu
  Downloading https://download.pytorch.org/whl/cpu/torch-1.6.0%2Bcpu-cp37-cp37m-win_amd64.whl (167.7 MB)
     |████████████████████████████████| 167.7 MB 1.8 kB/s
Collecting torchvision==0.7.0+cpu
  Downloading https://download.pytorch.org/whl/cpu/torchvision-0.7.0%2Bcpu-cp37-cp37m-win_amd64.whl (362 kB)
     |████████████████████████████████| 362 kB 345 kB/s
Collecting numpy
  Downloading numpy-1.19.1-cp37-cp37m-win_amd64.whl (12.9 MB)
     |████████████████████████████████| 12.9 MB 63 kB/s
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 3.3 MB/s
Collecting pillow>=4.1.1
  Downloading Pillow-7.2.0-cp37-cp37m-win_amd64.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 42 kB/s
Using legacy setup.py install for future, since package 'wheel' is not installed.
Installing collected packages: numpy, future, torch, pillow, torchvision
  WARNING: The script f2py.exe is installed in 'C:\Users\..\AppData\Local\Programs\Python\Python37\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
    Running setup.py install for future ... done
  WARNING: The scripts convert-caffe2-to-onnx.exe and convert-onnx-to-caffe2.exe are installed in 'C:\Users\..\AppData\Local\Programs\Python\Python37\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed future-0.18.2 numpy-1.19.1 pillow-7.2.0 torch-1.6.0+cpu torchvision-0.7.0+cpu
WARNING: You are using pip version 20.1.1; however, version 20.2.2 is available.
You should consider upgrading via the 'C:\Users\..\AppData\Local\Programs\Python\Python37\python.exe -m pip install --upgrade pip' command.

And finally, Im trying to run the .py file:
C:\Users\..\Downloads\places365-master>py run_placesCNN_basic.py
Traceback (most recent call last):
  File "run_placesCNN_basic.py", line 6, in <module>
    import torch
  File "C:\Users\..\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)
  File "C:\Users\..\AppData\Local\Programs\Python\Python37\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

Does anyone knows why I can't run the pytorch module even if had already installed it?


title: How to use torch.gather

body:
I have a tensor of size torch.Size([3307, 1, 3]) containing values and another tensor of size torch.Size([3307, 1, 1]) containing indices of values I want to select. I am unable to understand how this can be done using torch.gather method.
Can someone please help to let me know how this can be done.


title: Python-Install Pytorch Point3D

body:
I have a problem when install Pytorch Point3D in Python
(base) C:\Users\My Pc>pip install torch-point3d
ERROR: Could not find a version that satisfies the requirement torch-point3d (from versions: none)
ERROR: No matching distribution found for torch-point3d

(base) C:\Users\My Pc>pip install torch-points3d
Collecting torch-points3d
  Using cached torch_points3d-1.1.1-py3-none-any.whl (307 kB)
Requirement already satisfied: torch-geometric<2.0.0,>=1.3.2 in c:\anaconda\lib\site-packages (from torch-points3d) (1.6.1)
Requirement already satisfied: torch-scatter<3.0.0,>=2.0.0 in c:\anaconda\lib\site-packages (from torch-points3d) (2.0.5)
Requirement already satisfied: pytorch_metric_learning<0.10.0,>=0.9.87.dev0 in c:\anaconda\lib\site-packages (from torch-points3d) (0.9.92)
Requirement already satisfied: wandb<0.9.0,>=0.8.18 in c:\anaconda\lib\site-packages (from torch-points3d) (0.8.36)
Requirement already satisfied: torch-sparse<0.7.0,>=0.6.0 in c:\anaconda\lib\site-packages (from torch-points3d) (0.6.7)
Requirement already satisfied: tensorboard<3.0,>=2.1 in c:\anaconda\lib\site-packages (from torch-points3d) (2.3.0)
Collecting numba<0.50.0,>=0.49.0
  Using cached numba-0.49.1-cp38-cp38-win_amd64.whl (2.2 MB)
Collecting torch-cluster<1.5.6
  Using cached torch_cluster-1.5.5.tar.gz (37 kB)
    ERROR: Command errored out with exit status 1:
     command: 'C:\Anaconda\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\My Pc\\AppData\\Local\\Temp\\pip-install-d6d1644l\\torch-cluster\\setup.py'"'"'; __file__='"'"'C:\\Users\\My Pc\\AppData\\Local\\Temp\\pip-install-d6d1644l\\torch-cluster\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\My Pc\AppData\Local\Temp\pip-pip-egg-info-u5a0ogxr'
         cwd: C:\Users\My Pc\AppData\Local\Temp\pip-install-d6d1644l\torch-cluster\
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\My Pc\AppData\Local\Temp\pip-install-d6d1644l\torch-cluster\setup.py", line 6, in <module>
        import torch
      File "C:\Anaconda\lib\site-packages\torch\__init__.py", line 116, in <module>
        raise err
    OSError: [WinError 182] The operating system cannot run %1. Error loading "C:\Anaconda\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Please help me.


title: Sliding Window on 2D Tensor using PyTorch

body:
How can we use a sliding window on a 2D PyTorch tensor t with shape (6, 10) such that we end up with a 3D PyTorch tensor with shape (3, 4, 10)?
For example, if we have the tensor t:
t = torch.range(1, 6*10).reshape((7, 10))

tensor([[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.],
        [11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
        [21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.],
        [41., 42., 43., 44., 45., 46., 47., 48., 49., 50.],
        [51., 52., 53., 54., 55., 56., 57., 58., 59., 60.]]

how can we reshape it (using PyTorch) such that we get the tensor:
tensor([[ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.],
        [11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
        [21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.]],

        [[11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
        [21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.],
        [41., 42., 43., 44., 45., 46., 47., 48., 49., 50.]],

        [[21., 22., 23., 24., 25., 26., 27., 28., 29., 30.],
        [31., 32., 33., 34., 35., 36., 37., 38., 39., 40.],
        [41., 42., 43., 44., 45., 46., 47., 48., 49., 50.],
        [51., 52., 53., 54., 55., 56., 57., 58., 59., 60.]])


title: coreML model converted from pytorch model giving the wrong prediction probabilities

body:
I have a pytorch binary classification model that I converted to coreML. I converted my model directly and indirectly through onnx using the following tutorials/documentation respectively
https://coremltools.readme.io/docs/pytorch-conversion, and
https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb .
The output prior to the softmax function and the probabilities are similar for both the original pytorch and the onnx model converted from PyTorch.  But the output for the coreML model converted from PyTorch via the tutorial documentation is completely incorrect. I had no errors compiling the coreML method from either method.
Checking the weights of the last layer for coreML and Pytorch seem to be the same. the output of the coreML model prior to softmax gives me
{'classLabel': '_xx', 'classLabelProbs': {'_xx': 29.15625, 'xx': -22.53125}}
while the output from the pytorch model give me  [-3.2185674 3.4477997]
The output of the conversion from onnx to coreML looks like...
58/69: Converting Node Type Add
59/69: Converting Node Type Relu
60/69: Converting Node Type Conv
61/69: Converting Node Type BatchNormalization
62/69: Converting Node Type Relu
63/69: Converting Node Type Conv
64/69: Converting Node Type BatchNormalization
65/69: Converting Node Type Add
66/69: Converting Node Type Relu
67/69: Converting Node Type GlobalAveragePool
68/69: Converting Node Type Flatten
69/69: Converting Node Type Gemm
Translation to CoreML spec completed. Now compiling the CoreML model.
Model Compilation done.

While the output of the pytorch model when I print looks like this for the final layer....
(layer4): Sequential(
(0): BasicBlock(
(conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(downsample): Sequential(
(0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): BasicBlock(
(conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
(fc): Linear(in_features=512, out_features=2, bias=True).

How do I go about resolving the quantitative errors produced from my coreML model that was converted from PyTorch?


title: Why new lines aren't generated with my fine-tuned DistilGPT2 model?

body:
I'm currently trying to fine-tune DistilGPT-2 (with Pytorch and HuggingFace transformers library) for a code completion task. My corpus is arranged like the following example:
<|startoftext|>
public class FindCityByIdService {
    private CityRepository cityRepository = ...
<|endoftext|>

My first attempt was to run the following script from transformers library:
python run_clm.py
     --model_type=gpt2 \
     --model_name_or_path distilgpt2 \
     --do_train \
     --train_file $TRAIN_FILE \
     --num_train_epochs 100 \
     --output_dir $OUTPUT_DIR \
     --overwrite_output_dir \
     --save_steps 20000 \
     --per_device_train_batch_size 4 \

After doing some generation tests, I realized that the model is not predicting \ n for any given context. I imagine that some pre-process stage or something similar is missing. But anyway, what should I do so that \ n be predicted as expected?
HF Forum question
Thanks!!


title: My PyTorch GAN is changing from producing random noise to darkness with no convergence. Why is this?

body:
My code is very basic, just following the first lab from DeepLearning.ai's GAN specialization. However, my code does not have the same output, what is the reason for this. Sorry if this is just a silly mistake, this is my first experience with GANs. I begin by creating the Generator and Discriminator classes, my random noise function, and creating my models. I then run the training loop, but after 3 epochs, all of the outputs from the GAN are black.
import torch
from torch import nn
from tqdm.auto import tqdm
from torchvision import transforms
from torchvision.datasets import MNIST
from torchvision.utils import make_grid
from torch.utils.data import DataLoader, Dataset
import matplotlib.pyplot as plt

torch.manual_seed(0)

def show_tensor_images(image_tensor,num_images=25,size=(1,28,28)):
    image_unflat=image_tensor.detach().cpu().view(-1,*size)
    image_grid=make_grid(image_unflat[:num_images],nrow=5)
    plt.imshow(image_grid.permute(1,2,0).squeeze())
    plt.show()

class Generator(nn.Module):
    def __init__(self,z_dim):
        super(Generator,self).__init__()
        self.linear1=nn.Linear(z_dim,128)
        self.bn1=nn.BatchNorm1d(128)
        self.linear2=nn.Linear(128,256)
        self.bn2=nn.BatchNorm1d(256)
        self.linear3=nn.Linear(256,512)
        self.bn3=nn.BatchNorm1d(512)
        self.linear4=nn.Linear(512,1024)
        self.bn4=nn.BatchNorm1d(1024)
        self.linear5=nn.Linear(1024,784)
        self.relu=nn.ReLU(True)
        self.sigmoid=nn.Sigmoid()
    def forward(self,x):
        x=self.linear1(x)
        x=self.bn1(x)
        x=self.relu(x)
        x=self.linear2(x)
        x=self.bn2(x)
        x=self.relu(x)
        x=self.linear3(x)
        x=self.bn3(x)
        x=self.relu(x)
        x=self.linear4(x)
        x=self.bn4(x)
        x=self.relu(x)
        x=self.linear5(x)
        x=self.sigmoid(x)
        return(x)

def get_noise(n_samples,z_dim,device='cpu'):
    return torch.randn(n_samples,z_dim,device=device)

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator,self).__init__()
        self.linear1=nn.Linear(784,512)
        self.linear2=nn.Linear(512,256)
        self.linear3=nn.Linear(256,128)
        self.linear4=nn.Linear(128,1)
        self.relu=nn.LeakyReLU(0.2,True)
    def forward(self,x):
        x=self.linear1(x)
        x=self.relu(x)
        x=self.linear2(x)
        x=self.relu(x)
        x=self.linear3(x)
        x=self.relu(x)
        x=self.linear4(x)
        return(x)

criterion=nn.BCEWithLogitsLoss()
epochs=200
z_dim=64
display_step=500
batch_size=128
lr=0.00001
device='cuda'

dataloader=DataLoader(MNIST('.',download=True,transform=transforms.ToTensor()),batch_size=batch_size,shuffle=True)

gen=Generator(z_dim).to(device)
gen_opt=torch.optim.Adam(gen.parameters(),lr=lr)
disc=Discriminator().to(device)
disc_opt=torch.optim.Adam(disc.parameters(),lr=lr)

def get_disc_loss(gen,disc,criterion,real,num_images,z_dim,device):
    noise=get_noise(num_images,z_dim,device=device)
    gen_out=gen(noise)
    disc_fake_out=disc(gen_out.detach())
    fake_loss=criterion(disc_fake_out,torch.zeros_like(disc_fake_out))
    disc_real_out=disc(real)
    real_loss=criterion(disc_real_out,torch.zeros_like(disc_real_out))
    disc_loss=(fake_loss+real_loss)/2
    return(disc_loss)

def get_gen_loss(gen,disc,criterion,num_images,z_dim,device):
    noise=get_noise(num_images,z_dim,device=device)
    gen_out=gen(noise)
    disc_out=disc(gen_out)
    loss=criterion(disc_out,torch.ones_like(disc_out))
    return loss

cur_step=0
mean_generator_loss=0
mean_discriminator_loss=0
gen_loss=False
error=False
for epoch in range(epochs):
    for x,_ in tqdm(dataloader):
        cur_batch_size=len(x)
        x=x.view(cur_batch_size,-1).to(device)

        disc_opt.zero_grad()
        disc_loss=get_disc_loss(gen,disc,criterion,x,cur_batch_size,z_dim,device)
        disc_loss.backward(retain_graph=True)
        disc_opt.step()

        gen_opt.zero_grad()
        gen_loss=get_gen_loss(gen,disc,criterion,cur_batch_size,z_dim,device)
        gen_loss.backward()
        gen_opt.step()

        mean_discriminator_loss+=disc_loss.item()/display_step
        mean_generator_loss+=gen_loss.item()/display_step

        if cur_step%display_step==0 and cur_batch_size>0:
            print(f"Step {cur_step}: Generator loss: {mean_generator_loss}, discriminator loss: {mean_discriminator_loss}")
            fake_noise = get_noise(cur_batch_size, z_dim, device=device)
            fake = gen(fake_noise)
            show_tensor_images(fake)
            show_tensor_images(x)
            mean_generator_loss = 0
            mean_discriminator_loss = 0
        cur_step += 1


title: How to convert Categorical Data type into numpy array?

body:
I have tried with this code. The output is: Categorical(probs: torch.Size([12]))). I want to extract the values from output and convert it into numpy array. Could anyone give a suggestion?

I know I can return the value of a (which is commented). But still, Is there any solution?

class Actor(nn.Module):
def __init__(self, state_size, action_size):
    super(Actor, self).__init__()
    self.state_size = state_size
    self.action_size = action_size
    self.linear1 = nn.Linear(self.state_size, 128)
    self.linear2 = nn.Linear(128, 256)
    self.linear3 = nn.Linear(256, self.action_size)

def forward(self, state):
    output = F.relu(self.linear1(state))
    output = F.relu(self.linear2(output))
    output = self.linear3(output)
    distribution = Categorical(F.softmax(output, dim=-1))
    # a=F.softmax(output, dim=-1)
    # print(a.detach().numpy())
    return distribution

    output=Actor(state)
    print(output)


title: How to attach a tensor to a particular point in the computation graph in PyTorch?

body:
As stated in the question, I need to attach a tensor to a particular point in the computation graph in Pytorch.

What I'm trying to do is this:
while geting outputs from all mini-batches, accumulate them in a list and when one epoch finishes, calculate the mean. Then, I need to calculate loss according to the mean, therefore backpropagation must consider all these operations.

I am able to do that when the training data is not much (without detaching and storing). However, this is not possible when it gets bigger. If I don't detach output tensors each time, I'm running out of GPU memories and if I detach, I lose the track of output tensors from the computation graph. Looks like this is not possible no matter how many GPUs I have since PyTorch does only use first 4 for storing output tensors if I don't detach before saving them into a list even if I assign more than 4 GPUs.

Any help is really appreciated.

Thanks.


title: why the `test process` is inside the epoch loop in deep learning?

body:
I am new to deep learning, here is the code I saw

The code is fine, but I don't understand the follow:

for epoch in range(1, args.epochs + 1):
    train(epoch)
    test(epoch)
    with torch.no_grad():
        sample = torch.randn(64, 20).to(device)
        sample = model.decode(sample).cpu()

In machine learning, when we finish training, we fix the model parameters for the test dataset. here are my two questions:

(1) In deep learning, we have the training, validation, test dataset. Is the code test(epoch) actually for validation set?  There we fix model parameters and predict for test dataset outside the epoch loop?

(2) what is with torch.no_grad(): mean? and why it also inside the epoch loop?

Thanks a lot.


title: pytorch resnet18 return features and classification

body:
I want to train the resnet18 network from pytorch with centre loss and use last layer features as a descriptor (for object tracking).
Following the answer from this post I first create a sequence from the resnet18 without last layer and then process the output through a custom linear model. Unfortunately, I get an error RuntimeError: size mismatch, m1: [1536 x 1], m2: [512 x 4] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:197 which suggests the output from the self.feature_extractor is flattened before entering the self.classifier
import torch
import torch.nn as nn
import torch.nn.functional as F

from torchvision import models

class Net(nn.Module):
    def __init__(self, num_classes=4, reid=False):
        super(Net, self).__init__()

        self.reid = reid
        self.model = models.resnet18(pretrained=True)
        num_ftrs = self.model.fc.in_features

        self.feature_extractor = torch.nn.Sequential(
            *list(self.model.children())[:-1]
        )
        self.classifier = nn.Linear(num_ftrs, num_classes)

    def forward(self, x):
        x = self.feature_extractor.forward(x)

        if self.reid:
            x = x.div(x.norm(p=2, dim=1, keepdim=True))
            return x

        y = self.classifier(x)

        return x, y

if __name__ == '__main__':
    net = Net()
    net.to('cpu')
    x = torch.randn(3, 3, 224, 224)
    x.to('cpu')
    net.eval()
    with torch.no_grad():
        y = net(x)


title: Clarification in understanding TorchScripts and JIT on PyTorch

body:
Just wanted to clarify my understanding of the way JIT and TorchScripts work and clarify a particular example.
So if im not wrong torch.jit.script converts my method or module to TorchScript. I can use my TorchScript compiled module in an environment outside python but can also just use it within python with supposed improvements and optimizations. A similar case with torch.jit.trace where the weights and operations are traced instead but follows roughly a similar idea.
If this is the case, the TorchScripted module should, in general, be at least as fast as the python interpreter typical inference time. On experimenting a bit I observed that it is most often slower then the typical interpreter inference times and on reading up a bit found out that apparently the TorchScripted module needs to be "warmed up" a bit, to achieve its best performance. On doing so I saw no changes as such to the inference times, it got better but not enough to call an improvement over the typical way of doing things(python interpreter). Furthermore, I used a third party library called torch_tvm, which when enabled supposedly halves the inference times for any way of jit-ing the module.
None of this has happened until now and I am not really able to say why.
The following is my sample code in case I have done something wrongly -
class TrialC(nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(1024, 2048)
        self.l2 = nn.Linear(2048, 4096)
        self.l3 = nn.Linear(4096, 4096)
        self.l4 = nn.Linear(4096, 2048)
        self.l5 = nn.Linear(2048, 1024)

    def forward(self, input):
        out = self.l1(input)
        out = self.l2(out)
        out = self.l3(out)
        out = self.l4(out)
        out = self.l5(out)
        return out

if __name__ == '__main__':
    # Trial inference input
    TrialC_input = torch.randn(1, 1024)
    warmup = 10

    # Record time for typical inference
    model = TrialC()
    start = time.time()
    model_out = model(TrialC_input)
    elapsed = time.time() - start

    # Record the 10th inference time (10 warmup) for the optimized model in TorchScript
    script_model = torch.jit.script(TrialC())
    for i in range(warmup):
        start_2 = time.time()
        model_out_check_2 = script_model(TrialC_input)
        elapsed_2 = time.time() - start_2

    # Record the 10th inference time (10 warmup) for the optimized model in TorchScript + TVM optimization
    torch_tvm.enable()
    script_model_2 = torch.jit.trace(TrialC(), torch.randn(1, 1024))
    for i in range(warmup):
        start_3 = time.time()
        model_out_check_3 = script_model_2(TrialC_input)
        elapsed_3 = time.time() - start_3

    print("Regular model inference time: {}s\nJIT compiler inference time: {}s\nJIT Compiler with TVM: {}s".format(elapsed, elapsed_2, elapsed_3))

And the following are the results of the above code on my CPU -
Regular model inference time: 0.10335588455200195s
JIT compiler inference time: 0.11449170112609863s
JIT Compiler with TVM: 0.10834860801696777s

Any help or clarity on this would really be appreciated!


title: Train model in Pytorch with custom loss how to set up optimizer and run training?

body:
I am new to pytorch and I am trying to run a github model I found and test it. So the author's provided the model and the loss function.
like this:
#1. Inference the model
model = PhysNet_padding_Encoder_Decoder_MAX(frames=128)
rPPG, x_visual, x_visual3232, x_visual1616 = model(inputs)

#2. Normalized the Predicted rPPG signal and GroundTruth BVP signal
rPPG = (rPPG-torch.mean(rPPG)) /torch.std(rPPG)     # normalize
BVP_label = (BVP_label-torch.mean(BVP_label)) /torch.std(BVP_label)     # normalize

#3. Calculate the loss
loss_ecg = Neg_Pearson(rPPG, BVP_label)

Dataloading
    train_loader = torch.utils.data.DataLoader(train_set, batch_size = 20, shuffle = True)

    batch = next(iter(train_loader))

    data, label1, label2 = batch

    inputs= data

Let's say I want to train this model for 15 epochs.
So this is what I have so far:
I am trying to set the optimizer and training, but I am not sure how to tie the custom loss and data loading to the model and set the 15 epoch training correctly.
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

for epoch in range(15):
  ....

Any suggestions?


title: Show multiple images in interactive mode python

body:
I'm in an interactive program which marks specific part of an images through clicking the points around it. I want to show the segmented parts in a different image. it seems that plt.imshow(image) doesn't work well in an interactive environment and i cant turn it off cause then i cant work on the main image. Is there a better command to show an specific array in an interactive mode separately.

Thanks


title: Cannot open lmdb db file in Caffe2

body:
I installed Caffe2 from source for Ubuntu.
I don't use Anaconda and my python is python2.

which python gives me

/usr/bin/python

I can import lmdb in python.
I can run tutorials from this page

My problem in testing one of the examples is "Can't open lmdb file".

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-19-c9a97f268658> in <module>()
      1 # The parameter initialization network only needs to be run once.
      2 # Now all the parameter blobs are initialized in the workspace.
----> 3 workspace.RunNetOnce(train_model.param_init_net)
      4
      5 # Creating an actual network as a C++ object in memory.

/home/pytorch/build/caffe2/python/workspace.py in RunNetOnce(net)
    199         C.Workspace.current._last_failed_op_net_position,
    200         GetNetName(net),
--> 201         StringifyProto(net),
    202     )
    203

/home/pytorch/build/caffe2/python/workspace.py in CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs)
    178 def CallWithExceptionIntercept(func, op_id_fetcher, net_name, *args, **kwargs):
    179     try:
--> 180         return func(*args, **kwargs)
    181     except Exception:
    182         op_id = op_id_fetcher()

RuntimeError: [enforce fail at db.h:190] db_. Cannot open db: /home/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb of type lmdbError from operator:
output: "dbreader_/home/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" name: "" type: "CreateDB" arg { name: "db_type" s: "lmdb" } arg { name: "db" s: "/home/caffe2_notebooks/tutorial_data/mnist/mnist-train-nchw-lmdb" }

I have all files and have read access.

I compiled pytorch with USE_LMDB=ON.

I see the same issues but can't find how they solved the issues.
My laptop system is small with Ubuntu 16.4, with Nvidia Cuda GeForce GT 750M, so quite old model.

Then I found in one page. They solved the issue with changing LMDB_MAP_SIZE in lbdb.cc. So I changed to 4G, but still have error.

constexpr size_t LMDB_MAP_SIZE = 4294967296;//4gb 1099511627776;  // 1 TB

The problem is at this line

workspace.RunNetOnce(train_model.param_init_net)

How to solve the issue?


title: PyTorch - custom loss function - calculations outside graph

body:
I'm kinda new to pytorch and trying to wrap my head around it.
I've read about custom loss functions and as far as I've seen, they cannot be decoupled from internal computational graph. This means loss function consumes tensors, does operations on them, which are implemented in pytorch, and outputs tensor. Is there any way to have decoupled loss calculation and plug it back somehow?

Use case

I'm trying to train an encoder, where latent space will be optimized to some statistical quality. This means I don't train in batches and I calculate single loss value for whole epoch and whole data set. Is it even possible to teach net that way?

class Encoder(nn.Module):
    def __init__(self, genome_size: int):
        super(Encoder, self).__init__()

        self.fc1 = nn.Linear(genome_size, genome_size)
        self.fc2 = nn.Linear(genome_size, genome_size)
        self.fc3 = nn.Linear(genome_size, genome_size)

        self.genome_size = genome_size

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)

        return x

def train_encoder(
    net: nn.Module,
    optimizer: Optimizer,
    epochs: int,
    population: Tensor,
    fitness: Tensor,
):
    running_loss = 0.0

    for epoch in range(epochs):

        optimizer.zero_grad()
        outputs = net(population)

        # encoder_loss is computationally heavy and cannot be done only on tensors
        # I need to unwrap those tensors to numpy arrays and use them as an input to another model
        loss = encoder_loss(outputs, fitness)
        running_loss += loss

        running_loss.backward()
        optimizer.step()

        print('Encoder loss:', loss)

I've seen some examples with accumulated running_loss, but my encoder is unable to learn anything. Convergence plot just jumps all over the place.

Thanks for your time <3


title: Pytorch 1.5 optimizer causing inplace operations in .step() (possible bug)

body:
In reference to this issue. The PyTorch release notes mention this change but I'm not sure how to apply their guidelines and there could just be a bug.

This code will display the error.

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# Hyper Parameters
BATCH_SIZE = 64
LR_G = 0.0001
LR_D = 0.0001
N_IDEAS = 5
ART_COMPONENTS = 15
PAINT_POINTS = np.vstack([np.linspace(-1, 1, ART_COMPONENTS) for _ in range(BATCH_SIZE)])

def artist_works():  # painting from the famous artist (real target)
    r = 0.02 * np.random.randn(1, ART_COMPONENTS)
    paintings = np.sin(PAINT_POINTS * np.pi) + r
    paintings = torch.from_numpy(paintings).float()
    return paintings

G = nn.Sequential(  # Generator
    nn.Linear(N_IDEAS, 128),  # random ideas (could from normal distribution)
    nn.ReLU(),
    nn.Linear(128, ART_COMPONENTS),  # making a painting from these random ideas
)

D = nn.Sequential(  # Discriminator
    nn.Linear(ART_COMPONENTS, 128),  # receive art work either from the famous artist or a newbie like G
    nn.ReLU(),
    nn.Linear(128, 1),
    nn.Sigmoid(),  # tell the probability that the art work is made by artist
)

opt_D = torch.optim.Adam(D.parameters(), lr=LR_D)
opt_G = torch.optim.Adam(G.parameters(), lr=LR_G)

for step in range(10000):
    artist_paintings = artist_works()  # real painting from artist
    G_ideas = torch.randn(BATCH_SIZE, N_IDEAS)  # random ideas
    G_paintings = G(G_ideas)  # fake painting from G (random ideas)

    prob_artist0 = D(artist_paintings)  # D try to increase this prob
    prob_artist1 = D(G_paintings)  # D try to reduce this prob

    D_loss = - torch.mean(torch.log(prob_artist0) + torch.log(1. - prob_artist1))
    G_loss = torch.mean(torch.log(1. - prob_artist1))

    opt_D.zero_grad()
    D_loss.backward(retain_graph=True)  # reusing computational graph
    opt_D.step()

    opt_G.zero_grad()
    G_loss.backward()
    opt_G.step()

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 1]], which is output 0 of TBackward, is at version 4; expected version 3 instead.

The only [128,1] tensor parameter would be the linear layer in D(). The error comes from somewhere within the opt_D.step() call and then calling a backward() pass on the retained graph.

The release notes have this example to clone the parameter:

def model(input, target, param):
    return (input * param ** 2 - target).norm()

param = torch.randn(2, requires_grad=True)
input = torch.randn(2)
target = torch.randn(2)
sgd = optim.SGD([param], lr=0.001)
loss = model(input, target, param.clone())
loss.backward(retain_graph=True)
sgd.step()
loss.backward()
param.grad

Which solves the problem here, but not in the initial example because it's a bit more nuanced as the parameter is coming from the Linear layer of the discriminator model.


title: Does deleting intermediate tensors affect the computation graph in PyTorch?

body:
To free up memory, I was wondering if it was possible to remove intermediate tensors in the forward method of my model. Here's a minimalized example scenario:

def forward(self, input):
    x1, x2 = input
    x1 = some_layers(x1)
    x2 = some_layers(x2)
    x_conc = torch.cat((x1,x2),dim=1)
    x_conc = some_layers(x_conc)
    return x_conc

Basically, the model passes two tensors through two separate blocks, and then concatenates the results. Further operations are applied on that concatenated tensor. Will it affect the computation graph if I run del x1 and del x2 after creating x_conc?


title: Unable to use cuda compiled code with pytorch

body:
I would like to calculate the nms using the cuda based code and use it together with torch. Following is my code,
this_file = os.path.dirname(os.path.realpath(__file__))
print(this_file)
extra_objects = ['src/cuda/nms_kernel.cu.o']
extra_objects = [os.path.join(this_file, fname) for fname in extra_objects]

ffi = BuildExtension(
    '_ext.nms',
    headers=headers,
    sources=sources,
    define_macros=defines,
    relative_to=__file__,
    with_cuda=with_cuda,
    extra_objects=extra_objects
)

However it throws the following error,

File "build.py", line 19, in
extra_objects=extra_objects   File "C:\Users\anaconda3\envs\envname\lib\site-packages\torch\utils\cpp_extension.py",
line 292, in init
super(BuildExtension, self).init(*args, **kwargs)   File "C:\Userss\anaconda3\envs\envname\lib\site-packages\setuptools_init_.py",
line 184, in init
_Command.init(self, dist)   File "C:\Users\anaconda3\envs\envname\lib\distutils\cmd.py", line 57, in
init
raise TypeError("dist must be a Distribution instance") TypeError: dist must be a Distribution instance

what am I missing?


title: Libtorch cannot detect the gpu,but pytorch successfully detects the gpu

body:
Environment：
win10 + vs2017;
python 3.7.7 + pytorch 1.4.0;
libtorch 1.7.0 + cuda 10.1
In python
print(torch.cuda.is_available()) //True

In libtorch
std::cout << torch::cuda::device_count() << std::endl;//0
std::cout << torch::cuda::is_available() << std::endl;//0
std::cout << torch::cuda::cudnn_is_available() << std::endl;//0
std::cout << torch::hasCUDA() << std::endl;//0

And I set the vs2017 project properties as following：
VC++ Directories -> include directories
H:\libtorch\include;
H:\libtorch\include\torch\csrc\api\include
VC++ Directories -> Libraries Directories
H:\libtorch\lib
Linker -> Input -> Additional Dependencies：
asmjit.lib
caffe2_detectron_ops_gpu.lib
caffe2_module_test_dynamic.lib
c10d.lib
torch.lib
torch_cuda.lib
torch_cpu.lib
mkldnn.lib
c10_cuda.lib
caffe2_nvrtc.lib
dnnl.lib
c10.lib
gloo_cuda.lib
gloo.lib
fbgemm.lib
libprotocd.lib
cpuinfo.lib
clog.lib
libprotobufd.lib
libprotobuf-lited.lib

C/C++ -> Additional Include Directories：
H:\libtorch\include;
H:\libtorch\include\torch\csrc\api\include
The main.cpp is:
#include<torch/script.h>
#include<torch/csrc/api/include/torch/cuda.h>
//#include <torch/torch.h>
#include <opencv2/opencv.hpp>
#include <iostream>
#include <memory>

int main()
{
    //torch::jit::script::Module module;
    cv::Mat frame;
    cv::Mat input;
    cv::VideoCapture capture;
    capture.open(0);
    capture >> frame;
    while (!frame.empty())
    {
        cv::cvtColor(frame, input, CV_BGR2RGB);
        torch::Tensor tensor_image = torch::from_blob(input.data, { 1, input.rows, input.cols, 3 }, torch::kByte);
        std::cout << torch::cuda::device_count() << std::endl;
        std::cout << torch::cuda::is_available() << std::endl;
        std::cout << torch::cuda::cudnn_is_available() << std::endl;
        std::cout << torch::hasCUDA() << std::endl;
        //tensor_image = tensor_image.permute({ 0, 3, 1, 2 });
        //tensor_image = tensor_image.toType(torch::kFloat);
        //tensor_image = tensor_image.to(torch::kCUDA);
        //tensor_image = tensor_image.to(torch::kU8);
        //tensor_image = tensor_image.to(torch::kCPU);
        cv::Mat label(cv::Size(frame.cols, frame.rows), CV_8UC3, tensor_image.data_ptr());
        if (!label.empty()) //
        {
            //std::cout << label.type() << std::endl;
            cv::cvtColor(label, label, CV_BGR2RGB);
            imshow("camera", label);
        }

        cv::waitKey(30);
        capture >> frame;

    }
    capture.release();
}


title: Validation loss is neither increasing or decreasing

body: Usually when a model overfits, validation loss goes up and training loss goes down from the point of overfitting. But for my case, training loss still goes down but validation loss stays at same level. Hence validation accuracy also stays at same level but training accuracy goes up. I am trying to reconstruct a 2D image from a 3D volume using UNet. Same is the behavior when I am trying to reconstruct 3D volume from 2D image but at higher loss and lower accuracy. Can someone explain the curve that why validation loss is not going down from the point of overfitting?


title: Loading PyTorch model and receiving error "AttributeError: Can't get attribute 'Model' on "

body: I'm using torch.load() to load a Pytorch model and am receiving the error "AttributeError: Can't get attribute 'Model' on <module 'main'>". Does anyone know what is causing this?


title: How to set ntasks or ntasks-per-node in slurm for running multi-node distributed training in pytorch?

body:
I want to run the official video classification script here.
I plan to run it on slurm machine.
I am not sure how to properly set the SBATCH argument --ntasks or --ntasks-per-node  if I want to run this script on 2 nodes with 8 V100 GPUs each.
like --ntasks 16
or --ntasks-per-node 8 ?
Any advice? thanks


title: Disparate result after setting requires_grad=True

body:
I am currently using PyTorch for deep neural network. I wrote a toy neural network shown below and I found that whether or not I set requires_grad=True for label y makes huge difference. When y.requires_grad=True, the neural network diverges. I am wondering why this happens.

import torch

x = torch.unsqueeze(torch.linspace(-1, 1, 100), dim=1)
y = x.pow(2) + 10 * torch.rand(x.size())

x.requires_grad = True
# this is where problem occurs
y.requires_grad = True

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)
        self.predict = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        x = torch.relu(self.hidden(x))
        x = self.predict(x)
        return x

net = Net(1, 10, 1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.5)
criterion = torch.nn.MSELoss()

for t in range(200):
    y_pred = net(x)

    loss= criterion(y_pred, y)

    optimizer.zero_grad()
    loss.backward()
    print("Epoch {}: {}".format(t, loss))
    optimizer.step()


title: Can I input a Byte Tensor to my RNN/LSTM model?

body:
I am developing an RNN/LSTM model to which I want to encode the sequence in a ByteTensor to save memory as I am limited to a very tight memory. However, when I do so, the model returns the following error:

  Expected object of scalar type Byte but got scalar type Float for argument #2 'mat2'

So, there seems to be something else that is need to be Byte tensor as well, but I do not know what is it since the console only shows an error at the line:

output = model(predictor)


title: How to calculate correct batch size for LSTM?

body:
I have a daily time series data like below.

            CashIn  CashOut
Date
2016-01-01  0.0     6500.0
2016-01-02  0.0     23110.0
2016-01-03  0.0     7070.0
2016-01-04  0.0     18520.0
2016-01-05  20840.0 22200.0
.
.
.
2019-03-25  59880.0 25500.0
2019-03-26  49270.0 17860.0
2019-03-27  45160.0 48600.0
2019-03-28  39480.0 22840.0
2019-03-29  70260.0 25950.0
2019-03-30  19250.0 24350.0
2019-03-31  46870.0 14400.0

My total data size is 1186. I want to forecast the CashIn and CashOut values between 2019-04-01 and 2019-04-30 by using LSTM.

I wrote a batch calculator like below.

def get_batches(arr, batch_size, seq_length):

    batch_size_total = batch_size * seq_length

    n_batches = len(arr)//batch_size_total

    arr = arr[:n_batches * batch_size_total]
    arr = arr.reshape((batch_size, -1))

    for n in range(0, arr.shape[1], seq_length):
        x = arr[:, n:n+seq_length]
        y = np.zeros_like(x)
        try:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, n+seq_length]
        except IndexError:
            y[:, :-1], y[:, -1] = x[:, 1:], arr[:, 0]
        yield x, y

I am trying to divide this dataset by using get_batches function to batches which have 30 sequence length because I have daily time series and I want to predict the next 30 days.

batches = get_batches(np.array(data_cashIn), 40, 30)

If I write 39 instead of 40 as a parameter in get_bathces function then I am losing the last 16 daily data but I do not want to lose these data.

How can I do this correctly?


title: In pytorch, why is there a difference in indexing a tensor between these methods

body:
I have a tensor of shape: (batch_size, max_length, embedding_dim).

Each sample corresponds to a sentence (a list of tokens).
I also have a list indices, one per sample of shape (batch_size). I'd like to collect the embedded vector that corresponds to the index in each sentence into a tensor of shape: (batch_size, embedding_dim). Sounds simple right?

However, I can't figure out what I'm doing wrong with these options:

# option 1, doesn't work:
torch.gather(batch, dim=1, index=indices)
# option 2, doesn't work:
torch.index_select(batch, dim=1, index=indices)
# option 3, doesn't work:
batch[:, indices]
# option 4, the one that works:
batch[range(batch_size), indices]

What are the differences? How to use each properly?

Thanks


title: How do I add an image title to tensorboardX?

body:
I am currently using tensorboardX to visualize input images while training a ResNet image classifier. Is there a way to add the image title along with the added image? I would like to have the image name (as stored in the dataset) displayed below the image in the tensorboard display.

So far I have tried passing a comment parameter into my tensorboard writer, which does not seem to do the job.
Currently, the relevant lines of my code are:

pretrain_train_writer = SummaryWriter('log/pretrain_train')
img_grid = vutils.make_grid(inputs[tp_idx_0], normalize=True, scale_each=True, nrow=8)
pretrain_val_writer.add_image('true_positive_class_0', img_grid, global_step=epoch, comment = img_path)