title: Train time series in PyTorch

body:
I wish to train a RNN model such that I can predict for T steps ahead in a time series model. Most of the examples that I have seen so far are centred around text.

The toy example that I have is to predict 3 sine waves as shown below:

x = torch.arange(0,30,0.05)
y = [torch.sin(x), torch.sin(x-np.pi), torch.sin(x-np.pi/2)]
y = torch.stack(y)
y = y.t()

y is of shape 600,3. However in order for the LSTM to accept it the input needs to be of shape (seq_len, batch, input_size). I was wondering if there is a function in pytorch that converts them to required format. Suppose that in my case I want seq_len=50 and batch_size=32.

This snippet of code from machinelearningmastery was the only snippet of code I found.

# convert an array of values into a dataset matrix
def create_dataset(dataset, look_back=1):
    dataX, dataY = [], []
    for i in range(len(dataset)-look_back-1):
        a = dataset[i:(i+look_back), 0]
        dataX.append(a)
        dataY.append(dataset[i + look_back, 0])
    return numpy.array(dataX), numpy.array(dataY)

Does pad_packed_sequence or anything similar in pytorch natively do this.

If anyone is interested, this is my LSTM model:

class LSTM(nn.Module):
    def __init__(self, n_features, h, num_layers=2):
        super().__init__()
        self.lstm = nn.LSTM(n_features, h, num_layers)
        self.linear = nn.Linear(h, n_features)

    def forward(self, input, h=None):
        lstm_out, self.hidden = self.lstm(input, h)
        return self.linear(lstm_out)

[optional Q] For whatever solution that I end up with, is there a way to ensure that I can do stateful training?


title: Installing OpenCV into PyCharm

body:
Idk if this is a stackoverflow-appropriate post so forgive if the question is misplaced.  I'm trying to install OpenCV into my Pycharm IDE through the conda virtual environment.  I typed conda install -c conda-forge opencv inside the PyCharm terminal and it has been doing this for 11 hours and God knows how many more to go.

Pycharm did this with PyTorch as well.  Am I doing something wrong or is this normal?


title: Why does DistilBERT huggingface forward function doesn't return pooler_output ike BERT does?

body:
I'm working with huggingface in pytorch. I'm trying to use both BERT and distilBERT for text classification problem. I want to use BERT/distilBERT output as an input for a dropout layer and later linear layer.
In BERT, I understood I have to use the pooler_output parameter that the forward function of BERT returns. However, distilBERT forward function doesn't return such a parameter, so I don't how I can use its output for the dropout layer and later linear layer.
Can anyone pls help?
Thanks!


title: pytoch: RuntimeError: shape '[-1, 14]' is invalid for input of size 491520

body:
I am trying to follow this tutorial here: However, When I am trying to train my model. I am getting this.
RuntimeError: shape '[-1, 14]' is invalid for input of size 491520

here is the line that causes this error:
logits = outputs[0]
loss_func = BCEWithLogitsLoss()
loss = loss_func(logits.view(-1,num_labels),b_labels.type_as(logits).view(-1,num_labels))

I have 14 output as you can guess from the error. When I checked the shape of logits, I got this 32 * 20 * 768.
Any help is very appreciated.


title: Train and inference in parallel on GPU in Pytorch

body:
I have an RL agent which I'd like to both train and do inference at the same time using my gpu. Hiowever using cuda with torch.multiprocessing doesn't seem to work well, and gives this error upon initialization of the other process:
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
However using the 'spawn' start method gives me another issue, because this method tries to serialize everything using pickle, and this gives me another mysterious error upon optim.zero_grad(). I have detached everything in my replay buffer before optim.zero_grad() and after...
  File "env.py", line 60, in <module>
    main()
  File "env.py", line 55, in main
    process_two.start()
  File "/home/`/anaconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/zero/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/home/zero/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 136, in reduce_tensor
    raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "
RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).

So, my question is, is there any way to do this type of asynchronous training/inference setup in parallel on a GPU? The default start method of torch.multiprocessing seems to work well on my multi-core CPU. but I'd like to move to using my GPU for more optimized matmul operations...


title: can't convert pytorch tensor dtype from float64 to double

body:
I need to convert an int to a double tensor, and I've already tried several ways including torch.tensor([x], dtype=torch.double), first defining the tensor and then converting the dtype to double with x_tensor.double(), and also defining the tensor with torch.DoubleTensor([x]) but none actually change the dtype from torch.float64. Here's the code snippet
>>> x = 6
>>> x = torch.tensor([x], dtype=torch.double)
>>> x
tensor([6.], dtype=torch.float64)

>>> x = 6
>>> x_tensor = torch.Tensor([x])
>>> x_double = x_tensor.double()
>>> x_double
tensor([6.], dtype=torch.float64)

>>> x_tensor = torch.DoubleTensor([x])
>>> x_tensor
tensor([6.], dtype=torch.float64)

Any ideas as to why it's not converting?


title: Understanding the Input Parameters in RNN

body:
I'm having a hard time to understand the different "jargons" used in RNN. They are the following:
batch_size, time_steps, inputs and instances.

Let me go through my understanding of each input parameters & please correct me where I'm wrong.
Suppose I've got a sequence of numbers and I want to predict the next number. The numbers are the following:
[1,2,3,4,5,....,100]

time_steps: This parameter means how far RNN will look into past before it predicts the future. For simplicity, I want to predict 1 number ahead. And want to do after I see 10 numbers in the past. So, in this case, time_steps will be 10.

inputs: These are the values at each time_steps. In first time_step (t) the inputs are
t0: [1]

t1: [2]

.
.
.

t10: [10]`

batch_size: This helps in efficient computation of RNN model. Suppose my batch_size is 2. In that case, at time_step 2, the RNN input will be

t0: [1]
t0: [11]

Then what's the usage of instances? E.g. in this post, instances have been used. And there are multiple cases where instances are used. Is it means each loop over batch? E.g. there are 5 batches, each of size 2. Then there will be 5 instances.

Please help me correct my understanding.

Thanks!


title: Colab not recognizing local gpu

body:
Im trying to train a Neural Network that I wrote, but it seems that colab is not recognizing the gtx 1050 on my laptop. I can't use their cloud GPU's for this task, because I run into memory constraints

print(cuda.is_available())

is returning False


title: Deploy PyTorch model to GCP resulting in memory limit exceeded

body:
I have trained a language model using transformer-lm which uses PyTorch. I would like to deploy the resulting model to the Google Cloud Platform as a Cloud Function. Cloud Functions are limited to 2 GB of memory.

The problem is that loading the model leads to an error as too much memory is used (memory limit exceeded). The model.pt file is 1.32 GB, and I use

torch.load(model_path / 'model.pt', map_location='cpu')

to load the model. Is there a way to i) compress the model? ii) not load the full model at once? or any other possibility to make it run on GCP?


title: Average error and standard deviation of error within epoch not correctly updating - PyTorch

body:
I am attempting to use Stochastic Gradient Descent but I am unsure as to why my error/loss is not decreasing. The information I am using from the train dataframe is the index (each sequence) and the binding affinity, and the goal is to predict the binding affinity. Here is what the head of the dataframe looks like:

For the training, I make a one-hot of a sequence and calculate a score with another matrix, and the goal is to get this score to be as close to the binding affinity as possible (for any given peptide). How I calculate the score and my training loop is shown in my code below but I don't think an explanation is necessary to solve why my error fails to decrease.

#ONE-HOT ENCODING
AA=['A','R','N','D','C','Q','E','G','H','I','L','K','M','F','P','S','T','W','Y','V']
loc=['N','2','3','4','5','6','7','8','9','10','11','C']
aa = "ARNDCQEGHILKMFPSTWYV"
def p_one_hot(seq):
    c2i = dict((c,i) for i,c in enumerate(aa))
    int_encoded = [c2i[char] for char in seq]
    onehot_encoded = list()
    for value in int_encoded:
        letter = [0 for _ in range(len(aa))]
        letter[value] = 1
        onehot_encoded.append(letter)
    return(torch.Tensor(np.transpose(onehot_encoded)))

#INITALIZE TENSORS
a=Var(torch.randn(20,1),requires_grad=True) #initalize similarity matrix - random array of 20 numbers
freq_m=Var(torch.randn(12,20),requires_grad=True)
freq_m.data=(freq_m.data-freq_m.min().data)/(freq_m.max().data-freq_m.min().data)#0 to 1 scaling
optimizer = optim.SGD([torch.nn.Parameter(a), torch.nn.Parameter(freq_m)], lr=1e-6)
loss = nn.MSELoss()

#TRAINING LOOP
epochs = 100
for i in range(epochs):
    #RANDOMLY SAMPLE DATA
    train = all_seq.sample(frac=.03)
    names = train.index.values.tolist()
    affinities = train['binding_affinity']
    print('Epoch: ' + str(i))
    #forward pass
    iteration_loss=[]
    for j, seq in enumerate(names):
        sm=torch.mm(a,a.t()) #make simalirity matrix square symmetric
        freq_m.data=freq_m.data/freq_m.data.sum(1,keepdim=True) #sum of each row must be 1 (sum of probabilities of each amino acid at each position)
        affin_score = affinities[j]
        new_m = torch.mm(p_one_hot(seq), freq_m)
        tss_m = new_m * sm
        tss_score = tss_m.sum()
        sms = sm
        fms = freq_m
        error = loss(tss_score, torch.FloatTensor(torch.Tensor([affin_score])))
        iteration_loss.append(error.item())
        optimizer.zero_grad()
        error.backward()
        optimizer.step()
    mean = statistics.mean(iteration_loss)
    stdev = statistics.stdev(iteration_loss)
    print('Epoch Average Error: ' + str(mean) + '. Epoch Standard Deviation: ' + str(stdev))
    iteration_loss.clear()

After each epoch, I print out the average of all errors for that epoch as well as the standard deviation. Each epoch runs through about 45,000 sequences. However, after 10 epochs I'm still not seeing any improvement with my error and I'm unsure as to why. Here is the output I am seeing:

Are there any ideas as to what I'm doing wrong? I'm new to PyTorch so any help is appreciated! Thank you!


title: How to match items in two pytorch tensors

body:
I have two  pytorch tensors have the same number of rows,
  ( tensor1-size  is (6,4) and  tensor2-size  is (6), The first tensor is bounding-box coordinate points, whereas the second tensor is the prediction scores.

tensor1 =
([[780.8306,  98.1060, 813.8367, 149.8171],
[585.6562, 117.6804, 621.6012, 166.3151],
[ 88.4085, 117.1313, 129.3327, 173.1145],
[223.1263, 239.2682, 255.1892, 270.7897],
[194.4088, 117.9768, 237.3028, 166.1765],
[408.9165, 109.0131, 441.0802, 141.2362]])
tensor2 =
([0.9842, 0.9751, 0.9689, 0.8333, 0.7021, 0.6191])

How could I concatenate  every item in tensor2 with its correspondent row in tensor1)?

I want get predilection-score and the bounding box.
The output to be something like:

prediction  bounding box
0.9842      780.8306,  98.1060, 813.8367, 149.8171
0.9751      585.6562, 117.6804, 621.6012, 166.3151]
0.96898     88.4085, 117.1313, 129.3327, 173.1145
0.8333      223.1263, 239.2682, 255.1892, 270.7897
0.7021      194.4088, 117.9768, 237.3028, 166.1765
0.6191      408.9165, 109.0131, 441.0802, 141.2362

I appreciate If anyone could  help,
Thank you


RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

I am getting the above error for code:-

def get_model(path, device):
    model = models.vgg16(pretrained=False)

    for param in model.parameters():
        param.requires_grad = False
    n_inputs = model.classifier[6].in_features

    model.classifier[6] = torch.nn.Sequential(
        torch.nn.Linear(n_inputs, 256), torch.nn.ReLU(), torch.nn.Dropout(0.2),
        torch.nn.Linear(256, 10), torch.nn.LogSoftmax(dim=1))

    model.load_state_dict(torch.load(path), map_location=torch.device('cpu'))
    model.to(device)
    model.eval()
    return model

device = torch.device("cpu")
model = get_model('vgg16.pt', device)


title: Pruning using Pytorch on a complicated model

body:
So I am trying to use torch.nn.utils.prune.global_unstructured.
I did it on a simple model and that worked. model.cov2 or other layers and that works. I am trying to do it on a model that's (nested)? I get errors as:
AttributeError: 'CNN' object has no attribute 'conv1'

and other errors. I tried everything to access this deep cov1, but I couldn't.
You can find the model code below:
class CNN(nn.Module):
def __init__(self):
    """CNN Builder."""
    super(CNN, self).__init__()

    self.conv_layer = nn.Sequential(

        # Conv Layer block 1
        nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, padding=1),
        nn.BatchNorm2d(32),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=2, stride=2),

        # Conv Layer block 2
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Dropout2d(p=0.05),

        # Conv Layer block 3
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, padding=1),
        nn.BatchNorm2d(256),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=2, stride=2),
    )

    self.fc_layer = nn.Sequential(
        nn.Dropout(p=0.1),
        nn.Linear(4096, 1024),
        nn.ReLU(inplace=True),
        nn.Linear(1024, 512),
        nn.ReLU(inplace=True),
        nn.Dropout(p=0.1),
        nn.Linear(512, 100)
    )

def forward(self, x):
    """Perform forward."""
    # conv layers
    x = self.conv_layer(x)
    # flatten
    x = x.view(x.size(0), -1)
    # fc layer
    x = self.fc_layer(x)
    return x

How can I apply pruning on this model?


title: TensorBoard legend and label axis

body: Hi  Is it possible to have a legend and axis labels in TensorBoard? if so how? the last post regarding similar question was raised in 2017. I am wondering if there has been an update since then  Thanks a lot


title: How to use torchvision.datasets.ImageFolder on a folder stored in AWS S3 bucket

body:
Lets say I have a huge image data stored in AWS S3 bucket. I need to use trochvision.datasets.ImageFolder as my training data is in appropriate folders. So I want to use the following code in my Sagemaker notebook.
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

data_location = "s3_bucket/folder"
batch_size= 16

transforms_= transforms.Compose([ transforms.RandomHorizontalFlip(p= 0.25),
                                transforms.RandomRotation(30),
                               transforms.RandomCrop(256, pad_if_needed=True),
                               transforms.ToTensor(),
                               transforms.Normalize([0.5, 0.5,0.5], [0.5,0.5,0.5]),
                               ])

train_dataset= datasets.ImageFolder(data_location, transform= transforms_)
train_loader= DataLoader(train_dataset, batch_size= batch_size, shuffle= True)

How can I do this?


title: PyTorch how to compute second order Jacobian?

body:
I have a neural network that's computing a vector quantity u. I'd like to compute first and second-order jacobians with respect to the input x, a single element.
Would anybody know how to do that in PyTorch? Below, the code snippet from my project:
import torch
import torch.nn as nn

class PINN(torch.nn.Module):

    def __init__(self, layers:list):
        super(PINN, self).__init__()
        self.linears = nn.ModuleList([])
        for i, dim in enumerate(layers[:-2]):
            self.linears.append(nn.Linear(dim, layers[i+1]))
            self.linears.append(nn.ReLU())
        self.linears.append(nn.Linear(layers[-2], layers[-1]))

    def forward(self, x):
        for layer in self.linears:
            x = layer(x)
        return x

I then instantiate my network:
n_in = 1
units = 50
q = 500

pinn = PINN([n_in, units, units, units, q+1])
pinn

Which returns
PINN(
  (linears): ModuleList(
    (0): Linear(in_features=1, out_features=50, bias=True)
    (1): ReLU()
    (2): Linear(in_features=50, out_features=50, bias=True)
    (3): ReLU()
    (4): Linear(in_features=50, out_features=50, bias=True)
    (5): ReLU()
    (6): Linear(in_features=50, out_features=501, bias=True)
  )
)

Then I compute both FO and SO jacobians
x = torch.randn(1, requires_grad=False)

u_x = torch.autograd.functional.jacobian(pinn, x, create_graph=True)
print("First Order Jacobian du/dx of shape {}, and features\n{}".format(u_x.shape, u_x)

u_xx = torch.autograd.functional.jacobian(lambda _: u_x, x)
print("Second Order Jacobian du_x/dx of shape {}, and features\n{}".format(u_xx.shape, u_xx)

Returns
First Order Jacobian du/dx of shape torch.Size([501, 1]), and features
tensor([[-0.0310],
        [ 0.0139],
        [-0.0081],
        [-0.0248],
        [-0.0033],
        [ 0.0013],
        [ 0.0040],
        [ 0.0273],
        ...
        [-0.0197]], grad_fn=<ViewBackward>)

Second Order Jacobian du/dx of shape torch.Size([501, 1, 1]), and features
tensor([[[0.]],

        [[0.]],

        [[0.]],

        [[0.]],

        ...

        [[0.]]])

Should not u_xx be a None vector if it didn't depend on x?
Thanks in advance


title: How to resize all 4 dimensions (NCHW) in PyTorch with F.interpolate?

body:
I've been trying to figure out how to resize the Batch, Channels, Height, and Width dimensions in a tensor. Currently I am able to resize the Channels, Height, and Width dimensions, but the Batch dimension remains the same.
x = torch.ones(3,4,64,64)

x = F.interpolate(x.unsqueeze(0), size=(3,4,4), mode="trilinear").squeeze(0)

x.size() # (3,3,4,4) # batch dimension has not been resized.
# I need x to be resized so that it has a size of: (1,3,4,4)

# Is this a good idea?
x = x.permute(1,0,2,3)
x = F.interpolate(x.unsqueeze(0), size=(1, x.size(2), x.size(3)), mode="trilinear").squeeze(0)
x = x.permute(1,0,2,3)

x.size() # (1,3,4,4)

Should I permute the tensor to resize the batch dimension? Or iterate through it in some way?


title: Errno 22 when loading image

body:
I am having trouble opening an image with the neural_style script found here. When passing arguments to the script in Spyder's IPython console, I have continually received the following error:

OSError: [Errno 22] Invalid argument:'<C:/Users/Caleb/Documents/Python_Scripts/pytorch/project1/content_images/20180727_201120.jpg>'

I have attempted changing forward slashes to backslashes, using a filepath that starts at my working directory (not including the folders above it), and eliminating the filepath from the entry (i.e. simply entering the filename). None of these fixes have solved my problem. The arguments follow the form:

python neural_style/neural_style.py eval \
    --content-image </path/to/content/image> \
    --model </path/to/saved/model> \
    --output-image </path/to/output/image> \
    --cuda 1

What amendments must I make to my arguments to no longer receive this error? I am using Windows 10 and Python 3.6.5. The arguments are read by argparse.

Please note that I used () instead of <> above, not sure how to properly use <> without deleting the interior content.

Thank you!


title: Neural networks fails to approximate simple multiplication and division

body:
I am trying to fit simple feedforward neural networks on simple data where my goal is to just approximate (abc)/d

max_a=2
max_b = 3000
max_c=10
max_d=1

def generate_data(no_elements=10000):
    a = np.random.uniform(0,max_a,no_elements)
    b = np.random.uniform(1,max_b,no_elements)
    c=np.random.uniform(0.001,max_c,no_elements)
    d=np.random.uniform(0.00001,max_d,no_elements)
    df=pd.DataFrame({"a":a,"b":b,"c":c,"d":d})

    e=(df.a*df.b*df.c)/df.d
    df["e"]=e
    return(df)

this is how i am generating data

then  I did data normalization

df = generate_data(5000)
np_df=df.iloc[:,:4]
means=np.mean(np_df,axis=0,keepdims=True)
stds=np.std(np_df,axis=0,keepdims=True)
x_train = (np_df-means)/stds
y_train = np_df[:,4]

and I have  built a simple pytorch network for regression so as it has to predict 'e'

class network_Regression(nn.Module):
    def __init__(self,layers):
        super(network_Regression, self).__init__()
        self.linear = nn.ModuleList()
        self.relu = nn.ModuleList()
        self.layers = layers
        for i in range(len(layers)-1):
            self.linear.append(nn.Linear(layers[i],layers[i+1]))
            if i+1 !=  len(layers)-1:
                self.relu.append(nn.ReLU())

    def forward(self,out):
        for i in range(len(self.relu)):
            out = self.linear[i](out)
            out = self.relu[i](out)
        out = self.linear[-1](out)
        return out

model = network_Regression([4,10,10,1])
criterion= nn.MSELoss()
optimizer=optim.Adam(model.parameters())

but when I tried to train these networks tried epochs from [1000 to 0.5M]

still, it isn't able to find a simple formula ((abc)/d)=e

I tried to change various hidden layer levels but loss been around 9 digits

model.train()
num_epochs = 1000
loss_list=[]
for epoch in range(num_epochs):
    for batch_idx, (data, target) in enumerate(data_loader):
    #print(batch_idx)
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data.float())
        loss = criterion(output, target.float())
        #print(batch_idx, loss.data[0])
        loss.backward()
        optimizer.step()
        if epoch >2:
            if batch_idx % 200 == 0:
                loss_list.append(loss.data.item())
        if batch_idx % 400 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, batch_idx * len(data), len(data_loader.dataset),
            100. * batch_idx / len(data_loader), loss.data.item()))


title: Understanding linear layer in sequence labeling with pytorch lstm

body:
I'm trying to get a grip on LSTM and pytorch. Given are sequences of varying length. Every datapoint in a sequence is composed of 8 features, and every datapoint belongs to one of 6 classes (0-5). I want to learn how to use a LSTM to predict the labels of those datapoints.

So far i've made a couple of attempts to tackle this, but i fear that i miss some basic knowledge when it comes to this topic.

I've started by padding the sequences with -1 and labels to a max_amount.

I have dataloaders (train, test, val) which take care of delivering batches of shape [batch_size, max_seq_length, num_features]. The model is rather simple and consists of the lstm and a linear layer.

LSTM:

...
  self.lstm = nn.LSTM(
            input_size=self.sequence_dimension, # 8 features
            hidden_size=self.hidden_dim,
            num_layers=self.n_layers,
            batch_first=True,
        )
        self.linear_fc = nn.Linear(self.hidden_dim, self.output_size)
...

init:

    def init_hidden(self):
        # the weights are of the form (nb_layers, batch_size, nb_lstm_units)
        hidden_a = torch.randn(self.n_layers, self.batch_size, self.hidden_dim)
        hidden_b = torch.randn(self.n_layers, self.batch_size, self.hidden_dim)

        hidden_a = Variable(hidden_a)
        hidden_b = Variable(hidden_b)

        return (hidden_a, hidden_b)

forward:

     def forward(self, X):
        # shape of X: [batch_size, max_seq_len, feature_size]
       self.hidden = self.init_hidden()

        # get unpadded sequence lenghts (padding: -1)
        lenghts = []
        for batch in X:
            for i, elem in enumerate(batch):
                if elem[0] == -1:
                    lenghts.append(i-1)
                    break

        # pack the padded sequences, length contains unpadded lenghts (eg., [43,46,67,121]
        x_packed = torch.nn.utils.rnn.pack_padded_sequence(X, lenghts, batch_first=True, enforce_sorted=False)

        lstm_out, self.hidden = self.lstm(x_packed.float(), self.hidden)

        # unpack
        x_unpacked, seq_len = torch.nn.utils.rnn.pad_packed_sequence(lstm_out, batch_first=True)

        #squash the batches from [batch_size, max_seq_len_of_this_batch, hidden_dim]
        #to [batch_size*max_seq_len_of_this_batch, hidden dim]
        batches_squashed = x_unpacked.contiguous().view(-1, x_unpacked.shape[2])

        #feed it to linear
        y_pred = self.linear_fc(batches_squashed)

        #unsquash batches to [batch_size, max_seq_len_of_this_batch, hidden_dim]
        y_pred = y_pred.contiguous().view((batch_size, max(lenghts),self.output_size))
        return y_pred, max(lenghts)

I'm very unsure if i do this correctly, especially with the linear layer. It receives a tensor of shape [batchsize*max_seq_len, hidden_layers], is that correct? There are still padded values in there...

In my train loop i simply init the hidden state for every epoch and then loop over (input, labels) from the dataloader:

    loss_funtion = nn.CrossEntropyLoss(ignore_index=-1)
    optimizer = torch.optim.SGD(net.parameters(), lr=lr, momentum=0.9)

   for inputs, labels in train_loader:
        counter+=1
        net.zero_grad()
        if train_on_gpu:
            inputs = inputs.to(device)
            labels = labels.to(device)

        output, _ = net(inputs)

        #remove excessive padding for labels
        labels = decreasing_padding(labels, max_padding_for_this_batch)

        #transposing the output to fit the crossEntropy definiton
        loss = loss_funtion(output.transpose(1, 2),labels.long())
        loss.backward()
        optimizer.step()

The LSTM often just outputs a single label for the whole sequence (there are sequences for which this is correct). Loss and accurracy on train and validation set indicate that it is not learning. While the loss decreases slightly it fluctuates highly.

I suspect the linear layer to be implemented incorrectly.

Should i pass every hidden state (without padding?) one by one through the layer and stack its outputs?

I would be tremendously grateful if someone could point out my mistakes and misconceptions, i've spent quite some time with this so far.


title: How to run Tensorboard in parallel

body:
https://github.com/NVIDIA/DeepRecommender

According to the above page, I tried to run the NVIDIA's DeepRecommender program.After I activated the pytorch, I run the program as below but it failed.

[I run this Command]

$ python run.py --gpu_ids 0 \
 --path_to_train_data Netflix/NF_TRAIN \
 --path_to_eval_data Netflix/NF_VALID \
 --hidden_layers 512,512,1024 \
 --non_linearity_type selu \
 --batch_size 128 \
 --logdir model_save \
 --drop_prob 0.8 \
 --optimizer momentum \
 --lr 0.005 \
 --weight_decay 0 \
 --aug_step 1 \
 --noise_prob 0 \
 --num_epochs 12 \
 --summary_frequency 1000

[The comments of the Guide.]

  Note that you can run Tensorboard in parallel

  $ tensorboard --logdir=model_save

[My Question]

The guide says as above.I don't know how to run in parallel.Please tell me the way. Shoud I open 2 terminal  windows?

[Enviroment]

The detail of the enviroment is as follow.
---> Ubuntu 18.04 LTS, python 3.6, Pytorch 1.2.0, CUDA V10.1.168

[The 1st trial]

After I activated the pytorch,

 $source activate pytorch

 $python run.py --gpu_ids 0 \  (The long parameters are abbreviated here.)

[The Error messages of the 1st trial]

Traceback (most recent call last):
  File "run.py", line 13, in
    from logger import Logger
  File "/home/user/NVIDIA_DeepRecommender/DeepRecommender-mp_branch/logge                                                                                r.py", line 4, in
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

[The 2nd trial]

After I activated the tensorflow-gpu,

 $ source activate tensorflow-gpu

 $python run.py --gpu_ids 0 \  (The long parameters are abbreviated here.)

[The Error messages of the 2nd trial.]

Traceback (most recent call last):
  File "run.py", line 2, in
    import torch
ModuleNotFoundError: No module named 'torch'

[Expected result]

  $ python run.py --gpu_ids 0 \
   The program can run with no error and finish training the model.


title: resetting connections in neural networks with python

body:
I have a code that is subclassing nn.module .

i don't know what exactly reset_() function does and i did not find any reset_() function in nn.module source code.

who knows how can i use this for resetting connections in neural networks when there is no operations in that and also there is no function in name reset_() in parent class???

    class Connection(torch.nn.module):
      super().__init__()

      def reset_(self) -> None:

      #Contains resetting logic for the connection.#

      super().reset_()


title: how to get the row and column indices of the maximum element from a 2D pytorch tensor?

body:
Is there any way that I can retrieve the row and column indices of the greatest element contained in 2-dimensional pytorch tensor? For example, see the pytorch tensor a below:
a
>> torch.tensor([1,2,3],
                [9,5,4],
                [6,7,8])

The greatest element in the tensor a is 9, which happens at the first column of the second row. If I change that into a python column and row index that starts from zero, the column index  of the element would 0 and the row index would be 1.
Is there any way that I can retrieve the index [1,0] from the 2 dimensional pytorch tensor a?


title: Meta Learning with pytorch DistributedDataParallel, the result changes when rank changes?

body:
I want to implement meta learning with pytorch DistributedDataParallel. However, there are two issues:

After setting loss.backward(retain_graph=True, create_graph=True), an error occured, said RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. If only set retain_graph=True, it works fine.
When I set retain_graph=True and run the code, I found that the second order gradient changes with respect to the number of ranks, while the loss doesn't change. I found the key problem may come from the get_update_network(). However, I can work it out that how this happened and how to fix it.

e.g.

DataParallel

grad1 :  tensor(3.5100, device='cuda:0')
net2 :  tensor([[ -3.9810, -13.5981,  -4.1402, -36.6334]], device='cuda:0',
       grad_fn=<SubBackward0>)
grad2 :  tensor(7.0200, device='cuda:0')
loss1 : -8.781850814819336, loss2 : -1550.81201171875

DistributedDataParallel

rank 1

grad1 :  tensor(3.5100, device='cuda:0')
net2 :  tensor([[ -3.9810, -13.5981,  -4.1402, -36.6334]], device='cuda:0',
       grad_fn=<SubBackward0>)
grad2 :  tensor(7.0200, device='cuda:0')
loss1 : -8.781850814819336, loss2 : -1550.81201171875

rank 2

grad1 :  tensor(3.5100, device='cuda:0')
net2 :  tensor([[ -3.9810, -13.5981,  -4.1402, -36.6334]], device='cuda:0',
       grad_fn=<SubBackward0>)
grad2 :  tensor(6.5200, device='cuda:0')
loss1 : -8.781851768493652, loss2 : -1550.81201171875

rank 4

grad1 :  tensor(3.5100, device='cuda:0')
net2 :  tensor([[ -3.9810, -13.5981,  -4.1402, -36.6334]], device='cuda:0',
       grad_fn=<SubBackward0>)
grad2 :  tensor(5.5200, device='cuda:0')
loss1 : -8.781850814819336, loss2 : -1550.81201171875

The demo code is followed:

import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel
from torch.utils.data import DataLoader
from torch.utils.data.dataset import Dataset

torch.manual_seed(1234)

def put_theta(model, theta):
    def k_param_fn(tmp_model, name=None):
        if len(tmp_model._modules) != 0:
            for (k, v) in tmp_model._modules.items():
                if name is None:
                    k_param_fn(v, name=str(k))
                else:
                    k_param_fn(v, name=str(name + '.' + k))
        else:
            for (k, v) in tmp_model._parameters.items():
                if not isinstance(v, torch.Tensor):
                    continue
                tmp_model._parameters[k] = theta[str(name + '.' + k)]

    k_param_fn(model)
    return model

def get_updated_network(old, new, lr, load=False):
    updated_theta = {}
    state_dicts = old.state_dict()
    param_dicts = dict(old.named_parameters())
    # print(param_dicts['module.backbone.conv1.0.weight'].grad.sum(), '\n')
    for i, (k, v) in enumerate(state_dicts.items()):
        if k in param_dicts.keys() and param_dicts[k].grad is not None:
            updated_theta[k] = param_dicts[k] - lr * param_dicts[k].grad
        else:
            updated_theta[k] = state_dicts[k]
    if load:
        new.load_state_dict(updated_theta)
    else:
        new = put_theta(new, updated_theta)
    return new

class Datax(Dataset):
    def __getitem__(self, item):
        data = [0.01, 10, 0.4, 33]
        data = torch.tensor(data).view(4).float() + item
        # data = torch.randn((4,))
        return data

    def __len__(self):
        return 100

class Net2(nn.Module):
    def __init__(self):
        super(Net2, self).__init__()
        self.conv = nn.Sequential(
            nn.Linear(4, 1, bias=False),
        )

    def forward(self, x):
        x = self.conv(x)
        return x.mean()

def train(rank, world_size):
    if rank == -1:
        net = nn.DataParallel(Net2()).cuda()
        net2 = nn.DataParallel(Net2()).cuda()
        opt1 = torch.optim.SGD(net.parameters(), lr=1e-3)
        dataset = Datax()
        loader = DataLoader(dataset, batch_size=8, shuffle=False, pin_memory=True, sampler=None)
    else:
        net = DistributedDataParallel(Net2().cuda(), device_ids=[rank], find_unused_parameters=True)
        net2 = DistributedDataParallel(Net2().cuda(), device_ids=[rank], find_unused_parameters=True)
        opt1 = torch.optim.SGD(net.parameters(), lr=1e-3)
        dataset = Datax()
        sampler = torch.utils.data.distributed.DistributedSampler(dataset, rank=rank, shuffle=False)
        loader = DataLoader(dataset, batch_size=8//world_size, shuffle=False, pin_memory=True, sampler=sampler)

    for i, data in enumerate(loader):
        data = data.cuda()
        l1 = net(data).mean()
        opt1.zero_grad()
        # when set the create_graph=True, error occurs in the second loop
        l1.backward(retain_graph=True)
        if rank <= 0:
            # first gradients and losses are all the same for all ranks
            print('grad1 : ', net.module.conv[0].weight.grad[0, 0])

        # net2 = net  # this line works fine, so the error comes from this line
        # However, all ranks outputs same architecture and same loss, but the grad is different.
        net2 = get_updated_network(net, net2, 1)
        if rank <= 0:
            for k, v in net2.named_parameters():
                print('net2 : ', v)

        l2 = net2(data).mean()
        l2.backward()
        if rank != -1:
            dist.all_reduce(l1), dist.all_reduce(l2)
            l1 = l1 / world_size
            l2 = l2 / world_size
        if rank <= 0:
            print('grad2 : ', net.module.conv[0].weight.grad[0, 0])
            print('loss1 : {}, loss2 : {}'.format(l1.item(), l2.item()))

        opt1.step()

        if i == 0:
            break

def dist_train(proc, ngpus_per_node, args):
    backend = 'nccl'
    url = 'tcp://127.0.0.1:23458'
    world_size = args
    dist.init_process_group(backend=backend, init_method=url, world_size=world_size, rank=proc)
    torch.cuda.set_device(proc)
    train(proc, world_size)

if __name__ == '__main__':
    train(-1, 4)
    for i in [1, 2, 4]:
        print('\n')
        ranks = i
        torch.manual_seed(1234)
        mp.spawn(dist_train, nprocs=ranks, args=(ranks, ranks))


title: Pytorch can move tensor to gpu, but nvidia-smi shows no GPU memory in use

body:
hello~ I am very confused with this situation.
first, both my tf and pytorch can detect my gpu (use torch.cuda,is_available())
but my model which runs just fine on gpus just few days before can only run on cpus today.
it seems pytorch and tf skip passing model to gpu directly.
second, I have test in python interactive mode with:
import torch
x = torch.randn(10000,1000).cuda()

this line worked fine, and when I type
x.device

python shows me that x is on gpu device index 0
but at the same time.
NO GPU MEMORY in use in nvidia-smi
third, when I monitor my gpu states with
watch -n 1 nvidia-smi

I found temperature or power of my gpus does not vary at all for a long while.
any help will be appreciated!!


title: Why is convolution in cuDNN non-deterministic?

body: The PyTorch documentary says, when using cuDNN as backend for a convolution, one has to set two options to make the implementation deterministic. The options are torch.backends.cudnn.deterministic = True and torch.backends.cudnn.benchmark = False. Is this because of the way weights are initialized?


title: Writing a dropout layer using nn.Sequential() method + Pytorch

body:
I am trying to create a Dropout Layer for my neural network using nn.Sequential() like this:
class DropoutLayer(nn.Module):
    def __init__(self, p):
        super().__init__()
        self.p = p
    def forward(self, input):
        if self.training:
            u1 = (np.random.rand(*input.shape)<self.p)
            u1 *= input
            return u1
        else:
            input *= self.p
model = nn.Sequential(Flatten(),DropoutLayer(p = 0.7),nn.LogSoftmax(dim = -1))
opt = torch.optim.Adam(modelDp.parameters(), lr=0.005)
train(modelDp, opt, 5)

But I get this error:
ValueError: optimizer got an empty parameter list


title: Pytorch NN error: Expected input batch_size (64) to match target batch_size (30)

body:
I'm currently training a neural network to classify food groups of food images, resulting in 5 output classes. However, whenever I begin training the network, I get this error:
ValueError: Expected input batch_size (64) to match target batch_size (30).

Here's my neural network definition and training code. I'd really appriciate help, I'm relatively new to pytorch and can't figure out exactly what the problem is in my code. Thanks!
#Define the Network Architechture

model = nn.Sequential(nn.Linear(7500, 4950),
                      nn.ReLU(),
                      nn.Linear(4950, 1000),
                      nn.ReLU(),
                      nn.Linear(1000, 250),
                      nn.ReLU(),
                      nn.Linear(250, 5),
                      nn.LogSoftmax(dim = 1))

#Define loss
criterion = nn.NLLLoss()

#Initial forward pass
images, labels = next(iter(trainloader))
images = images.view(images.shape[0], -1)
print(images.shape)

logits = model(images)
print(logits.size)
loss = criterion(logits, labels)
print(loss)

#Define Optimizer
optimizer = optim.SGD(model.parameters(), lr = 0.01)

Training the Network:
epochs = 10

for e in range(epochs):
    running_loss = 0
    for image, labels in trainloader:
        #Flatten Images
        images = images.view(images.shape[0], -1)
        #Set gradients to 0
        optimizer.zero_grad()

        #Output
        output = model(images)
        loss = criterion(output, labels) #Where the error occurs
        loss.backward()

        #Gradient Descent Step
        optimizer.step()
        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")


title: How to add new dimension in tensor

body: I currently have following tensor with dimensions [batch_size, channel, width, height] but I want to add one dimension to the above tensors at third index as [batch_size, channel, length, width, height] where length is the dimension that I want to add into my tensor. Length value will be defined as the no_of_iterations for batch size. Can you please tell me how should I add need dimension and set its value.


title: Translate Keras functional API to PyTorch nn.Module - Conv2d

body:
I'm trying to translate the following Inception code from tutorial in Keras functional API (link) to PyTorch nn.Module:
    def conv_module(x, K, kX, kY, stride, chanDim, padding="same"):
        # define a CONV => BN => RELU pattern
        x = Conv2D(K, (kX, kY), strides=stride, padding=padding)(x)
        x = BatchNormalization(axis=chanDim)(x)
        x = Activation("relu")(x)
        # return the block
        return x

    def inception_module(x, numK1x1, numK3x3, chanDim):
        # define two CONV modules, then concatenate across the
        # channel dimension
        conv_1x1 = conv_module(x, numK1x1, 1, 1, (1, 1), chanDim)
        conv_3x3 = conv_module(x, numK3x3, 3, 3, (1, 1), chanDim)
        x = concatenate([conv_1x1, conv_3x3], axis=chanDim)
        # return the block
        return x

I'm having trouble translating the Conv2D. If I understand correctly:

There is no in_features in Keras - how should I represent it in PyTorch?
Keras filters is PyTorch out_features
kernel_size, stride and padding are the same (maybe a few options for padding are called differently)

Do I understand this correctly? If so, what should I do with in_features? My code so far:
class BasicConv2d(nn.Module):
    def __init__(
            self,
            in_channels: int,
            out_channels: int,
            kernel_size: int,
            stride: int
    ) -> None:
        super().__init__()
        self.conv = nn.Conv2d(in_channels,
                              out_channels,
                              kernel_size=kernel_size,
                              stride=stride)
        self.bn = nn.BatchNorm2d(out_channels, eps=0.001)
        self.relu = nn.ReLU()

    def forward(self, x: Tensor) -> Tensor:
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x

class Inception(nn.Module):
    def __init__(
            self,
            in_channels: int,
            num_1x1_filters: int,
            num_3x3_filters: int,
    ) -> None:
        super().__init__()
        # how to fill this further?
        self.conv_1d = BasicConv2d(
            num_1x1_filters,
            )


title: How to transform networkx data into pytorch-geometric Data?

body: How to load networkx list of graphs into pytorch-geometric Data using torch_geometric.utils.convert.from_networkx() ? There are only graph level labels are available.


title: Pytorch WeightedRandomSampler , weights sampling on index data

body:
Pytorch Modelling on datasets:
Hi , I have a dataset and I would like to add weights for more recent datasets to be selected , what I intend to do is to give more emphasis to certain index data in dataset (for example , out of 5000 data instances , I would like to pick the last 1000 data instances which happened recently more often than the rest).
I understand that weightedRandomSampler with DataLoader which allows u to add more weights for data imbalance issues.
However , for my case , I am looking to add emphasis to certain sample indexes , and not by class imbalances.
Anyone able to advise on this? Fully appreciate it. Below is an example to solve class imbalance issue:
Get the class counts and calculate the weights/class by taking its reciprocal.
class_count = [i for i in get_class_distribution(natural_img_dataset).values()]
class_weights = 1./torch.tensor(class_count, dtype=torch.float)
class_weights
###################### OUTPUT ######################
tensor([0.0014, 0.0010, 0.0011, 0.0014, 0.0012, 0.0010, 0.0013, 0.0010])

Assign the weight of each class to all the samples.
class_weights_all = class_weights[target_list]
class_weights_all

###################### OUTPUT ######################
tensor([0.0010, 0.0012, 0.0014,  ..., 0.0010, 0.0014, 0.0010])

Instead of class weights , can we add in weights to specific samples (eg: last 1000 data instances) instead


I was trying to convert my PyTorch model into TensorFlow lite for mobile. My model was pre-trained DenseNet 169 so I did this:-
import sys
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import onnx
from collections import OrderedDict
import tensorflow as tf

from torch.autograd import Variable
from onnx_tf.backend import prepare
dummy_input = Variable(torch.randn(32, 3, 224, 224))
torch.onnx.export(trained_model, dummy_input, "mymodel.onnx")

model = onnx.load("mymodel.onnx")
tf_rep = prepare(model)

print('inputs:', tf_rep.inputs)

# Output nodes from the model
print('outputs:', tf_rep.outputs)

# All nodes in the model
print('tensor_dict:')
print(tf_rep.tensor_dict)

tf_rep.export_graph("mymodel.pb")
converter = tf.lite.TFLiteConverter.from_frozen_gragh("mymodel.pb/saved_model.pb",
tf_rep.inputs, tf_rep.outputs) # **ERROR HERE**

tflite_model = converter.convert()
open("mymodel.tflite", "wb").write(tflite_model)

HERE IS MY ERROR
AttributeError                            Traceback (most recent call last)
<ipython-input-37-0abbde392f91> in <module>()
----> 1 converter = tf.lite.TFLiteConverter.from_frozen_gragh("flowers.pb/saved_model.pb", tf_rep.inputs, tf_rep.outputs)

      2 tflite_model = converter.convert()
      3 open("flowers.tflite", "wb").write(tflite_model)

AttributeError: type object 'TFLiteConverterV2' has no attribute 'from_frozen_gragh'

When I tried with compat.v1 I got the same error but instead of TFLiteConverterV2 I got TFLiteConverter
Thanks, in Advance.
EDIT
So I tried with compat.v1 and fixed the typo in 'from_frozen_gragh' and got this ugly error
---------------------------------------------------------------------------
DecodeError                               Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/lite.py in from_frozen_graph(cls, graph_def_file, input_arrays, output_arrays, input_shapes)
   1804           graph_def = _graph_pb2.GraphDef()
-> 1805           graph_def.ParseFromString(file_content)
   1806         except (_text_format.ParseError, DecodeError):

DecodeError: Error parsing message

During handling of the above exception, another exception occurred:

UnicodeDecodeError                        Traceback (most recent call last)
2 frames
<ipython-input-32-46dac4006b0d> in <module>()
----> 1 tflitconverter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph("flowers.pb/saved_model.pb", tf_rep.inputs, tf_rep.outputs)
      2 e_model = converter.convert()
      3 open("flowers.tflite", "wb").write(tflite_model)

/usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/lite.py in from_frozen_graph(cls, graph_def_file, input_arrays, output_arrays, input_shapes)
   1812                 file_content = six.ensure_binary(file_content, "utf-8")
   1813               else:
-> 1814                 file_content = six.ensure_text(file_content, "utf-8")
   1815             graph_def = _graph_pb2.GraphDef()
   1816             _text_format.Merge(file_content, graph_def)

/usr/local/lib/python3.6/dist-packages/six.py in ensure_text(s, encoding, errors)
    933     """
    934     if isinstance(s, binary_type):
--> 935         return s.decode(encoding, errors)
    936     elif isinstance(s, text_type):
    937         return s

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 3: invalid start byte

PLEASE HELP


title: How to convert TensorFlow tensor to PyTorch tensor without converting to Numpy array?

body:
I want to use a pre-trained Pytorch model in Tensorflow and I need to convert the tensorflow tensors to pytorch tensors. But I don't want to convert the pytorch tensor to a numpy array and convert that to a tensorflow tensor since I'm getting the error of " You must feed a value for placeholder tensor".
I need this conversion when I'm making the graph so the tensorflow tensor doesn't have value and cannot be converted to numpy!
Any solution for that?


title: Implement forward pass of Pixel RNN's RowLSTM pytorch

body:
The question pretty much says it all. I am trying to implement the gate calculations of the forward pass of Pixel RNN's Row LSTM in PyTorch. I am especially curious as to how the first gates of row 1 are computed, since there are no previous cell states or hidden states to use.

I am aware there is a question very similar to this, but the accepted answer had no code, which is what I am looking for.

Furthermore, I have implemented my own code, but it is very slow to train, which is why I am looking to see if someone has a faster solution.

class RLSTM(nn.Module):
    def __init__(self,ch):
        super(RLSTM,self).__init__()
        self.ch=ch
        self.input_to_state = torch.nn.Conv2d(self.ch,4*self.ch,kernel_size=(1,3),padding=(0,1)).cuda()
        self.state_to_state = torch.nn.Conv2d(self.ch,4*self.ch,kernel_size=(1,3),padding=(0,1)).cuda() # error is here: hidPrev is an array - not a valid number of input channel
        self.cell_list = []

        # check if these convolutions are changing their weights

    def forward(self, image):
       # print("starting forward")
       # if(self.ch==64):
      #      print self.input_to_state.weight[0][0][0][0]
        size = image.size()
        b = size[0]
        indvs = list(image.split(1,0))
        tensor_array = []

        for i in range(b):
            tensor_array.append(self.RowLSTM(indvs[i]))
        seq=tuple(tensor_array)
        trans = torch.cat(seq,0)
        global total
        total+=1
      #  print("finished forward")
        return trans.cuda()
    def RowLSTM(self, image):
     # input-to-state (K_is * x_i) : 1x3 convolution. generate h x n x n tensor. hxnxn tensor contains all i -> s info
       # print("Starting LSTM")
        self.cell_list=[]
        igates = []
       # print(image.size())
        n = image.size()[2]
        ch=image.size()[1]
        for i in range(n):
            if i==0:
                # COULD BE THIS AREA, SHOULD EVERYTHING BE 0?
                isgates = self.splitIS(self.input_to_state(image)) # convolve, then split into gates (4 per row)
                #print("{} and {}".format(len(isgates),len(isgates[0])))
                #cell=RowLSTMCell(0,torch.zeros(ch,n,1).cuda(),torch.zeros(ch,n,1).cuda(),torch.zeros(ch,n,1).cuda(),torch.zeros(ch,n,1).cuda(),torch.zeros(ch,n,1).cuda(),torch.zeros(ch,n,1).cuda())
                cell=RowLSTMCell(0,isgates[0][0],isgates[0][1],isgates[0][2],isgates[0][3],torch.zeros(ch,n,1).cuda(),torch.zeros(ch,n,1).cuda())
                cell.c=isgates[0][0]*isgates[0][3]
                cell.h=torch.tanh(cell.c)*isgates[0][1]
                # now have dummy variables for first row
                self.cell_list.append(cell)
            else:
                cell_prev = self.cell_list[i-1]
                hid_prev = cell_prev.getHiddenState()
                ssgates = self.splitSS(self.state_to_state(hid_prev.unsqueeze(0)))
                gates = self.addGates(isgates, ssgates,i)
                ig, og, fg, gg = gates[0], gates[1], gates[2], gates[3]
                cell = RowLSTMCell(cell_prev, ig, og, fg, gg, 0 ,0) #MORE zeros
                cell.compute()

                self.cell_list.append(cell)

        # now have a list of all cell data, concatenate hidden state into 1 x h x n x n
        hidden_layers = []
        for i in range(n):
            hid = self.cell_list[i].h
            hidden_layers.append(torch.unsqueeze(hid,0))

        seq = tuple(hidden_layers)
        tensor = torch.cat(seq,3)
      #  print("finished lstm")
        #print(tensor.size())
        return tensor

    def splitIS(self, tensor): #always going to be splitting into 4 pieces, so no need to add extra parameters
        inputStateGates={}
        size=tensor.size() # 1 x 4h x n x n
        out_ft=size[1] # get 4h for the nxnx4h tensor
        num=size[2] # get n for the nxn image
        hh=out_ft/4 # we want to split the tensor into 4, for the gates
        tensor = torch.squeeze(tensor).cuda() # 4h x n x n

        # First, split by row: Creates n tensors of 4h x n x 1
        rows = list(tensor.split(1,2))

        for i in range(num):
            # Each row is a tensor of 4h x n x 1, split it into 4 of h x n x 1
            row=rows[i]
          #  print("Each row using cuda: "+str(row.is_cuda))
            inputStateGates[i]=list(row.split(hh,0))

        return inputStateGates

    def splitSS(self, tensor): # 1 x 4h x n x 1, create 4 of 1 x h x n x 1
        size=tensor.size()
        out_ft=size[1] # get 4h for the 1x4hxn tensor
        num=size[2] # get n for the 1xhxn row
        hh=out_ft/4 # we want to split the tensor into 4, for the gates
        tensor = tensor.squeeze(0).cuda() # 4h x n x 1
        splitted=list(tensor.split(hh,0))
        return splitted

    def addGates(self, i2s,s2s,key):
        """ these dictionaries are of form {key : [[i], [o], [f], [g]]}
            we want to add pairwise elemeents """

        # i2s is of form key: [[i], [o], [f], [g]] where each gate is hxn
        # s2s is of form [[h,n],[h,n],[h,n], [h,n]]
        gateSum = []
        for i in range(4): # always of length 4, representing the gates
            gateSum.append(torch.sigmoid(i2s[key][i] + s2s[i]))
        return gateSum

And here is each cell's class:

class RowLSTMCell(): #inherit torch.nn.LSTM?
    def __init__(self,prev_row, i, o, f, g, c, h):
        self.c=c
        self.h=h
        self.i=i
        self.i = self.i.cuda()
        self.o=o
        self.o = self.o.cuda()
        self.g=g
        self.g = self.g.cuda()
        self.f=f
        self.f = self.f.cuda()
        self.prev_row=prev_row
    def getStateSize(self):
        return self._state_size

    def getOutputSize(self):
        return self._output_size

    def compute(self):
        c_prev = self.prev_row.getCellState()
        h_prev = self.prev_row.getHiddenState()
        self.c = self.f * c_prev + self.i * self.g
        self.h = torch.tanh(self.c) * self.o
    def getHiddenState(self):
        return self.h

    def getCellState(self):
        return self.c


title: PPO / TRPO Implementation

body:
So, I recently watched this video on PPO and want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.

Given algorithm in the paper:

says to run pi_theta_old, compute advantage estimates and optimize the objective, but how can we calculate pi_theta for the objective since we have not updated the policy_old yet?

EDIT : To all those looking for an clear answer, I have posted the same question on Stack Exchange Artificial Intelligence (https://ai.stackexchange.com/questions/8212/ppo-trpo-implementation).


title: Numpy/Pytorch dtype conversion / compatibility

body:
I'm trying to find some documentation understand how dtypes are combined. For example:

x : np.int32 = ...
y : np.float64 = ...

What is going to be the type of x + y ?
does it depend on the operator (here +) ?
does it depend where it is stored (z = x + y vs z[...] = x + y) ?

I'm looking for part of the documentation that describe these kind of scenario but so far I'm empty-handed.


title: Use PyTorch to adjust Tensor matrix values based on numbers I calculate from the Tensors?

body:
I have two tensors (matrices) that I've initialized:

sm=Var(torch.randn(20,1),requires_grad=True)
sm = torch.mm(sm,sm.t())
freq_m=Var(torch.randn(12,20),requires_grad=True)

I am creating two lists from the data inside these 2 matrices, and I am using spearmanr to get a correlation value between these 2 lists. How I am creating the lists is not important, but the goal is to adjust the values inside the matrices so that the calculated correlation value is as close to 1 as possible.

If I were to solve this problem manually, I would tweak values in the matrices by .01 (or some small number) each time and recalculate the lists and correlation score. If the new correlation value is higher than the previous one, I would save the 2 matrices and tweak a different value until I get the 2 matrices that give me the highest correlation score possible.

Is PyTorch capable of doing this automatically? I know PyTorch can adjust based on an equation but the way I want to adjust the matrix values is not against an equation, it's against a correlation value that I calculate. Any guidance with this is greatly appreciated!


title: Fastest way to retrieve word vectors of a sequence and fed into model?

body:
For training, I have to feed the model sequence of word vector. Each sequence has on average 40 words. So, if I use a dictionary of pre-trained word embedding (like Glove), For each sequence have to hit the embedding dictionary around 40 times and for each batch, it will be around batch_size*40 times. The dataset is divided into many batches and the whole dataset has to be iterate (epoch) several times also. So, you can imagine how many times the dictionary will get hit.
This is the approach I have done already and it is taking really a lot of time.

To solve this, I tried to make a dictionary of sequence to vector. This dictionary should contain a sequence as key and a 2d python list (each row is a word vector) as a value of the key. The hope is to, I just have to look for the sequence and get the values. This should decrease the time a lot but the dictionary would be very big (I estimated the size by saved the data (sequence->vectors) in mongodb and exported it and the file is 23gb). A dictionary of size 23gb should not be problem because my I am using shared server where I can allocate as much as 100gb memory. But the program gets killed while loading the dictionary. So this is not working.

Another approach I am thinking about is to copy the word embedding vector into pytorch's nn.Embedding().

input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
embedding(input)`

Here the numbers are indices of the word. Regarding this approach,pytorch embedding uses numpy matrix as lookup table. So, my concern is, isn’t for executing the previous code, there will be 7 hits on the numpy matrix? Or it will be retrieved parallelly? Even it runs parallelly, there should be another dictionary to convert word to indices. That also needs 7 hits on the word2indices dictionary.

So, what do you think, what is the fastest and efficient way to retrieve word vectors of a sequence and fed into model?


title: How do I clear GPU memory Usage?

body:
I iterate Grad-CAM code per an image.
In my opinion, CUDA code repeatedly stack GPU memory.
So, I used torch.cuda.empty_cache(), but there is the same problem.
How do I clear the usage of GPU??
전용메모리 사용량 = Dedicated GPU memory usage
enter image description here
def demo1(input_directory, target_layer, cnn_model, topk, output_dir, cuda):

device = get_device(cuda)

# Model from torchvision
model = torch.load(cnn_model)
model.to(device)
model.eval()

images = glob.glob(input_directory + '/' + '*.png')

for image_paths in images:
    print(image_paths)
    a = [image_paths,]
    b= tuple(a)
    image_paths = b
    images, raw_images, class_id = load_images(image_paths)
    images = torch.stack(images).to(device)

    for i in range(topk):
        # Guided Backpropagation
        gbp.backward(ids=ids[:, [i]])
        gradients = gbp.generate()

        # Grad-CAM
        gcam.backward(ids=ids[:, [i]])
        regions = gcam.generate(target_layer=target_layer)

        for j in range(1):

            print("\t#{}: {} ({:.5f})".format(j, classes[ids[j, i]], probs[j, i]))

            class_id = class_id.split("\\")[0]
            # Grad-CAM
            if class_id == classes[ids[j, i]]:
                if probs[j, i] >= 0.3:
                    save_gradcam(
                        filename=osp.join(
                            output_dir,
                            "{}-gradcam-{}.png".format(
                                f_name, classes[ids[j, i]]
                            ),
                        ),
                        gcam=regions[j, 0],
                        raw_image=raw_images[j],
                    )

                    # Guided Grad-CAM
                    save_gradient(
                        filename=osp.join(
                            output_dir,
                            "{}-{}-guided_gradcam-{}-{}.png".format(
                                j, cnn_model, target_layer, classes[ids[j, i]]
                            ),
                        ),
                        gradient=torch.mul(regions, gradients)[j],
                    )
            gc.collect()
            torch.cuda.empty_cache()


title: Difference between these implementations of LSTM Autoencoder?

body:
Specifically what spurred this question is the return_sequence argument of TensorFlow's version of an LSTM layer.
The docs say:

Boolean. Whether to return the last output. in the output sequence,
or the full sequence. Default: False.

I've seen some implementations, especially autoencoders that use this argument to strip everything but the last element in the output sequence as the output of the 'encoder' half of the autoencoder.
Below are three different implementations. I'd like to understand the reasons behind the differences, as the seem like very large differences but all call themselves the same thing.
Example 1 (TensorFlow):
This implementation strips away all outputs of the LSTM except the last element of the sequence, and then repeats that element some number of times to reconstruct the sequence:
model = Sequential()
model.add(LSTM(100, activation='relu', input_shape=(n_in,1)))
# Decoder below
model.add(RepeatVector(n_out))
model.add(LSTM(100, activation='relu', return_sequences=True))
model.add(TimeDistributed(Dense(1)))

When looking at implementations of autoencoders in PyTorch, I don't see authors doing this. Instead they use the entire output of the LSTM for the encoder (sometimes followed by a dense layer and sometimes not).
Example 1 (PyTorch):
This implementation trains an embedding BEFORE an LSTM layer is applied... It seems to almost defeat the idea of an LSTM based auto-encoder... The sequence is already encoded by the time it hits the LSTM layer.
class EncoderLSTM(nn.Module):
  def __init__(self, input_size, hidden_size, n_layers=1, drop_prob=0):
    super(EncoderLSTM, self).__init__()
    self.hidden_size = hidden_size
    self.n_layers = n_layers

    self.embedding = nn.Embedding(input_size, hidden_size)
    self.lstm = nn.LSTM(hidden_size, hidden_size, n_layers, dropout=drop_prob, batch_first=True)

  def forward(self, inputs, hidden):
    # Embed input words
    embedded = self.embedding(inputs)
    # Pass the embedded word vectors into LSTM and return all outputs
    output, hidden = self.lstm(embedded, hidden)
    return output, hidden

Example 2 (PyTorch):
This example encoder first expands the input with one LSTM layer, then does its compression via a second LSTM layer with a smaller number of hidden nodes. Besides the expansion, this seems in line with this paper I found: https://arxiv.org/pdf/1607.00148.pdf
However, in this implementation's decoder, there is no final dense layer. The decoding happens through a second lstm layer that expands the encoding back to the same dimension as the original input. See it here. This is not in line with the paper (although I don't know if the paper is authoritative or not).
class Encoder(nn.Module):
  def __init__(self, seq_len, n_features, embedding_dim=64):
    super(Encoder, self).__init__()
    self.seq_len, self.n_features = seq_len, n_features
    self.embedding_dim, self.hidden_dim = embedding_dim, 2 * embedding_dim
    self.rnn1 = nn.LSTM(
      input_size=n_features,
      hidden_size=self.hidden_dim,
      num_layers=1,
      batch_first=True
    )
    self.rnn2 = nn.LSTM(
      input_size=self.hidden_dim,
      hidden_size=embedding_dim,
      num_layers=1,
      batch_first=True
    )
  def forward(self, x):
    x = x.reshape((1, self.seq_len, self.n_features))
    x, (_, _) = self.rnn1(x)
    x, (hidden_n, _) = self.rnn2(x)
    return hidden_n.reshape((self.n_features, self.embedding_dim))

Question:
I'm wondering about this discrepancy in implementations. The difference seems quite large. Are all of these valid ways to accomplish the same thing? Or are some of these mis-guided attempts at a "real" LSTM autoencoder?


title: Final step of PyTorch Gradient Accumulation for small datasets

body:
I am training a BERT model on a relatively small dataset and cannot afford to lose any labelled sample as they must all be used for training. Due to GPU memory constraints, I am using gradient accumulation to train on larger batches (e.g. 32). According to PyTorch documentation, gradient accumulation is implemented as follows:
scaler = GradScaler()

for epoch in epochs:
    for i, (input, target) in enumerate(data):
        with autocast():
            output = model(input)
            loss = loss_fn(output, target)
            loss = loss / iters_to_accumulate

        # Accumulates scaled gradients.
        scaler.scale(loss).backward()

        if (i + 1) % iters_to_accumulate == 0:
            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)

            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad()

However, if you are using e.g. 110 training samples, with batch size 8 and accumulation step 4 (i.e. effective batch size 32), this method would only train the first 96 samples (i.e. 32 x 3), i.e. wasting 14 samples. In order to avoid this, I'd like to modify the code as follows (notice change to the final if statement):
scaler = GradScaler()

for epoch in epochs:
    for i, (input, target) in enumerate(data):
        with autocast():
            output = model(input)
            loss = loss_fn(output, target)
            loss = loss / iters_to_accumulate

        # Accumulates scaled gradients.
        scaler.scale(loss).backward()

        if (i + 1) % iters_to_accumulate == 0 or (i + 1) == len(data):
            # may unscale_ here if desired (e.g., to allow clipping unscaled gradients)

            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad()

Is this correct and really that simple, or will this have any side effects? It seems very simple to me, but I've never seen it done before. Any help appreciated!


title: Parallel aplication of the function on pytorch cuda tensor strings

body: I need to apply user defined function to all strings of cuda tensor. Because of that function takes very a few of GPU resources but significant time I want to speed up the calculation applying that function to the several strings contemporaneously. The calculation for different string could take significantly different time. What is the most efficient way to do that?


title: Watson generated Pytorch results in: "ValueError: optimizer got an empty parameter list"

body:
I am experimenting Watson Neural Network Modeler.

I've create a model from the built in demo "Single Convolution layer on MNIST". The only customization I did was to specify the training data files.

I then exported the Pytorch code and I am trying to run in on my local computer.

The generated code is pretty readable. The relevant code excerpt is:

# Define network architecture
class Net(nn.Module):
    def __init__(self, inp_c):
        super(Net, self).__init__()

    def forward(self, ImageData_4, target):

        Convolution2D_9 = self.Convolution2D_9(ImageData_4)
        ReLU_1 = self.ReLU_1(Convolution2D_9)
        Pooling2D_8 = self.Pooling2D_8(ReLU_1)
        Flatten_2 = Pooling2D_8.view(-1, 10816)
        Dense_3 = self.Dense_3(Flatten_2)
        Softmax_5 = self.Softmax_5(Dense_3)
        Accuracy_6 = torch.topk(Softmax_5, 1)[0]
        CrossEntropyLoss_7 = self.CrossEntropyLoss_7(Softmax_5, target)
        return Softmax_5, Accuracy_6

# Model Initialization
inp_c = 1
model = Net(inp_c)
model.cuda()

# Define optimizer
learning_rate = 0.001000
decay = 0.000000
beta_1 = 0.900000
beta_2 = 0.999000
optim = optim.Adam(
    model.parameters(),
    lr=learning_rate,
    betas=(beta_1, beta_2),
    weight_decay=decay)

I am getting the error:

"ValueError: optimizer got an empty parameter list"

on the optim = optim.Adam() statement.

Is there any Watson user/expert over there to bring some light on this issue? I am basically running the demo. It was not supposed to fail.

Thanks!


title: Pyro change AutodiagonalNormal settings

body:
I use pyro-ppl 3.0 for probabilistic programming. When I go through the tutorial on Bayesian regression. I used AutoGuide and pyro.random_module to transfer a normal feed-forward network to bayesian network.

# linear regression
class RegressionModel(nn.Module):
    def __init__(self, p):
        # p number of feature
        super(RegressionModel, self).__init__()
        self.linear1 = nn.Linear(p, 2)
        self.linear2 = nn.Linear(2, 1)
        self.softplus = nn.Softplus()

    def forward(self, x):
        x = self.softplus(self.linear1(x))
        return self.linear2(x)

# model
def model(x_data, y_data):
    # weight and bias prior
    w1_prior = Normal(torch.zeros(2,2), torch.ones(2,2)).to_event(2)
    b1_prior = Normal(torch.ones(2)*8, torch.ones(2)*1000).to_event(1)
    w2_prior = Normal(torch.zeros(1,2), torch.ones(1,2)).to_event(1)
    b2_prior = Normal(torch.ones(1)*3, torch.ones(1)*500).to_event(1)

    priors = {'linear1.weight': w1_prior, 'linear1.bias': b1_prior,
             'linear2.weight': w2_prior, 'linear2.bias': b2_prior}

    scale = pyro.sample("sigma", Uniform(0., 10.))

    # lift module parameters to random variables sampled from the priors
    lifted_module = pyro.random_module("module", regression_model, priors)
    # sample a nn (which also samples w and b)
    lifted_reg_model = lifted_module()
    with pyro.plate("map", len(x_data)):
        # run the nn forward on data
        prediction_mean = lifted_reg_model(x_data).squeeze(-1)
        # condition on the observed data
        pyro.sample("obs",
                    Normal(prediction_mean, scale),
                    obs=y_data)
        return prediction_mean

guide = AutoDiagonalNormal(model)

#================

#================

# inference
optim = Adam({"lr": 0.03})
svi = SVI(model, guide, optim, loss=Trace_ELBO(), num_samples=1000)

def train():
    pyro.clear_param_store()
    for j in range(num_iterations):
        # calculate the loss and take a gradient step
        loss = svi.step(x_data, y_data)
        if j % 100 == 0:
            print("[iteration %04d] loss: %.4f" % (j + 1, loss / len(data)))

train()

for name, value in pyro.get_param_store().items():
    print(name, pyro.param(name))

the results are showing as following:
auto_loc tensor([-2.1585, -0.9799, -0.0378, -0.5000, -1.0241,  2.6091, -1.3760,  1.6920, 0.2553,  4.5768], requires_grad=True)
auto_scale tensor([0.1432, 0.1017, 0.0368, 0.7588, 0.4160, 0.0624, 0.6657, 0.0431, 0.2972, 0.0901], grad_fn=)

the number of latent variable is automatically set to be 10. I want to change the number. as mentioned in the tutorial, I add

##
latent_dim = 5
pyro.param("auto_loc", torch.randn(latent_dim))
pyro.param("auto_scale", torch.ones(latent_dim),
           constraint=constraints.positive)

between #============= mentioned above.

but the result still the same. the number doesn't change. So, How to set up the AutoDiagnalNormal function to change the number of latent variables


title: pytorch RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument

body:
I am implementing categorical dqn for reinforcement learning and I am getting the following error:

  proj_dist.view(-1).index_add_(0, (l + offset).view(-1), (next_dist *
  (u.float() - b)).view(-1))

  RuntimeError: Expected object of type torch.cuda.LongTensor but found
  type torch.LongTensor for argument #3 'other'

I have already tried modifying the third argument of the first call to index_add_ as:

.long().cuda()
long().to(device)
...

nothing works

Here is the code, where the parameters to the function are all tensors. Note I am operating on a batch of 64, so next_states, for instance, will be some 64 tensors:

def projection_distribution(self, next_states, rewards, dones):
        batch_size  = next_states.size(0)

        delta_z = float(self.V_max - self.V_min) / (self.num_atoms - 1)
        support = torch.linspace(self.V_min, self.V_max, self.num_atoms)

        # torch.Size([64, 4, 51])
        next_dist   = self.qnetwork_target(next_states).data.cpu() *
                       support.data.cpu()
        # torch.Size([64])
        next_actions = next_dist.sum(2).max(1)[1]
        # torch.Size([64, 1, 51])
        next_actions =
        next_actions.unsqueeze(1).unsqueeze(1).expand(next_dist.size(0), 1,
        next_dist.size(2))
        # these are the batch_size distributions relative to the optimal
        actions 'next_actions'
        # torch.Size([64, 51])
        next_dist   = next_dist.gather(1, next_actions).squeeze(1)

        rewards = rewards.expand_as(next_dist)
        print('rewards: ', rewards.size())
        dones   = dones.expand_as(next_dist)
        support = support.unsqueeze(0).expand_as(next_dist)
        print('support: ', support.size())

        Tz = rewards + (1 - dones) * GAMMA * support.to(device)
        Tz = Tz.clamp(min=self.V_min, max=self.V_max)
        b  = (Tz - self.V_min) / delta_z
        l  = b.floor().long()
        u  = b.ceil().long()
        print('TYPE: ' , b.type(), l.type(), u.type(), next_dist.type())

        offset = torch.linspace(0, (batch_size - 1) * self.num_atoms, batch_size).long()\
                        .unsqueeze(1).expand(batch_size, self.num_atoms)
        print('offset: ', offset.size())

        proj_dist = torch.zeros(next_dist.size())
        print('proj_dist_1: ', proj_dist.size())
        proj_dist.view(-1).index_add_(0, (l + offset).view(-1), (next_dist * (u.float() - b)).view(-1))
        print('proj_dist_2: ', proj_dist.size())
        proj_dist.view(-1).index_add_(0, (u + offset).view(-1), (next_dist * (b - l.float())).view(-1))
        print('proj_dist_3: ', proj_dist.size())

        return proj_dist

Why do I get that error? How do I fix it?

Many Thanks
Max


title: Python Script is running itself twice

body:
I am learning PyTorch and working my way through the tutorials. I find that when I attempt to use torchvision my script is running twice. I wonder if it is configured wrong or I accidentally introduced some sort of parallelism.

Here is my code:

# show images
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
print('Type of trainloader')
print(type(trainloader))

dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

My output is:

C:\Users\Avi\Anaconda3\python.exe C:/Users/Avi/PycharmProjects/ExampleNN/test.py
Files already downloaded and verified
Files already downloaded and verified
Type of trainloader
<class 'torch.utils.data.dataloader.DataLoader'>
Files already downloaded and verified
Files already downloaded and verified
Type of trainloader
<class 'torch.utils.data.dataloader.DataLoader'>

And then there is a crash on dataiter = iter(trainloader) the second time around, presumably because the iteration is finished already.

Can anyone tell me what I am doing wrong?

UPDATE:

I've discovered that if I set num_workers=0 in the DataLoader constructor it only runs once and has no crash. If I set it o 1, it runs twice and crashes just like when I set it to 2. I still don't understand why this is happening, but at least I can work around my issue.


title: How to solve the problem of PyTorch stack?

body:
I want to run Python program using PyTorch. How should I make each tensor in batch equal? Because the following problem appears:

  Traceback (most recent call last):
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 311, in <module>
    fire.Fire(demo)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 468, in _Fire
    target=component.__name__)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 289, in demo
    n_epochs=n_epochs, batch_size=batch_size, seed=seed)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 168, in train
    n_epochs=n_epochs,
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 42, in train_epoch
    for batch_idx, (input, target) in enumerate(loader):
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\dataloader.py", line 346, in __next__
    data = self._next_data()
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\dataloader.py", line 386, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\collate.py", line 87, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\collate.py", line 87, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\collate.py", line 72, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\collate.py", line 63, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [650] at entry 0 and [108] at entry 1


title: Problem with padding sequences in Keras from PyTorch

body:
I tried to convert the PyTorch code for padding a batch to Keras. It is a code for multi-modal sentiment analysis on CMU-MOSI datase. I basically need to convert the whole of the original code https://github.com/Justin1904/CMU-MultimodalSDK-Tutorials to Keras and as I am new to this it is a struggle for me. Would really appreciate some help and guidance. Immediately, I need all the feature vectors padded so they are all in the same length as one another: visual, acoustic, sentences and labels.

Here they have done it batch wise, doing it on the entire dataset or dataset splits would work as well. Need all the help I can get.

*Mine(Keras)*

    def multi_collate(batch):
        '''
        Collate functions assume batch = [Dataset[i] for i in index_set]
        '''
        # for later use we sort the batch in descending order of length
        batch = sorted(batch, key=lambda x: x[0][0].shape[0], reverse=True)

        # get the data out of the batch - use pad sequence util functions from PyTorch to pad things
        labels = np.array([sample[1] for sample in batch])
        sentences = tf.keras.preprocessing.sequence.pad_sequences([sample[0][0] for sample in batch], padding=PAD)
        visual = tf.keras.preprocessing.sequence.pad_sequences([sample[0][1] for sample in batch])
        acoustic = tf.keras.preprocessing.sequence.pad_sequences([sample[0][2] for sample in batch])

        # lengths are useful later in using RNNs
        lengths = np.array([sample[0][0].shape[0] for sample in batch])
        return sentences, visual, acoustic, labels, lengths

*Original (PyTorch)*

    def multi_collate(batch):
        '''
        Collate functions assume batch = [Dataset[i] for i in index_set]
        '''
        # for later use we sort the batch in descending order of length
        batch = sorted(batch, key=lambda x: x[0][0].shape[0], reverse=True)

        # get the data out of the batch - use pad sequence util functions from PyTorch to pad things
        labels = torch.cat([torch.from_numpy(sample[1]) for sample in batch], dim=0)
        sentences = pad_sequence([torch.LongTensor(sample[0][0]) for sample in batch], padding_value=PAD)
        visual = pad_sequence([torch.FloatTensor(sample[0][1]) for sample in batch])
        acoustic = pad_sequence([torch.FloatTensor(sample[0][2]) for sample in batch])

        # lengths are useful later in using RNNs
        lengths = torch.LongTensor([sample[0][0].shape[0] for sample in batch])
        return sentences, visual, acoustic, labels, lengths


title: How can I put individual Tensors into a batch and process them in PyTorch?

body:
My network processes each Tensor 1 by 1, so it will generate an input and output pair, feed the input into the network, get an output for this single tensor pair and compare it to the output.
An example of a Tensor pair would be tensor as an input of: [10, 1] and an output of [12, 1] etc. This is without padding. Here is the general process of the tensor pairs being generated:
Input Tensor:  torch.Size([6, 1])
Target Tensor:  torch.Size([7, 1])

Input Tensor:  torch.Size([8, 1])
Target Tensor:  torch.Size([10, 1])

Input Tensor:  torch.Size([8, 1])
Target Tensor:  torch.Size([10, 1])

A deeper look into the tensor being generated would be as such:
Input Tensor:
tensor([[22],
        [ 7],
        [18],
        [ 5],
        [ 1]])

Target Tensor:
tensor([[23],
        [ 8],
        [ 6],
        [19],
        [ 6],
        [ 5],
        [ 1]])

What I would like to do now is implement Batching into this process. So instead of generating a single pair and then doing a feed forward on this pair, I would like to generate a large Batch of pairs (32, 64...) and feed this in instead.
How would I go about creating the input batch and output batch? I believe it should look something like [40, 32] or [40, 64]
Here is my program currently.
def trainIters(encoder, decoder, n_iters, print_every=1000, plot_every=100, learning_rate=0.001, batch_size=64):

    training_pairs = [tensorsFromPair(random.choice(pairs))
                      for i in range(n_iters)]
    criterion = nn.CrossEntropyLoss()#nn.NLLLoss()

    for iter in range(1, n_iters + 1):

        training_pair = training_pairs[iter - 1]
        input_tensor = training_pair[0]
        target_tensor = training_pair[1]

        print("Input Tensor: ", input_tensor.shape)
        print("Target Tensor: ",target_tensor.shape)

        loss = train(input_tensor, target_tensor, encoder,
                     decoder, encoder_optimizer, decoder_optimizer, criterion)

def train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length=MAX_LENGTH):
    encoder_hidden = encoder.initHidden()

    encoder_optimizer.zero_grad()
    decoder_optimizer.zero_grad()

    input_length = input_tensor.size(0)
    target_length = target_tensor.size(0)

    encoder_outputs = torch.zeros(max_length, encoder.hidden_size, device=device)

    loss = 0
    for ei in range(input_length):

        encoder_output, encoder_hidden = encoder(
            input_tensor[ei], encoder_hidden)
        encoder_outputs[ei] = encoder_output[0, 0]

    decoder_input = torch.tensor([[SOS_token]], device=device)

    decoder_hidden = encoder_hidden

    use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False

    if use_teacher_forcing:
        # Teacher forcing: Feed the target as the next input
        for di in range(target_length):
            decoder_output, decoder_hidden, decoder_attention = decoder(
                decoder_input, decoder_hidden, encoder_outputs)
            #print(decoder_output, target_tensor[di])
            loss += criterion(decoder_output, target_tensor[di])
            decoder_input = target_tensor[di]  # Teacher forcing

    else:
        # Without teacher forcing: use its own predictions as the next input
        for di in range(target_length):
            decoder_output, decoder_hidden, decoder_attention = decoder(
                decoder_input, decoder_hidden, encoder_outputs)
            topv, topi = decoder_output.topk(1)
            decoder_input = topi.squeeze().detach()  # detach from history as input

            loss += criterion(decoder_output, target_tensor[di])
            if decoder_input.item() == EOS_token:
                break

    loss.backward()

    encoder_optimizer.step()
    decoder_optimizer.step()

    return loss.item()


title: Predict only one class (person) in YOLACT/YOLACT++

body:
I want to predict only one class i.e. person from all the 84 classes that are being checked for and predicted.
For YOLACT reference https://github.com/dbolya/yolact
The results are pretty fine but I guess I just need to modify one of the codes and in a very short way but I cant manage to find out
There is one issue related to this in which I did what he mentioned like adding the 4 lines in Yolact/layers/output_utils.py and changing nothing else. Those lines are as following:
boxes = torch.cat((boxes[classes==0], boxes[classes==2]),dim=0)
scores = torch.cat((scores[classes==0], scores[classes==2]),dim=0)
masks = torch.cat((masks[classes==0], masks[classes==2]),dim=0)
classes = torch.cat((classes[classes==0], classes[classes==2]),dim=0)

But it gives the following error:
RuntimeError: strides[cur - 1] == sizes[cur] * strides[cur] INTERNAL ASSERT FAILED at
/opt/conda/conda-bld/pytorch_1573049310284/work/torch/csrc/jit/fuser/executor.cpp:175,
please report a bug to PyTorch.
The above operation failed in interpreter, with the following stack trace:

terminate called without an active exception
Aborted (core dumped)

I tried adding the if condition as mentioned but still it gives error. I am using pytorch 1.3


title: My LSTM model does not learn , the weights do not get update

body:
My LSTM model in pytorch does not learn and it does not get any update while training .... after each epoch I print the sum of weights for different each layer but still it does not get any update ...
The y array is (n,3) , first column is keeping the actual size , second is the label (0 or 1) and the last one is the weight for penalizing loss function.
Apparently the optimizer.step() does not work and does not apply gradients to weights. On a separate note ; I have tried the model with different learning rates and mini batches and no different result. The result is shown is from a dummy variables randomly generated and y label is an imbalanced dataset ~ 3% ! I have tried different weights to overcome the imbalanced ration but I guess something is wrong with the model.
Also, if I run the model with my own dataset , the gradients for the lstm layer (all four params ) would be just zero ! but in the linear layer there are gradients .
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence

class LSTMClassifier(nn.Module):
    """
    This is the simple RNN model we will be using to perform Sentiment Analysis.
    """

    def __init__(self, feature_size, hidden_dim , layer_dim = 1):
        """
        Initialize the model by settingg up the various layers.
        """
        super(LSTMClassifier, self).__init__()

        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(feature_size, hidden_dim , layer_dim,  batch_first = True)
        self.dense = nn.Linear(in_features=hidden_dim, out_features=1)
        self.sig = nn.Sigmoid()

    def init_hidden(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim)
        return [t for t in (h0, c0)]

    def forward(self, x , y):
        #import pdb; pdb.set_trace()
        """
        Perform a forward pass of our model on some input.
        """
        #h0, c0 = self.init_hidden(x)
        x_seq = y[:,0]
        x = pack_padded_sequence(x, x_seq, batch_first=True , enforce_sorted = False)
        lstm_out, _ = self.lstm(x)
        lstm_out, _ = torch.nn.utils.rnn.pad_packed_sequence(lstm_out, batch_first=True)
        lstm_out = lstm_out.contiguous()
        out = self.dense(lstm_out)[:,-1,:]
        #out = out[range(len(x_seq)), (x_seq - 1)]
        return self.sig(out.squeeze())

def _get_train_data_loader(batch_size, X , y):
    print("Get train/test data loader.")

    train_y = torch.from_numpy(y).long()
    try :
        train_X = torch.from_numpy(X).float()
    except :
        train_X = X

    train_ds = torch.utils.data.TensorDataset(train_X, train_y)

    return torch.utils.data.DataLoader(train_ds, batch_size=batch_size , shuffle=True)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print ('train dist' , sum(y_train[:,1])/y_train.shape[0] , '\n ****\n' ,
      'test dist' , sum(y_test[:,1])/y_test.shape[0])
optimizer = optim.SGD(model.parameters() , lr=.01 )
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

train_loader = _get_train_data_loader(batch_size = 128, X = X_ , y = y_weighted)
test_loader = _get_train_data_loader(batch_size = 512, X = X_test , y = y_test)

epochs = 2
model = LSTMClassifier (87 , 5)

loss_fn = torch.nn.BCELoss(reduction='none')
train_loss = []
test_loss = []
for epoch in range(1, epochs + 1):
    print ('epoch = ' , epoch)
    model.train()
    total_loss = 0
    for batch in train_loader:
        batch_X, batch_y = batch

        #batch_X = batch_X.to(device)
        #batch_y = batch_y.to(device)

        # TODO: Complete this train method to train the model provided.
        optimizer.zero_grad()

         # Forward pass
        outputs = model(batch_X , batch_y)
        y_ = batch_y[:,1].float()
        loss = loss_fn(outputs, y_)
        weight=batch_y[:,2].float()
        loss = (loss * weight).mean()

        # Backward and optimize
        loss.backward()
        optimizer.step()

        total_loss += loss.data.item()

    for p in model.parameters():
           print(torch.sum(p.grad))

    print ('total loss' , total_loss)
    print ('lstm weight' , torch.sum(model.lstm.weight_hh_l0.data) , 'dense_weight' , torch.sum(model.dense.weight.data))

    train_loss.append(total_loss)
    with torch.no_grad():
        n_correct = 0
        n_samples = 0
        for test, labels in test_loader:
            #labels = labels.to(device)
            outputs = model(test , labels)
            # max returns (value ,index)
            predicted = torch.round(outputs)
            n_samples += labels.size(0)
            n_correct += (predicted == labels[:,1]).sum().item()

        acc = 100.0 * n_correct / n_samples
        print(f'Accuracy of the network on the 10000 test images: {acc} %')
        test_loss.append(acc)

******************************************
epoch =  1
tensor(6518.8760)
tensor(-236.9392)
tensor(149.6967)
tensor(149.6966)
tensor(-1551.1709)
tensor(5021.9199)
total loss 1871.447255373001
lstm weight tensor(3.0054) dense_weight tensor(-0.5395)
Accuracy of the network on the 10000 test images: 96.92037099752012 %
epoch =  2
tensor(7822.5503)
tensor(-284.3271)
tensor(179.6338)
tensor(179.6338)
tensor(-1861.4037)
tensor(6026.2920)
total loss 1871.465574145317
lstm weight tensor(3.0054) dense_weight tensor(-0.5395)
Accuracy of the network on the 10000 test images: 96.92037099752012 %
***********************************************

Another Question is how to use weight in BCEloss(weight ) while we load data with data loader ? The only way left is instantiating a loss_fn within each loop ?
Any help is much appreciated !


title: Exporting a torchscript model

body:
I'm trying to convert a torchscript model to ONNX format. I'm using Pytorch 1.8.0. However I'm getting the errors when I try to run the following code.
device = torch.device('cpu')

trained_model.eval()
if torch.cuda.is_available():
    trained_model.load_state_dict(torch.load('model.pt'))
else:
    trained_model.load_state_dict(torch.load('model.pt'),map_location=device)

dummy_input = Variable(torch.randn(1, 1, 1, 1, 1))

trained_model.eval()
torch.onnx.export(trained_model, dummy_input, "model.onnx",input_names = ['input'], output_names = ['output'])

The error is:
/usr/local/lib/python3.6/dist-packages/torch/serialization.py:589: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)
  " silence this warning)", UserWarning)
Traceback (most recent call last):
  File "script.py", line 131, in <module>
    trained_model.load_state_dict(torch.load('model.pt',map_location=device))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 591, in load
    return torch.jit.load(opened_file)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_serialization.py", line 164, in load
    cu, f.read(), map_location, _extra_files
RuntimeError: Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend.This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty_strided' is only available for these backends: [CPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradNestedTensor, UNKNOWN_TENSOR_TYPE_ID, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

When I tried to replace torch.load with torch.jit.load, I'm getting the following error:
  File "script.py", line 131, in <module>
    trained_model.load_state_dict(torch.jit.load('model.pt',map_location=device))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1196, in load_state_dict
    state_dict = state_dict.copy()
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_script.py", line 561, in __getattr__
    return super(RecursiveScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_script.py", line 291, in __getattr__
    return super(ScriptModule, self).__getattr__(attr)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 948, in __getattr__
    type(self).__name__, name))
AttributeError: 'RecursiveScriptModule' object has no attribute 'copy'

Basically how can I load the saved weights in the model to export them to onnx format? The Pytorch documentation covers normal pytorch models, but when I try to eecute the same steps for torxhscript models, its failing.