title: Image classification in Pytorch

body:
I'm working on facenet-pytorch library in Pytorch, I want to know

the data augmentation should be in train dataset or test data set?

how many images should I put to test data set at least (I've used 2% of images in test data set)

I have 21 classes(21 persons face) and with (vggface2 dataset ) with evaluation mode , does it enough for training and test data set?

how to visualize the images in test dataset to display if a face matched or not
I tried this but it will rise this error :

TypeError: Invalid shape (3, 160, 160) for image data

The shape of images are : (10, 3, 160, 160)
dataiter = iter(test_loader)
images, labels = dataiter.next()
# get predictions
preds = np.squeeze(net(images).data.max(1, keepdim=True)[1].numpy())
images = images.numpy()

# plot the images in the batch, along with predicted and true labels
fig = plt.figure(figsize=(25, 4))
for idx in np.arange(batch_size):
    ax = fig.add_subplot(2, batch_size/2, idx+1, xticks=[], yticks=[])
    ax.imshow(np.squeeze(images[idx]), cmap='gray')
    ax.set_title("{} ({})".format(classes[preds[idx]], classes[labels[idx]]),
             color=("green" if preds[idx]==labels[idx] else "red"))

how to take input faces from webcam after detected the face (prediction function)?

cap = cv.VideoCapture(0)
while True:
    ret, frame = cap.read()
    frame = cv.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
    image = predict_draw_bounding_box(frame)
    cv.imshow('Output', image)
    c = cv.waitKey(1)
    if c == 27:
        break
cap.release()
cv.destroyAllWindows()

But I don't know to implement predict_draw_bounding_box function?
Thanks for any advice


title: Onnx to mlmodel conversion fails to generate .mlmodel file

body:
I'm trying to convert Pytorch model to MLModel with Onnx.

My code:

import torch
from onnx_coreml import convert
import coremltools

net = BiSeNet(19)
net.cuda()
net.load_state_dict(torch.load('model.pth'))
#net.eval()

dummy = torch.rand(1,3,512,512).cuda()

torch.onnx.export(net, dummy, "Model.onnx", input_names=["image"], output_names=["output"], opset_version=11)

finalModel = convert(model='Model.onnx', minimum_ios_deployment_target='12')
finalModel.save('ModelML.mlmodel')

After the code runs Model.onnx is generated, however, .mlmodel file is not generated. There're no errors in the console. This is the output:

2020-04-15 21:49:32.367179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:root:TensorFlow version 2.2.0-rc2 detected. Last version known to be fully compatible is 1.14.0 .
WARNING:root:Keras version 2.3.1 detected. Last version known to be fully compatible of Keras is 2.2.4 .
1.4.0
/content/drive/My Drive/Collab/fp/model.py:116: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size_array = [int(s) for s in feat32.size()[2:]]
/content/drive/My Drive/Collab/fp/model.py:80: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size_array = [int(s) for s in feat.size()[2:]]
/content/drive/My Drive/Collab/fp/model.py:211: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size_array = [int(s) for s in feat.size()[2:]]

What could be the issue?


title: Stacking binary mask frames in pytorch?

body:
I am using Pytorch to attempt to create a 4-dimensional tensor (binary mask) using a "stack" of three-dimensional tensors that each hold binary masks.

The three-dimensional tensors have n instances of some segmented object in a binary mask that is 704 wide and 1080 high.

Lets lets say I have three of these 3-dimensional tensors. They are all size [n,704,1080], where n is the numbers of individual obejcts (and thus, individual binary masks) for the frame, 704 is height and 1080 is width:

t1.size = torch.Size([9, 704, 1280])
t2.size = torch.Size([12, 704, 1280])
t3.size = torch.Size([10, 704, 1280])

I want to create a stack of them by adding a fourth dimension, a. ie. [a, n, 704, 1280], where a is the original 3-dimensional tensor.

The goal is to have a 4-dimensional tensor that can hold the data of numerous 3-dimensional tensors.

I have tried to torch.stack([t1, t2, t3]) but that does not work since the second dimension, n, is not consistent between all the tensors.

How can I go about this since stack does not work?


title: Why did this CUDA error occured in PyTorch?

body:
On the process in building a RNN model, I met with the below error. The following are part of my code:
class RNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(emb_num, emb_size)
        self.dropout1 = nn.Dropout(dropout_rate)
        self.LSTM = nn.LSTM(50, 128, 1, bidirectional = True)
        self.dropout2 = nn.Dropout(dropout_rate)
        self.full_connect = nn.Linear(256 , 5) #  biLSTM state * 2
    def forward(self, x):
        x = self.embedding(x)
        x = x.permute(1,0,2)
        x = self.dropout1(x)
        _, (hn, cn) = self.LSTM(x)
        out = self.dropout2(hn)
        #print(out.shape)
        out = torch.cat([out[i, :, :] for i in range(2)], 1)
        out = out.squeeze()
        out = self.full_connect(out)
        return out

def train():
    optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate, weight_decay = 0.001)
    Loss = nn.CrossEntropyLoss()

    for epoch in range(epochs):
        model.train()
        max_acc = 0
        print('epoch:{}'.format(epoch + 1))
        for i, data in enumerate(trainloader, 0):
            X_train, y_train = data
            optimizer.zero_grad()
            X_train = X_train.long().to(device)
            y_train = y_train.long().to(device)
            output = model(X_train)
            loss = Loss(output, y_train)
            loss.backward()
            optimizer.step()
        print('loss:{:3f}'.format(loss))
        model.eval()
        acc = valid(validloader)
        print('epoch:{} acc:{}'.format(epoch+1, acc))
        if epoch + 1 == 50:
            torch.save(model.state_dict(), 'epoch50.pt')
        if acc > max_acc:
            max_acc = acc
            torch.save(model.state_dict(), 'max_acc model.pt')
    torch.save(model.state_dict(), 'final model.pt')

def valid(dataloader):
    correct = 0
    total = 0
    with torch.no_grad():
        for i, data in enumerate(dataloader, 0):
            X_train, y_train = data
            #optimizer.zero_grad()
            X_train = X_train.long().to(device)
            y_train = y_train.long().to(device)
            output = model(X_train)
            #loss = Loss(output, y_train)
            #loss.backward()
            #optimizer.step()
            correct += (torch.argmax(output, dim = 1) == y_train).sum().item()
            total += y_train.shape[0]
    return correct / total

In the above code, I created a devset to test the model in training. But after 4 epoch or more, this error occured:
Traceback (most recent call last):
  File "c:\Users\hhhh\Desktop\NLP-beginner\task2\task2.py", line 287, in <module>
    train()
  File "c:\Users\hhhh\Desktop\NLP-beginner\task2\task2.py", line 185, in train
    acc = valid(validloader)
  File "c:\Users\hhhh\Desktop\NLP-beginner\task2\task2.py", line 207, in valid
    correct += (torch.argmax(output, dim = 1) == y_train).sum().item()
RuntimeError: CUDA error: unspecified launch failure

I have tried to switch to cpu device to train the model, yet the training speed is to slow even for 1 epoch. Is it because my computer configuration that not enough to run this?


title: Vectorize Pytorch Tensor Indexing Operation

body:
I'm trying to vectorize an operation in PyTorch, but I'm not sure how to do it. This is the code using for loops right now. 'm' is a dictionary with int keys and a 1d tensor as value.
The output mask is 2d. L is the number of layers, this loop is probably necessary. I am therefore looking to mostly replace the 2 inner loops. I was thinking of using torch.gather somehow, but without success
for l in range(L):
    mask = torch.zeros((m[l].shape[0], m[l-1].shape[0]))
    for i in range(m[l].shape[0]):
        for j in range(m[l-1].shape[0]):
            mask[i,j] = R[m[l-1][j], m[l][i]]
    masks.append(mask)

I would appreciate any help! Thanks in advance.


title: rllib - obtain TensorFlow or PyTorch model output from checkpoint

body: I'd like to use the rllib trained policy model in a different code where I need to track which action is generated for specific input states. Using a standard TensorFlow or PyTorch (preferred) network model would provide that flexibility but I can't find clear documentation on how to produce a usable dat or H5 file from a trained rllib agent that I can then load into a torch or tf/Keras model.


title: Reading from cuda array in torch c++ without transferring array to CPU

body:
I am trying to read cuda array located in GPU into torch c++ cuda. Although I am read the array, but it seems that the array is first transferring the cuda array to CPU and then again in torch is transferring array to cuda in its framework. I can say this because the CPU usage is quite high and the transfer of array to torch is not that smooth. I am using the following code, which is working.
The pOutData data is the data pointer to a GPU memory where RGB data is stored. The length of the data at pointer location is {224, 224, 3}. I am receiving pointer from another program, so cannot include the code here.
auto options = torch::TensorOptions().dtype(torch::kUInt8).device(torch::kCUDA, 0);
auto input_tensor = torch::from_blob(pOutData, {224, 224, 3}, options);

Ideally, when there is a data transfer between GPU to GPU location, there should be no or negligible CPU utilization, but I am getting like this:

Can anyone help to read the cuda array directly to the cuda in torch without transferring in CPU?


title: Decoder always predicts the same token

body:
I have the following decoder for machine translation that after a few steps only predicts the EOS token. Overfitting on a dummy, tiny dataset is impossible because of this so it seems that there is a big error in the code.
Decoder(
  (embedding): Embeddings(
    (word_embeddings): Embedding(30002, 768, padding_idx=3)
    (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (dropout): Dropout(p=0.5, inplace=False)
  )
  (ffn1): FFN(
    (dense): Linear(in_features=768, out_features=512, bias=False)
    (layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
    (dropout): Dropout(p=0.5, inplace=False)
    (activation): GELU()
  )
  (rnn): GRU(512, 512, batch_first=True, bidirectional=True)
  (ffn2): FFN(
    (dense): Linear(in_features=1024, out_features=512, bias=False)
    (layernorm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
    (dropout): Dropout(p=0.5, inplace=False)
    (activation): GELU()
  )
  (selector): Sequential(
    (0): Linear(in_features=512, out_features=30002, bias=True)
    (1): LogSoftmax(dim=-1)
  )
)

The forward is relatively straightforward (see what I did there?): pass the input_ids to the embedding and a FFN, then use that representation in the RNN with the given sembedding as initial hidden state. Pass the output through another FFN and do softmax. Return logits and last hidden states of the RNN. In the next step, use those hidden states as the new hidden states, and the highest predicted token as the new input.
def forward(self, input_ids, sembedding):
    embedded = self.embedding(input_ids)
    output = self.ffn1(embedded)
    output, hidden = self.rnn(output, sembedding)
    output = self.ffn2(output)
    logits = self.selector(output)

    return logits, hidden

sembedding is the initial hidden_state for the RNN. This is similar to an encoder-deocder architecture only here we do not train the encoder but we do have access to pretrained encoder representations.
In my training loop I start off each batch with a SOS token and feed every top predicted token to next step until target_len is reached. I also swap randomly between teacher forced training.
def step(self, batch, teacher_forcing_ratio=0.5):
    batch_size, target_len = batch["input_ids"].size()[:2]
    # Init first decoder input woth SOS (BOS) token
    decoder_input = torch.tensor([[self.tokenizer.bos_token_id]] * batch_size).to(self.device)
    batch["input_ids"] = batch["input_ids"].to(self.device)

    # Init first decoder hidden_state: one zero'd second embedding in case the RNN is bidirectional
    decoder_hidden = torch.stack((batch["sembedding"],
                                  torch.zeros(*batch["sembedding"].size()))
                                 ).to(self.device) if self.model.num_directions == 2 \
        else batch["sembedding"].unsqueeze(0).to(self.device)

    loss = torch.tensor([0.]).to(self.device)

    use_teacher_forcing = random.random() < teacher_forcing_ratio
    # contains tuples of predicted and correct words
    tokens = []
    for i in range(target_len):
        # overwrite previous decoder_hidden
        output, decoder_hidden = self.model(decoder_input, decoder_hidden)
        batch_correct_ids = batch["input_ids"][:, i]

        # NLLLoss compute loss between predicted classes (bs x classes) and correct classes for _this word_
        # set to ignore the padding index
        loss += self.criterion(output[:, 0, :], batch_correct_ids)

        batch_predicted_ids = output.topk(1).indices.squeeze(1).detach()

        # if use teacher training: use current correct word for next prediction
        # else do NOT use teacher training: us current predction for next prediction
        decoder_input = batch_correct_ids.unsqueeze(1) if use_teacher_forcing else batch_predicted_ids

    return loss, loss.item() / target_len

I also clip the gradients after each step:
clip_grad_norm_(self.model.parameters(), 1.0)

At first subsequent predictions are already relatively identical, but after a few iterations there's a bit more variation. But relatively quickly ALL predictions turn into other words (but always the same ones), eventually turning into EOS tokens (edit: after changing the activation to ReLU, another token is always predicted - it seems like a random token that always gets repeated). Note that this already happens after 80 steps (batch_size 128).
I found that the returned hidden state of the RNN contains a lot of zeros. I am not sure if that is the problem but it seems like it could be related.
tensor([[[  3.9874e-02,  -6.7757e-06,   2.6094e-04,  ...,  -1.2708e-17,
            4.1839e-02,   7.8125e-03],
         [ -7.8125e-03,  -2.5341e-02,   7.8125e-03,  ...,  -7.8125e-03,
           -7.8125e-03,  -7.8125e-03],
         [ -0.0000e+00, -1.0610e-314,   0.0000e+00,  ...,   0.0000e+00,
            0.0000e+00,   0.0000e+00],
         [  0.0000e+00,   0.0000e+00,   0.0000e+00,  ...,   0.0000e+00,
           -0.0000e+00,  1.0610e-314]]], device='cuda:0', dtype=torch.float64,
       grad_fn=<CudnnRnnBackward>)

I have no idea what might be going wrong although I suspect that the issue is rather with my step than with the model. I already tried playing with the learning rate, disabling some layers (LayerNorm, dropout, ffn2), using pretrained embeddings and freezing or unfreezing them, and disabling teacher forcing, using bidrectional vs unidirectional GRU. The end result is always the same.
If you have any pointers, that would be very helpful. I have googled many things concerning neural networks always predicting the same item and I have tried all the suggestions that I could find. Any new ones, no matter how crazy, are welcome!


title: Why does using a numpy array instead of a torch tensor use more memory in this example?

body:
Using this method gives me a train_data size of about 700Mb and test of about 200Mb
data_tensor = torch.Tensor([np.repeat(image[...,np.newaxis],3,-1)/255.0 for image in data])

np.save(f"/home/vedank/Desktop/code/facial_sentiment/{dataset_type}_data.npy",np.array(data_tensor))

However, using the one below increases the size of train_data to 1.6 Gb and test to ~400Mb
data_tensor = np.array([np.repeat(image[...,np.newaxis],3,-1)/255.0 for image in data])

np.save(f"/home/vedank/Desktop/code/facial_sentiment/{dataset_type}_data.npy",data_tensor)


title: How to install ROCm in anaconda for PyTorch

body:
I am new to Machine Learning and am trying to build an essential MLP. I am trying to increase the speed of my program by running the code on my GPU instead (my CPU is an Intel i7-8550U and GPU Radeon 500 series). I understand that I need to install ROCm however I am not sure how to do so on anaconda.
I am currently running my code using JupyterNotebook on anaconda


title: How to get the last index of model's prediction?

body:
I am new to PyTorch. I have a variable pred which has a list of a tensor.
print(pred)
output: [tensor([[176.64380, 193.86154, 273.84702, 306.30405, 0.83492, 2.00000]])]

So I wanted to access the last element which is the class. I did that by first converting the list into a tensor.
x = torch.stack(pred)
output: tensor([[[176.64380, 193.86154, 273.84702, 306.30405, 0.83492, 2.00000]]])

Now, how do I access the last element or is there any better/efficient way of doing this?
EDIT: For further reference here's the code that performs classification task.
def classify_face(image):
    device = torch.device("cpu")
    img = process_image(image)
    print('Image processed')
    # img = image.unsqueeze_(0)
    # img = image.float()

    pred = model(img)[0]

    # Apply NMS
    pred = non_max_suppression(pred, 0.4, 0.5, classes = [0, 1, 2], agnostic = None )
    if classify:
        pred = apply_classifier(pred, modelc, img, im0s)
    #print(pred)

    model.eval()
    model.cpu()

    print(pred)

    # output = non_max_suppression(output, 0.4, 0.5, classes = class_names, agnostic = False)

    #_, predicted = torch.max(output[0], 1)
    #print(predicted.data[0], "predicted")

    classification = torch.cat(pred)[:, -1]
    index = int(classification)
    print(names[index])
    return names[index]

During prediction pred consists of x1, y1, x2, y2, conf, and class.
e.g. pred = [tensor([[176.64380, 193.86154, 273.84702, 306.30405, 0.83492, 2.00000]])]
If there are no predictions made by the model then pred is simply empty.
e.g. pred = [tensor([], size=(0, 6))]
Presently my program stops prediction if it receives an empty tensor and throws an error:
Traceback (most recent call last):
  File "WEBCAM_DETECT.py", line 168, in <module>
    label = classify_face(frame)
  File "WEBCAM_DETECT.py", line 150, in classify_face
    index = int(classification)
ValueError: only one element tensors can be converted to Python scalars

EDIT1:
It seems to work when I check length of the pred but I get this error when there are two or more rows in tensor.
[tensor([[212.38568, 117.47020, 339.35773, 266.00513,   0.74144,   2.00000],
        [214.60651, 118.50694, 339.90192, 265.91696,   0.44277,   0.00000]])]
#################
#################
Traceback (most recent call last):
  File "WEBCAM_DETECT.py", line 172, in <module>
    label = classify_face(frame)
  File "WEBCAM_DETECT.py", line 154, in classify_face
    index = int(classification)
ValueError: only one element tensors can be converted to Python scalars

How do I make my program sort of just ignore if there are no predictions made at a certain frame and continue onto the next frame?


title: "TextInputSequence must be str” error on Hugging Face Transformers

body:
I’m very new to HuggingFace, I’ve come around this error “TextInputSequence must be str” on a notebook which is helping me a lot to do some practice on various hugging face models. The boilerplate code on the notebook is throwing this error (I guess) due to some changes in huggingface’s API or something. So I was wondering if someone could suggest some changes that I can make to the code to resolve the error.
The error can easily be reproduced by just running all the cells of the notebook.
Link: Colab Notebook
This is the line that is throwing the error-
Here is the error-


title: How can i build pytorch from the source on M1 Mac?

body:
I tried to build pytroch from the source file as right now we can't install pytroch directly using pip.
So i ran few commands which are mentioned below:
1. git clone https://github.com/pytorch/pytorch.git
2. cd pytorch
3. git submodule update --init --recursive
4. python3 setup.py build
5. python3 setup.py install

after the final command, i got the error :
Building wheel torch-1.9.0a0+gitaeaa91b
-- Building version 1.9.0a0+gitaeaa91b
cmake --build . --target install --config Release -- -j 8
Consolidate compiler generated dependencies of target gtest
Consolidate compiler generated dependencies of target clog
Consolidate compiler generated dependencies of target fmt
Consolidate compiler generated dependencies of target pthreadpool
Consolidate compiler generated dependencies of target benchmark
Consolidate compiler generated dependencies of target libprotobuf-lite
[  0%] Built target gtest
[  0%] Built target clog
Consolidate compiler generated dependencies of target c10
[  0%] Built target fmt
[  0%] Built target pthreadpool
Consolidate compiler generated dependencies of target mkrename
Consolidate compiler generated dependencies of target common
Consolidate compiler generated dependencies of target libprotobuf
Consolidate compiler generated dependencies of target mkalias
[  0%] Built target benchmark
[  0%] Built target mkrename
[ 16%] Built target python_copy_files
[ 16%] Built target common
[ 16%] Built target mkalias
[ 17%] Building C object third_party/foxi/CMakeFiles/foxi_loader.dir/foxi/onnxifi_loader.c.o
[ 17%] Building C object sleef/src/libm/CMakeFiles/mkdisp.dir/mkdisp.c.o
[ 17%] Generating ../aten/src/ATen/CPUFunctions.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions.h, ../aten/src/ATen/Declarations.yaml, ../aten/src/ATen/Functions.cpp, ../aten/src/ATen/Functions.h, ../aten/src/ATen/MetaFunctions.h, ../aten/src/ATen/NativeFunctions.h, ../aten/src/ATen/RedispatchFunctions.cpp, ../aten/src/ATen/RedispatchFunctions.h, ../aten/src/ATen/RegisterBackendSelect.cpp, ../aten/src/ATen/RegisterCPU.cpp, ../aten/src/ATen/RegisterCompositeExplicitAutograd.cpp, ../aten/src/ATen/RegisterCompositeImplicitAutograd.cpp, ../aten/src/ATen/RegisterMeta.cpp, ../aten/src/ATen/RegisterMkldnnCPU.cpp, ../aten/src/ATen/RegisterQuantizedCPU.cpp, ../aten/src/ATen/RegisterSchema.cpp, ../aten/src/ATen/RegisterSparseCPU.cpp, ../aten/src/ATen/RegisterSparseCsrCPU.cpp, ../aten/src/ATen/RegistrationDeclarations.h, ../aten/src/ATen/CUDAFunctions.h, ../aten/src/ATen/RegisterCUDA.cpp, ../aten/src/ATen/RegisterQuantizedCUDA.cpp, ../aten/src/ATen/RegisterSparseCUDA.cpp, ../aten/src/ATen/RegisterSparseCsrCUDA.cpp, ../aten/src/ATen/core/ATenOpList.cpp, ../aten/src/ATen/core/TensorBody.h, ../aten/src/ATen/core/TensorMethods.cpp
[ 17%] Built target libprotobuf-lite
[ 17%] Building C object caffe2/CMakeFiles/torch_global_deps.dir/__/torch/csrc/empty.c.o
[ 17%] Building C object sleef/src/libm/CMakeFiles/mkmasked_gnuabi.dir/mkmasked_gnuabi.c.o
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 17%] Building C object sleef/src/libm/CMakeFiles/mkrename_gnuabi.dir/mkrename_gnuabi.c.o
[ 19%] Built target c10
[ 19%] Linking C static library ../../lib/libfoxi_loader.a
[ 19%] Linking C shared library ../lib/libtorch_global_deps.dylib
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 19%] Building C object sleef/src/common/CMakeFiles/arraymap.dir/arraymap.c.o
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 19%] Linking C executable ../../bin/mkdisp
[ 19%] Linking C executable ../../bin/mkmasked_gnuabi
[ 19%] Built target torch_global_deps
[ 19%] Built target foxi_loader
[ 19%] Linking C executable ../../bin/mkrename_gnuabi
[ 19%] Building C object sleef/src/common/CMakeFiles/addSuffix.dir/addSuffix.c.o
[ 21%] Built target libprotobuf
[ 21%] Generating ../../../torch/version.py
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 21%] Generating ../aten/src/ATen/CPUFunctions.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions.h, ../aten/src/ATen/Declarations.yaml, ../aten/src/ATen/Functions.cpp, ../aten/src/ATen/Functions.h, ../aten/src/ATen/MetaFunctions.h, ../aten/src/ATen/NativeFunctions.h, ../aten/src/ATen/RedispatchFunctions.cpp, ../aten/src/ATen/RedispatchFunctions.h, ../aten/src/ATen/RegisterBackendSelect.cpp, ../aten/src/ATen/RegisterCPU.cpp, ../aten/src/ATen/RegisterCompositeExplicitAutograd.cpp, ../aten/src/ATen/RegisterCompositeImplicitAutograd.cpp, ../aten/src/ATen/RegisterMeta.cpp, ../aten/src/ATen/RegisterMkldnnCPU.cpp, ../aten/src/ATen/RegisterQuantizedCPU.cpp, ../aten/src/ATen/RegisterSchema.cpp, ../aten/src/ATen/RegisterSparseCPU.cpp, ../aten/src/ATen/RegisterSparseCsrCPU.cpp, ../aten/src/ATen/RegistrationDeclarations.h, ../aten/src/ATen/CUDAFunctions.h, ../aten/src/ATen/RegisterCUDA.cpp, ../aten/src/ATen/RegisterQuantizedCUDA.cpp, ../aten/src/ATen/RegisterSparseCUDA.cpp, ../aten/src/ATen/RegisterSparseCsrCUDA.cpp, ../aten/src/ATen/core/ATenOpList.cpp, ../aten/src/ATen/core/TensorBody.h, ../aten/src/ATen/core/TensorMethods.cpp
[ 21%] Built target mkmasked_gnuabi
[ 21%] Built target mkdisp
[ 21%] Built target mkrename_gnuabi
[ 21%] Linking C executable ../../bin/addSuffix
[ 21%] Built target arraymap
[ 21%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/init.c.o
[ 21%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/init.c.o
[ 21%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/code_generator.cc.o
[ 21%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/convolution-output.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/api.c.o
[ 22%] Built target addSuffix
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/api.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/convolution-input-gradient.c.o
[ 22%] Building CXX object third_party/googletest/googlemock/CMakeFiles/gmock.dir/src/gmock-all.cc.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/cache.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/cache.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/convolution-kernel.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/arm/uarch.c.o
[ 22%] Built target gen_torch_version
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/arm/uarch.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/fully-connected-output.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/arm/cache.c.o
[ 22%] Building CXX object third_party/googletest/googlemock/gtest/CMakeFiles/gtest_main.dir/src/gtest_main.cc.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/arm/cache.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/max-pooling-output.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/arm/mach/init.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/arm/mach/init.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/softmax-output.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/relu-output.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo.dir/src/mach/topology.c.o
[ 22%] Building C object confu-deps/cpuinfo/CMakeFiles/cpuinfo_internals.dir/src/mach/topology.c.o
[ 22%] Building C object confu-deps/NNPACK/CMakeFiles/nnpack_reference_layers.dir/src/ref/relu-input-gradient.c.o
[ 22%] Linking C static library ../../lib/libcpuinfo.a
[ 22%] Linking C static library ../../lib/libcpuinfo_internals.a
[ 22%] Built target cpuinfo
[ 22%] Built target cpuinfo_internals
[ 22%] Building CXX object third_party/benchmark/src/CMakeFiles/benchmark_main.dir/benchmark_main.cc.o
[ 22%] Linking C static library ../../lib/libnnpack_reference_layers.a
[ 22%] Building CXX object c10/benchmark/CMakeFiles/c10_intrusive_ptr_benchmark.dir/intrusive_ptr_benchmark.cpp.o
[ 22%] Built target nnpack_reference_layers
[ 22%] Linking CXX static library ../../../../lib/libgtest_main.a
[ 22%] Built target renamePURECFMA_SCALAR.h_generated
[ 22%] Built target renamePUREC_SCALAR.h_generated
[ 22%] Built target gtest_main
[ 22%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/command_line_interface.cc.o
[ 22%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_enum.cc.o
[ 22%] Built target renameADVSIMDNOFMA.h_generated
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_enum_field.cc.o
[ 23%] Linking CXX static library ../../../lib/libbenchmark_main.a
[ 23%] Built target benchmark_main
[ 23%] Built target renameADVSIMD.h_generated
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_extension.cc.o
[ 23%] Linking CXX executable ../../bin/c10_intrusive_ptr_benchmark
[ 23%] Built target c10_intrusive_ptr_benchmark
[ 23%] Generating alias_advsimd.h
[ 23%] Built target alias_advsimd.h_generated
[ 23%] Built target renameSVE.h_generated
[ 23%] Built target renameSVENOFMA.h_generated
[ 23%] Building C object sleef/src/libm/CMakeFiles/sleefdetadvsimd.dir/sleefsimdsp.c.o
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_field.cc.o
[ 23%] Building C object sleef/src/libm/CMakeFiles/sleefdetadvsimd.dir/sleefsimddp.c.o
[ 23%] Linking CXX static library ../../../lib/libgmock.a
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_file.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_generator.cc.o
[ 23%] Built target gmock
[ 23%] Building C object sleef/src/libm/CMakeFiles/sleefsvenofma.dir/sleefsimdsp.c.o
clang: warning: argument unused during compilation: '-mfpu=neon' [-Wunused-command-line-argument]
[ 23%] Built target sleefdetadvsimd
[ 23%] Built target headers
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_helpers.cc.o
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Apple clang version 12.0.5 (clang-1205.0.22.9)
Target: aarch64-apple-darwin20.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
clang: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /var/folders/d0/f685f_rj5dqdy0876td4bqz00000gn/T/sleefsimdsp-dd8ced.c
clang: note: diagnostic msg: /var/folders/d0/f685f_rj5dqdy0876td4bqz00000gn/T/sleefsimdsp-dd8ced.sh
clang: note: diagnostic msg: Crash backtrace is located in
clang: note: diagnostic msg: /Users/aman/Library/Logs/DiagnosticReports/clang_<YYYY-MM-DD-HHMMSS>_<hostname>.crash
clang: note: diagnostic msg: (choose the .crash file that corresponds to your crash)
clang: note: diagnostic msg:

********************
gmake[2]: *** [sleef/src/libm/CMakeFiles/sleefsvenofma.dir/build.make:81: sleef/src/libm/CMakeFiles/sleefsvenofma.dir/sleefsimdsp.c.o] Error 254
gmake[1]: *** [CMakeFiles/Makefile2:7845: sleef/src/libm/CMakeFiles/sleefsvenofma.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_map_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_message.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_message_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_padding_optimizer.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_primitive_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_service.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/cpp/cpp_string_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_doc_comment.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_enum.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_enum_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_field_base.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_generator.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_helpers.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_map_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_message.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_message_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_primitive_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_reflection_class.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_repeated_enum_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_repeated_message_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_repeated_primitive_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_source_generator_base.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/csharp/csharp_wrapper_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_context.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_doc_comment.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_enum.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_enum_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_enum_field_lite.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_enum_lite.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_extension.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_extension_lite.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_field.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_file.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_generator.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_generator_factory.cc.o
[ 23%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_helpers.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_map_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_map_field_lite.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_message.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_message_builder.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_message_builder_lite.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_message_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_message_field_lite.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_message_lite.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_name_resolver.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_primitive_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_primitive_field_lite.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_service.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_shared_code_generator.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_string_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/java/java_string_field_lite.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/js/js_generator.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/js/well_known_types_embed.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_enum_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_enum.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_extension.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_file.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_generator.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_helpers.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_map_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_message.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_message_field.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_oneof.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/objectivec/objectivec_primitive_field.cc.o
[ 24%] Built target ATEN_CPU_FILES_GEN_TARGET
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/php/php_generator.cc.o
[ 24%] Generating ../../torch/csrc/autograd/generated/Functions.cpp, ../../torch/csrc/autograd/generated/VariableType_0.cpp, ../../torch/csrc/autograd/generated/VariableType_1.cpp, ../../torch/csrc/autograd/generated/VariableType_2.cpp, ../../torch/csrc/autograd/generated/VariableType_3.cpp, ../../torch/csrc/autograd/generated/VariableType_4.cpp, ../../torch/csrc/autograd/generated/TraceType_0.cpp, ../../torch/csrc/autograd/generated/TraceType_1.cpp, ../../torch/csrc/autograd/generated/TraceType_2.cpp, ../../torch/csrc/autograd/generated/TraceType_3.cpp, ../../torch/csrc/autograd/generated/TraceType_4.cpp, ../../torch/csrc/autograd/generated/ADInplaceOrViewType_0.cpp, ../../torch/csrc/autograd/generated/ADInplaceOrViewType_1.cpp, ../../torch/csrc/autograd/generated/Functions.h, ../../torch/csrc/autograd/generated/variable_factories.h, ../../torch/csrc/autograd/generated/VariableType.h, ../../torch/csrc/autograd/generated/python_functions.cpp, ../../torch/csrc/autograd/generated/python_variable_methods.cpp, ../../torch/csrc/autograd/generated/python_torch_functions.cpp, ../../torch/csrc/autograd/generated/python_nn_functions.cpp, ../../torch/csrc/autograd/generated/python_fft_functions.cpp, ../../torch/csrc/autograd/generated/python_linalg_functions.cpp, ../../torch/csrc/autograd/generated/python_special_functions.cpp, ../../torch/csrc/autograd/generated/python_functions.h, ../../torch/testing/_internal/generated/annotated_fn_args.py
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/plugin.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/plugin.pb.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/python/python_generator.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/ruby/ruby_generator.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/subprocess.cc.o
[ 24%] Building CXX object third_party/protobuf/cmake/CMakeFiles/libprotoc.dir/__/src/google/protobuf/compiler/zip_writer.cc.o
[ 24%] Linking CXX static library ../../../lib/libprotoc.a
[ 24%] Built target libprotoc
[ 24%] Built target generate-torch-sources
gmake: *** [Makefile:146: all] Error 2


title: Issue training RNN model with pytorch with trivial goal

body:
I'm trying to train a simple RNN model with a trivial goal where the output matches a fixed vector regardless of the input

import torch
import torch.nn as nn

from torch.autograd import Variable
import numpy as np

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        print "i2h WEIGHT size ", list(self.i2h.weight.size())
        print "i2h bias size ", list(self.i2h.bias.size())
        self.i2o = nn.Linear(hidden_size, output_size)
        print "i2o WEIGHT size ", list(self.i2o.weight.size())
        print "i2o bias size ", list(self.i2o.bias.size())
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(hidden)
        output = self.softmax(output)
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

n_hidden = 20
rnn = RNN(10, n_hidden, 3)

learning_rate = 1e-3
loss_fn = torch.nn.MSELoss(size_average=False)
out_target = Variable( torch.FloatTensor([[0.0 , 1.0, 0.0]] ) , requires_grad=False)

print "target output::: ", out_target
def train(category_tensor, line_tensor):
    hidden = rnn.initHidden()

    rnn.zero_grad()

    for i in range(line_tensor.size()[0]):
        #print "train iteration ", i, ": input data: ", line_tensor[i]
        output, hidden = rnn(line_tensor[i], hidden)

    loss = loss_fn(output, out_target)
    loss.backward()

    # Add parameters' gradients to their values, multiplied by learning rate
    for p in rnn.parameters():
        #print "parameter: ", p, " gradient: ", p.grad.data
        p.data.add_(-learning_rate, p.grad.data)

    return output, loss.data[0]

current_loss = 0
n_iters = 500

for iter in range(1, n_iters + 1):
    inp = Variable(torch.randn(100,1,10) + 5)
    output, loss = train(out_target, inp)
    current_loss += loss
    if iter % 1 == 0:
      print "weights: ",rnn.i2h.weight
      print "LOSS: ", loss
      print output

As it shows, the loss stays above 6 and never goes down. Notice also that I am biasing all the random inputs normal distributions by 5, so they are mostly positive numbers, so there should exist a weight distribution that approaches the goal output

What am I doing wrong in this example that is failing to output to approach the goal?


title: How to calculate eingenvalues for each matrix in a 4D tensor?

body: I have a 4D tensor at size of [batch_size, channels, height, width]([16, 64, 24, 24]), but I need to calculate the eigenvalue for each 24x24 matrix to obtain a 2D tensor at size of [batch_size, channels]([16, 64]) in which each element represents the eigenvalue of corresponding matrix.


title: Plotting torch.tensor in Pytorch: Invalid dimension error

body:
I have a torch.Tensor object and this has the shape of torch.Size([9, 1, 28, 28])).

I tried something like

for digit in range(10):
  similar_img = create_interpolates(/*something...*/)
  plt.figure(figsize=(10,10))
  plt.imshow(similar_img.detach().numpy())

But this gives me: TypeError: Invalid dimensions for image data

Any help?


title: Multiply every row of a matrix with every row of another matrix

body:
In numpy / PyTorch, I have two matrices, e.g. X=[[1,2],[3,4],[5,6]], Y=[[1,1],[2,2]]. I would like to dot product every row of X with every row of Y, and have the results

[[3, 6],[7, 14], [11,22]]

How do I achieve this?, Thanks!


title: Simple Conv Net

body:
I'm new on pytorch and I would like your light on a small net I try to setup.

class PzConv2d(nn.Module):
    """ Convolution 2D Layer followed by PReLU activation
    """
    def __init__(self, n_in_channels, n_out_channels, **kwargs):
        super(PzConv2d, self).__init__()
        self.conv = nn.Conv2d(n_in_channels, n_out_channels, bias=True,
                            **kwargs)
        self.activ = nn.ReLU()

    def forward(self, x):
        x = self.conv(x)
        return self.activ(x)

class PzPool2d(nn.Module):
    """ Average Pooling Layer
    """
    def __init__(self, kernel_size, stride, padding=0):
        super(PzPool2d, self).__init__()
        self.pool = nn.AvgPool2d(kernel_size=kernel_size,
                                 stride=stride,
                                 padding=padding,
                                 ceil_mode=True,
                                 count_include_pad=False)

    def forward(self, x):
        return self.pool(x)

class PzFullyConnected(nn.Module):
    """ Dense or Fully Connected Layer followed by ReLU
    """
    def __init__(self, n_inputs, n_outputs, withrelu=True, **kwargs):
        super(PzFullyConnected, self).__init__()
        self.withrelu = withrelu
        self.linear = nn.Linear(n_inputs, n_outputs, bias=True)
        self.activ = nn.ReLU()

    def forward(self, x):
        x = self.linear(x)
        if self.withrelu:
            x = self.activ(x)
        return x

class NetCNN(nn.Module):
    def __init__(self,n_input_channels,debug=False):
        super(NetCNN, self).__init__()

        self.n_bins = 180
        self.debug = debug
        self.conv0 = PzConv2d(n_in_channels=n_input_channels,
                              n_out_channels=64,
                              kernel_size=5,padding=2)
        self.pool0 = PzPool2d(kernel_size=2,stride=2,padding=0)

        self.conv1 = PzConv2d(n_in_channels=64,
                              n_out_channels=92,
                              kernel_size=3,padding=2)
        self.pool1 = PzPool2d(kernel_size=2,stride=2,padding=0)

        self.conv2 = PzConv2d(n_in_channels=92,
                              n_out_channels=128,
                              kernel_size=3,padding=2)
        self.pool2 = PzPool2d(kernel_size=2,stride=2,padding=0)

        self.fc0 = PzFullyConnected(n_inputs=12800,n_outputs=1024)
        self.fc1 = PzFullyConnected(n_inputs=1024,n_outputs=self.n_bins)

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

    def forward(self, x, dummy):
        # x:image tensor N_batch, Channels, Height, Width
        #    size N, Channels:=5 filtres, H,W = 64 pixels
        # dummy: is not used

        # stage 0 conv 64 x 5x5
        x = self.conv0(x)
        x = self.pool0(x)

        # stage 1 conv 92 x 3x3
        x = self.conv1(x)
        x = self.pool1(x)

        # stage 2 conv 128 x 3x3
        x = self.conv2(x)
        x = self.pool2(x)

        x = self.fc0(x.view(-1,self.num_flat_features(x)))
        x = self.fc1(x)

        output = x

        return output

I have checked that the dimensions of the intermediate "x" tensor in the forward process is well shaped (at least when I send a random input image tensor). But if you see something strange let me know, please.

Now, I have seen code with F."function" sequence in the forward method, instead of declaring the different layers as I have done. Does it make a difference?

(Notice that I'm using  F.cross_entropy as loss function, so I do not end may network by a SoftMax.)

Thanks.


title: Pytorch: LSTM input and output dimentions

body:
I am a bit confused about LSTM input and output dimensions:

Here is my network:
Intent_LSTM(
  (embedding): Embedding(41438, 400)
  (lstm): LSTM(400, 512, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=512, out_features=3, bias=True)
).

Here the shape of my embdeddings is : [50, 150, 400] 50 being batch size, 150 seq lenth of my input. 400 being my embedded dimensions. I am feeding this into my LSTM. But when I was going through pytorch documentation. It states that input has to be in the form :

input of shape (seq_len, batch, input_size)

So should the input be converted to this format. ([150,50,400]) ?

If yes how do I do that?

Here is my forward pass:

def forward(self, x):
        """
        Perform a forward pass

        """
        batch_size = x.size(0)

        x = x.long()
        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds)

        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        out = self.dropout(lstm_out)
        out = self.fc(out)

        # reshape to be batch_size first
        out = out.view(batch_size, -1,3)
        #print("sig_out",sig_out.shape)
        out = out[:, -1,:] # get last batch of labels

        # return last sigmoid output and hidden state
        return out


title: how to convert TensorFlow Conv2D with kernel_regularizer into PyTorch

body:
hello I have a layer defined as bellow:

tf.keras.layers.Conv2D(filters=32, kernel_size=1, kernel_regularizer=tf.keras.regularizers.l2(l=0.05))

right now I would like to convert it into PyTorch but I don't know how. Can anyone help?


title: Is there any difference between tensor2tensor and pytorch in view of memory?

body:
I'm trying to train seq2seq model(transformer) with pytorch and tensor2tensor.
When using tensor2tensor, the batch size can be like 1024, while pytorch model shows CUDA out of memory error with 8 batch size.

Is there any technique used in tensor2tensor to make best use of memory.

If anyone know this, please tell me.

Thanks in advance.


title: Multiple (2) pytorch optimizers for a single model

body:
I have a use case, where I think using two optimisers would help the model learn better. Let me explain: so, I have one model like so:

my_model=Net()

which is trained over N samples belonging to training_set1. Now, I use results of this trained model (embeddings to be precise) to generate another training_set2, which is a much harder_to_learn training set than training_set1 (btw: I cannot generate training_set2 without training on training_set1).

So, my plan is to use two optimisers like so:

my_model=Net()
optimiser1=Adam(my_model.params()) # rough syntax ;)
optimiser2=Adam(my_model.params()) # rough syntax ;)

for epoch in epochs:
    # train my_model on train_set1 using optimiser1

    # generate training_set2, much harder samples
    # and run a small number of epochs on these harder samples

    for training_set2_epoch in training_set2_epochs:
          # train my_model using optimiser2

is this a valid method to train a single model?

When I use two optimisers like above, my results are much better than when I just use a single optimiser to train the model.


title: Should the queries, keys and values of the transformer be split before or after being passed through the linear layers?

body:
I have seen two different implementations of Multi-Head Attention.

In one of the approaches the queries, keys and values are split into heads before being passed through the linear layers as shown below:

    def split_heads(self, x, batch_size):
        return x.reshape(batch_size, -1, self.heads, self.projection_dim)

    def forward(self, queries, keys, values, mask):
        batch_size = queries.size()[0]

        # split queries keys and values into heads
        queries = self.split_heads(queries, batch_size)
        keys = self.split_heads(keys, batch_size)
        values = self.split_heads(values, batch_size)

        queries = self.queries_linear(queries)
        keys = self.keys_linear(keys)
        values = self.values_linear(values)
        #...more code

The second approach is to split the queries, keys and values into heads after passing them through linear layers:

   def forward(self, queries, keys, values, mask=None):

        batch_size = q.size(0)

        # perform linear operation and split into h heads
        k = self.keys_linear(keys).view(batch_size, -1, self.heads, self.projection_dim)
        q = self.queries_linear(queries).view(batch_size, -1, self.heads, self.projection_dim)
        v = self.values_linear(values).view(batch_size, -1, self.heads, self.projection_dim)
        #...more code

According to the paper Attention Is All You Need, from what I can deduce from the image the queries and keys should be split before being passed through the linear layers, but from most implementation online they are split after.
Are the two approaches similar or is one better than the other?


title: Finding TensorFlow equivalent of Pytorch GRU feature

body:
I am confused about how to reconstruct the following Pytorch code in TensorFlow. It uses both the input size x and the hidden size h to create a GRU layer
import torch
torch.nn.GRU(64, 64*2, batch_first=True, return_state=True)

Instinctively, I first tried the following:
import tensorflow as tf
tf.keras.layers.GRU(64, return_state=True)

However, I realize that it does not really account for h or the hidden size. What should I do in this case?


title: How to call a network in tf.keras.Sequential()?

body:
If I use pytorch, I could use [index] to loop the layers:
layers = nn.ModuleList()
q = nn.ModuleList()
for _ in range(10):
        layers.append(attn)
        q.append(nn.Linear(dim1, dim2))
list = []
for index, layer in enumerate(self.layers):
        Q = q[index](inputTensor)
        list.append(layer(attn))

so when we use tensorflow, can we still use index like pytorch?
layers = tf.keras.Sequential()
q = tf.keras.Sequential()
for _ in range(10):
        layers.add(attn)
        q.add(Dense(dim2))
list = []
for index, layer in enumerate(layers):
        Q = layer[index](inputTensor)
        list.add(layer(att))


title: Using tensordot with torch.sparse tensors

body:
Is it possible to use a similar method as "tensordot" with torch.sparse tensors?
I am trying to apply a 4 dimensional tensor onto a 2 dimensional tensor. This is possible using torch or numpy. However, I did not find the way to do it using torch.sparse without making the sparse tensor dense using ".to_dense()".
More precisely, here is what I want to do without using ".to_dense()":
import torch
import torch.sparse

nb_x = 4
nb_y = 3
coordinates = torch.LongTensor([[0,1,2],[0,1,2],[0,1,2],[0,1,2]])
values = torch.FloatTensor([1,2,3])
tensor4D = torch.sparse.FloatTensor(coordinates,values,torch.Size([nb_x,nb_y,nb_x,nb_y]))
inp = torch.rand((nb_x,nb_y))

#what I want to do
out = torch.tensordot(tensor4D.to_dense(),inp,dims=([2,3],[0,1]))

print(inp)
print(out)

(here is the output: torch_code)
Alternatively, here is a similar code using numpy:
import numpy as np

tensor4D = np.zeros((4,3,4,3))
tensor4D[0,0,0,0] = 1
tensor4D[1,1,1,1] = 2
tensor4D[2,2,2,2] = 3
inp = np.random.rand(4,3)

out = np.tensordot(tensor4D,inp)

print(inp)
print(out)

(here is the output: numpy_code)
Thanks for helping!


title: pytorch network quantization doesn't work

body:
I am doing CNN quantization into different bits, like 16,8,4,2 bits.
Here are the functions I used in the quantization process.
from collections import namedtuple
import torch
import torch.nn as nn
import torch.nn.functional as F

QTensor = namedtuple('QTensor', ['tensor', 'scale', 'zero_point'])

## Quantisation Functions
def calcScaleZeroPoint(min_val, max_val,num_bits=16):
  # Calc Scale and zero point of next
  qmin = 0.
  qmax = 2.**num_bits - 1.

  scale = (max_val - min_val) / (qmax - qmin)

  initial_zero_point = qmin - min_val / scale

  zero_point = 0
  if initial_zero_point < qmin:
      zero_point = qmin
  elif initial_zero_point > qmax:
      zero_point = qmax
  else:
      zero_point = initial_zero_point

  zero_point = int(zero_point)

  return scale, zero_point

def quantize_tensor(x, num_bits=16, min_val=None, max_val=None):

    if not min_val and not max_val:
      min_val, max_val = x.min(), x.max()

    qmin = 0.
    qmax = 2.**num_bits - 1.

    scale, zero_point = calcScaleZeroPoint(min_val, max_val, num_bits)
    q_x = zero_point + x / scale
    q_x.clamp_(qmin, qmax).round_()
    q_x = q_x.round().byte()

    return QTensor(tensor=q_x, scale=scale, zero_point=zero_point)

def dequantize_tensor(q_x):
    return q_x.scale * (q_x.tensor.float() - q_x.zero_point)

## Rework Forward pass of Linear and Conv Layers to support Quantisation
def quantizeLayer(x, layer, stat, scale_x, zp_x):
  # for both conv and linear layers

  # cache old values
  W = layer.weight.data
  B = layer.bias.data

  # quantise weights, activations are already quantised
  w = quantize_tensor(layer.weight.data)
  b = quantize_tensor(layer.bias.data)

  layer.weight.data = w.tensor.float()
  layer.bias.data = b.tensor.float()

  # This is Quantisation Artihmetic
  scale_w = w.scale
  zp_w = w.zero_point
  scale_b = b.scale
  zp_b = b.zero_point

  scale_next, zero_point_next = calcScaleZeroPoint(min_val=stat['min'], max_val=stat['max'])

  # Preparing input by shifting
  X = x.float() - zp_x
  layer.weight.data = scale_x * scale_w*(layer.weight.data - zp_w)
  layer.bias.data = scale_b*(layer.bias.data + zp_b)

  # All int computation
  x = (layer(X)/ scale_next) + zero_point_next

  # Perform relu too
  x = F.relu(x)

  # Reset weights for next forward pass
  layer.weight.data = W
  layer.bias.data = B

  return x, scale_next, zero_point_next

## Get Max and Min Stats for Quantising Activations of Network.
# This is done by running the network with around 1000 examples and getting the
# average min and max activation values before and after each layer.
# Get Min and max of x tensor, and store it
def updateStats(x, stats, key):
  max_val, _ = torch.max(x, dim=1)
  min_val, _ = torch.min(x, dim=1)

  if key not in stats:
    stats[key] = {"max": max_val.sum(), "min": min_val.sum(), "total": 1}
  else:
    stats[key]['max'] += max_val.sum().item()
    stats[key]['min'] += min_val.sum().item()
    stats[key]['total'] += 1

  return stats

# Reworked Forward Pass to access activation Stats through updateStats function
def gatherActivationStats(model, x, stats):

  stats = updateStats(x.clone().view(x.shape[0], -1), stats, 'conv1')

  x = F.relu(model.conv1(x))

  x = model.bn1(x)

  x = F.max_pool1d(x, 2, stride=3)

  stats = updateStats(x.clone().view(x.shape[0], -1), stats, 'conv2')

  x = model.dropout(x)

  x = F.relu(model.conv2(x))

  x = model.bn2(x)

  x = F.max_pool1d(x, 2, stride=2)

  stats = updateStats(x.clone().view(x.shape[0], -1), stats, 'conv3')

  x = model.dropout(x)

  x = F.relu(model.conv3(x))

  stats = updateStats(x, stats, 'fc1')

  x = model.flatten(x)

  x = model.dropout(x)

  x = F.relu(model.fc1(x))

  stats = updateStats(x, stats, 'fc2')

  x = model.fc2(x)

  return stats

# Entry function to get stats of all functions.
def gatherStats(model, test_loader):
    device = 'cuda'

    model.eval()
    test_loss = 0
    correct = 0
    stats = {}
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            stats = gatherActivationStats(model, data, stats)

    final_stats = {}
    for key, value in stats.items():
      final_stats[key] = { "max" : value["max"] / value["total"], "min" : value["min"] / value["total"] }
    return final_stats

## Forward Pass for Quantised Inference
def quantForward(model, x, stats):

  # Quantise before inputting into incoming layers
  x = quantize_tensor(x, min_val=stats['conv1']['min'], max_val=stats['conv1']['max'])

  x, scale_next, zero_point_next = quantizeLayer(x.tensor, model.conv1, stats['conv2'], x.scale, x.zero_point)

  x = model.bn1(x)

  x = F.max_pool1d(x, 2, stride=3)

  x, scale_next, zero_point_next = quantizeLayer(x, model.conv2, stats['conv3'], scale_next, zero_point_next)

  x = model.dropout(x)

  x = model.bn2(x)

  x = F.max_pool1d(x, 2, stride=2)

  x, scale_next, zero_point_next = quantizeLayer(x, model.conv3, stats['fc1'], scale_next, zero_point_next)

  x = model.dropout(x)

  x = x.view(-1, 32)

  x, scale_next, zero_point_next = quantizeLayer(x, model.fc1, stats['fc2'], scale_next, zero_point_next)

  x = model.flatten(x)

  x = model.dropout(x)

  # Back to dequant for final layer
  x = dequantize_tensor(QTensor(tensor=x, scale=scale_next, zero_point=zero_point_next))

  x = model.fc2(x)

  return F.log_softmax(x, dim=1)

My test function is as follow
    def testQuant(self, dataloader, loss_fn, epoch=-1, info='', quant=False, stats=None):
        self.model.eval()
        nb_classes = 5
        confusion_matrix = torch.zeros(nb_classes, nb_classes)
        Sen = torch.zeros(1, nb_classes)
        pre = torch.zeros(1, nb_classes)
        Spe = torch.zeros(1, nb_classes)
        F1 = torch.zeros(1, nb_classes)
        desc = f'{info}Epoch #{epoch + 1}'
        with torch.no_grad():
            with tqdm(total=len(dataloader), desc=desc) as progress_bar:
                info_avg = {}
                for batch_idx, (data, labels) in enumerate(dataloader):
                    if quant:
                        preds = quantForward(self.model, data, stats)
                    else:
                        preds = self.model(data)
                    loss = loss_fn(preds, labels)
                    info_show = self._update_info(preds, labels, loss, info_avg)
                    progress_bar.set_postfix(**info_show)
                    progress_bar.update(1)
                    _, preds = torch.max(preds, 1)
                    for t, p in zip(labels.view(-1), preds.view(-1)):
                        confusion_matrix[t.long(), p.long()] += 1
        FP = confusion_matrix.sum(axis=0) - np.diag(confusion_matrix)
        FN = confusion_matrix.sum(axis=1)  - np.diag(confusion_matrix)
        TP = np.diag(confusion_matrix)
        TN = confusion_matrix.sum() - (FP + FN + TP)

        FP = torch.Tensor(FP)
        FN = torch.Tensor(FN)
        TP = torch.Tensor(TP)
        TN = torch.Tensor(TN)

        # Sensitivity, hit rate, recall, or true positive rate
        TPR = TP/(TP+FN)
        # Specificity or true negative rate
        TNR = TN/(TN+FP)
        # Precision or positive predictive value
        PPV = TP/(TP+FP)
        # # Negative predictive value
        # NPV = TN/(TN+FN)
        # # Fall out or false positive rate
        # FPR = FP/(FP+TN)
        # # False negative rate
        # FNR = FN/(TP+FN)
        # # False discovery rate
        # FDR = FP/(TP+FP)
        F1 = 2 * (TPR * PPV) / (TPR+PPV)

        # Overall accuracy
        ACC = (TP+TN)/(TP+FP+FN+TN)
        # print('Sensitivity for each class',confusion_matrix.diag()/confusion_matrix.sum(1))
        print('Acc',ACC)
        print('Sensitivity',TPR)
        print('Specificity',TNR)
        print('Precision',PPV)
        print('F1 score',F1)
        plt.figure(figsize=(10,10))
        plot_confusion_matrix(confusion_matrix.numpy(), ['N','SVEB','VEB','F','Q'])
        return info_show

The problem is I get many 'nan' in my metrics, when I run
stats = gatherStats(model, loader_test)
trainer.testQuant(loader_test, loss_fn, info='Test_Quantization ',quant=True, stats=stats)

I get the following results
!python "/content/drive/My Drive/ECG_quantization/main.py"
Test_Quantization Epoch #0: 100% 129/129 [00:02<00:00, 58.98it/s, Acc=0.8231, F1 score=nan, Loss=0.8739, Sensitivity(recall)=0.0000, Specificity=1.0000, precision=nan]
/content/drive/My Drive/ECG_quantization/trainer.py:219: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  TP = torch.Tensor(TP)
Acc tensor([0.8257, 0.9743, 0.9350, 0.9924, 0.9240])
Sensitivity tensor([1., 0., 0., 0., 0.])
Specificity tensor([0., 1., 1., 1., 1.])
Precision tensor([0.8257,    nan,    nan,    nan,    nan])
F1 score tensor([0.9046,    nan,    nan,    nan,    nan])
Normalized confusion matrix

But when I run
trainer.testQuant(loader_test, loss_fn, info='Test_Quantization ',quant=False)

I can get correct results without quantization.
I want to ask what's wrong with my quantization code?


title: How to assign NaN to tensor element?

body:
I want to assign NaN to a tensor element.
import torch
x = torch.tensor([1, 2, 3])
x[x == 2] = None

I have the error TypeError: can't assign a NoneType to a torch.LongTensor.
I need it to make sure that some later sophisticated calculations are not made for certain values of x.


title: pytorch doesn't give expected output

body:
Firstly, a bunch of data is classified by the CNN model. Then, I'm trying to make prediction on correctly classified data from first step, which is expected to give an accuracy of 100%. However, I found the result is unstable, sometimes 99+%, but not 100%. Is there anybody know what is the problem with my code? Thank you very much in advance, it has troubled me several days ~ ~

torch.version

'0.3.1.post2'

import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable

n = 2000
data = np.random.randn(n, 1, 10, 10)
label = np.random.randint(2, size=(n, ))

def test_pred(model, data_test, label_test):

    data_batch = data_test
    labels_batch = label_test

    images = torch.autograd.Variable(torch.FloatTensor(data_batch))
    labels = torch.autograd.Variable(torch.FloatTensor(labels_batch))

    outputs = model(images)

    _, predicted = torch.max(outputs.data, 1)

    correct = (np.array(predicted) == labels_batch).sum()

    label_pred = np.array(predicted)

    acc = correct/len(label_test)
    print(" acc:", acc)

    return acc, label_pred

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.fc = nn.Linear(128, 2)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out

cnn = CNN()

[_, label_pred] = test_pred(cnn, data, label)

print("Acc:", np.mean(label_pred==label))
# Given the correctly classified data in previous step, expect to get 100% accuracy
# Why it sometimes doesn't give a 100% accuracy ?
print("Using selected data size {}:".format(data[label_pred==label].shape))
_, _ = test_pred(cnn, data[label_pred==label], label[label_pred==label])

output:

acc: 0.482

Acc: 0.482

Using selected data size (964, 1, 10, 10):

acc: 0.9979253112033195


title: Getting FileNotFoundError exception on Pytorch 0.4, python 2.7 - after applying fix

body:
I'm working with Pytorch 0.4, I got the  FileNotFoundError Exception in functional.py when it invokes the binary_cross_entropy loss. Here is the stack trace:

loss += torch.mean(F.binary_cross_entropy(F.sigmoid(out0), all0) +

File "/home/local/.../anaconda2/envs/py27/lib/python2.7/site-packages/torch/nn/functional.py", line 1483, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(input, target, weight, size_average, reduce)
RuntimeError: cudaEventSynchronize in future::wait: device-side assert triggered

I'm aware of this type of issues like https://github.com/pytorch/pytorch/issues/6932. I added the fix in the locations suggested by the users in threads, however, the exception still occurs.  I'd be happy for any help.
Best,
S.


title: Find all ReLU layer in a torchvision model

body:
After I fetch a pre-trained model from torchvision.models, I want all the ReLU instance to register_backward_hook(f),which is like this:

for pos, module in self.model.features._modules.items():
    for sub_module in module:
        if isinstance(module, ReLU):
            module.register_backward_hook(f)

The problems for me is how to find all ReLU in a model. For densenet161, the ReLU exists not only in model.features._modules but also in self-defined dense layer, eg. model.features._modules['denseblock1'][0]. For  resnet151, the ReLU exists in model._modules and its self-define layer, eg model._modules['layer1'].

Is there any way to find all ReLU inside a model?


title: Common class for Linear, Conv1d, Conv2d,..., LSTM,

body:
Is there any class that all torch::nn::Linear, torch::nn::Conv1d, torch::nn::Conv2d, ... torch::nn::GRU, .... all inherit from that? torch::nn::Module seems be a good option, though there is a middle class, called torch::nn::Cloneable, so that torch::nn::Module does not work. Also, torch::nn::Cloneable itself is a template so that needs type in the declaration.
I want to create a general class model, which has std::vector<the common class> layers, so that later I can fill layers with any type of layer that I want, e.g., Linear, LSTM, etc. Is there such a capability in the current API? This can be done easily in python, though here we need declaration and this hinders the python's easiness.

Thanks,
Afshin


title: Pytorch nn Module generalization

body:
Let us take a look at the simple class:

class Temp1(nn.Module):

    def __init__(self, stateSize, actionSize, layers=[10, 5], activations=[F.tanh, F.tanh] ):

        super(Temp1, self).__init__()
        self.layer1 = nn.Linear(stateSize, layers[0])
        self.layer2 = nn.Linear(layers[0], layers[1])
        self.fcFinal = nn.Linear( layers[1], actionSize )
        return

This is a fairly straight forward pytorch module. It creates a simple sequential dense network. If we check its hidden parameters, we see the following:

t1 = Temp1(2, 2)
list(t1.parameters())

This is the expected result ...

[Parameter containing:
 tensor([[-0.0311, -0.5513],
         [-0.0634, -0.3783],
         [-0.2514,  0.6139],
         [ 0.4711, -0.0241],
         [-0.1739,  0.2208],
         [-0.1533,  0.3838],
         [-0.6490, -0.5784],
         [ 0.5312,  0.6703],
         [ 0.3506,  0.3652],
         [ 0.1768, -0.4158]], requires_grad=True), Parameter containing:
 tensor([-0.3199, -0.4154, -0.5530, -0.6738, -0.4411,  0.2641, -0.3576,  0.0447,
          0.0254,  0.0965], requires_grad=True), Parameter containing:
 tensor([[-2.8257e-01,  6.7583e-02,  9.0356e-02,  1.0868e-01,  4.0876e-02,
           4.0616e-02,  4.4419e-02, -8.1544e-02,  2.5244e-01,  3.8777e-03],
         [-8.0950e-03, -1.4175e-01, -2.9492e-01,  3.1439e-01, -2.3065e-01,
          -6.6631e-02,  3.0047e-01,  2.8353e-01,  2.3457e-01, -3.1399e-03],
         [-5.2522e-02, -2.2183e-01, -1.5485e-01,  2.6317e-01,  2.8273e-01,
          -7.4823e-02, -5.3704e-02,  9.3526e-02, -1.7916e-01, -3.1132e-04],
         [ 8.9063e-02,  2.9263e-01, -1.0052e-01,  8.7005e-02, -1.1246e-01,
          -2.7968e-01,  4.1411e-02, -1.6776e-01,  1.2363e-01, -2.2808e-01],
         [ 2.9244e-02,  5.8296e-02, -2.9729e-01, -3.1437e-01, -9.3182e-02,
          -7.5236e-03,  5.6159e-02, -2.2075e-02,  1.0337e-01,  8.1123e-02]],
        requires_grad=True), Parameter containing:
 tensor([ 0.2240,  0.0997, -0.0047, -0.1784, -0.0369], requires_grad=True), Parameter containing:
 tensor([[ 0.3546, -0.2180,  0.1723, -0.0463,  0.2572],
         [-0.1669, -0.1364, -0.0398,  0.2233, -0.1805]], requires_grad=True), Parameter containing:
 tensor([ 0.0871, -0.1698], requires_grad=True)]

Now, let us try to generalize this a bit:

class Temp(nn.Module):

    def __init__(self, stateSize, actionSize, layers=[10, 5], activations=[F.tanh, F.tanh] ):

        super(Temp, self).__init__()

        # Generate the fullly connected layer functions
        self.fcLayers = []

        oldN = stateSize
        for i, layer in enumerate(layers):
            self.fcLayers.append( nn.Linear(oldN, layer) )
            oldN = layer
        self.fcFinal = nn.Linear( oldN, actionSize )
        return

It turns out that the number of parameters within this module is no longer the same ...

t = Temp(2, 3)
list(t.parameters())

[Parameter containing:
 tensor([[-0.3342,  0.4111,  0.0418,  0.4457,  0.0648],
         [ 0.4364, -0.0360, -0.2239,  0.4025,  0.1661],
         [ 0.1932, -0.0896,  0.3269, -0.2179,  0.1035]], requires_grad=True),
 Parameter containing:
 tensor([-0.2867, -0.1354, -0.0026], requires_grad=True)]

I believe understand why this is happening. The bigger question is, how do we overcome this problem? The second, generalized method for example will not be sent to the GPU properly, and will not be trained by an optimizer.


title: Pytorch neural network (probably) does not learn

body:
my homework is to train a network on a given data set of 3000 frogs, cats and dogs. The network I built doesn't seem to improve at all. Why is that?

The training data x_train is a numpy ndarray of shape (3000,32,32,3).

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(3,28,5)
        self.conv2 = nn.Conv2d(28,100,5)
        self.fc1 = nn.Linear(2500,120)
        self.fc2 = nn.Linear(120,3)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x,2)
        x = F.relu(x)
        x = x.view(-1,2500)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x)

model = Netz()
optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)

def train(epoch):
    model.train()
    avg_loss = 0
    correct = 0
    criterion = F.nll_loss
    for i in range(len(x_train)):
        optimizer.zero_grad()
        x = torch.tensor(x_train[i])
        x = x.permute(2, 0, 1)
        x = Variable(x)
        x = x.unsqueeze(0)
        target = Variable(torch.Tensor([y_train[i]]).type(torch.LongTensor))
        out = model(x)
        loss = criterion(out, target)
        avg_loss += loss
        pred = out.argmax(dim=1, keepdim=True)
        correct += pred.eq(target.view_as(pred)).sum().item()
        loss.backward()
        optimizer.step()
        if i%64==0:
            print("epoch ", epoch, " [", i, "/", len(x_train), "] average loss: ", avg_loss.item() / 64, " correct: ", correct, "/64")
            avg_loss = 0
            correct = 0

I expect the mean error to decrease over time, but it seems to keep fluctuating around the same number...


title: flask-opencv-streamer werkzeug.routing.BuildError: Could not build url for endpoint 'calc'. Did you mean 'static' instead?

body:
I am trying to figure out how to actually run the flask-opencv-streamer python module properly, I am currently trying to run the example given here without authentication : https://pypi.org/project/flask-opencv-streamer/

I've attached the error here :
https://pasteboard.co/Iso8Z3C.png


title: rnnFusedPointwise no longer available in >=PyTorch 1.0

body:
I had a codebase that runs fine in PyTorch 0.4 but whenever I try to run it using PyTorch 1.0+, it complains about the following import.

from torch.nn._functions.thnn import rnnFusedPointwise as fusedBackend

I use fusedBackend in the following code snippet.

if input.is_cuda:
    igates = F.linear(input, w_ih)
    hgates = F.linear(hx, w_hh)
    state = fusedBackend.LSTMFused.apply
    return state(igates, hgates, cx, b_ih, b_hh)

Is there any way I can fix the import error for PyTorch 1.0+?


title: Create all binary valued vectors of specific length

body: How do I efficiently create a matrix of shape (2**N, N) with all binary vectors of length N as its rows? I call a vector binary iff each of its components equals either 0 or 1. I want the matrix to have dtype=torch.float.


title: Slow video streaming while using pytorch with cuda

body:
I’m using Jetson Xavier (with maxn config) and I have created python multiprocess application for video analytics, containing the following 2 processes (Entirely Separate with no inter process communication):

Simple process for capture video source with openCV and showing it with openCV imshow .
Process that taking constant tenzor and run it through a pytorch nn model with cuda in an infinite loop.
(I started with 2 connected processes for capture the video and then process the frames concurrent but to debug the problem I broke all of the connection and ran it on a constant tensor)

The problem is that when only the video process is running the video is smooth. But when the NN process works concurrently the video become not smooth, that, even though the fps for the video process is 25, like I designed it by waitkey.
I suspect that I’m using all my gpu resources for the NN and then the frame show rendering is on hold for a few milliseconds until the gpu is free for render. If it’s indeed the reason for my problem, can I determine priority for gpu usage between the two processes? Do you have another idea how to solve it?

This is the pseudo code (the original is offline and I can’t upload it):

Process1:

Video = cv2.capture (video_source)
While True
    Frame = Video.get()
    Cv2.imshow(frame)
    Waitkey(40)
    # Check fps

Process2:

model = # loading the pytorch NN model
tenzor = # creating zeros pytorch cuda tenzor
while True
    model(tenzor)

Attaching tegrastats from running time:

Thanks
moti


title: Pythonic Nested for - loops in Python

body:
I am working on this code where I have nested for loops. a_list and b_list are list of tuples, where each tuple is made up of two tensors [(tens1, tens2), ...]. I am trying to compute the similarity of every tens1 in a_list to every tens1 in b_list. Below is the code I have. And the nested loop appears to be a bottleneck. Is there a better way(pythonic) that I can re-write the loops?

a2b= defaultdict(dict)
b2a= defaultdict(dict)
ab_sim = []

for a, vec_a in a_list:
    for b, vec_b in b_list:
        # Ignore combination if the first element in both a and b are same
        if a[0] == b[0]:
            continue
        # Calculate cosine similarity of combination
        sim = self.calculate_similarity(vec_a, vec_b )
        a2b[a][b] = sim
        b2a[b][a] = sim
        ab_sim.append(sim)

The calculate_similarity is just a method computing cosine similarity. a_list and b_list could be of any size. I have b2a and a2b because I need them for other computations.


title: Is there any method in reinforcement Learning to select multiple simultaneous actions?

body:
I'm working on a research project that involves the application of reinforcement learning to planning and decision-making problems. Typically, these problems involve picking (sampling) multiple actions within a state based on ranking [max_q to min_q]. The RL literature seems focused on policies that map from a set of states to a single individual action not multiple. Does anyone know of approaches that can not only map states to multiple simultaneous actions but also that can maintain relationships between these actions?
Here is the DQN action selection function source code

Thanks


title: Extracting indexes from a max pool over uniform data

body:
I'm trying to find max points in a 2D tensor for a given kernel size, but I'm having issues with a special case where all the values are uniform. For example, given the following example, I would like to mark each point as a max point:

+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+
| 5 | 5 | 5 | 5 |
+---+---+---+---+

If I run torch.nn.functional.max_pool2d with a kernel size=3, stride=1, and padding=1, I get the following indicies:

+---+---+---+----+
| 0 | 0 | 1 |  2 |
+---+---+---+----+
| 0 | 0 | 1 |  2 |
+---+---+---+----+
| 4 | 4 | 5 |  6 |
+---+---+---+----+
| 8 | 8 | 9 | 10 |
+---+---+---+----+

What changes do I need to account for to instead obtain the following indicies?

+----+----+----+----+
| 1  | 2  | 3  |  4 |
+----+----+----+----+
|  5 |  6 |  7 |  8 |
+----+----+----+----+
|  9 | 10 | 11 | 12 |
+----+----+----+----+
| 13 | 14 | 15 | 16 |
+----+----+----+----+


title: PyTorch equivalent of numpy's np.random.RandomState

body:
I'm looking for a way to create random objects without actually altering the pytorch global seed.

i.e. an equivalent to numpy's:

rand_gen = np.random.RandomState(seed)
rand_gen.randint(0, 256, self.image_dim))  # for example


title: How to convolute 2d image, tensor depth 3, using Linear Layer? (in PyTorch)

body:
Given a tensor shape (3, 256, 256).
I would like to convolute or loop through it pixel by pixel to return me a tensor shape (1, 256, 256).

This may sound a bit confusing so here is my code till now so you know I mean.

class MorphNetwork(nn.Module):
    def __init__(self):
        super().__init__()

        self.fc1 = nn.Linear(3, 8)
        self.fc2 = nn.Linear(8, 1)

    def forward(self, x):
        # The input here is shape (3, 256, 256)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        # Returned shape should be (1, 256, 256)
        return x

As you can see my Linear layer accept's shape 3 which matches the depth of my original tensor. What is the best way of looping through all 256x256 to return me tensor shape (1, 256, 256)


title: Is there a difference between multiple times loss.backward() and loss.backward() after multiplying loss by n in Pytorch?

body:
Is there a difference between these two codes?

1

Loss.backward(retain_graph=True)
Loss.backward(retain_graph=True)
Loss.backward()
optimizer.step

2

Loss = 3 * Loss
Loss.backward()
optimizer.step

When I checked the gradient of the parameter after the last backward(), there was no difference between the two codes. However, there is a little difference in test accuracy after training.

I know this is not a common case, but it is related to the research I'm doing.


title: Is there a hardware benchmark for deep learning?

body:
In this article from Tim Dettmers: Full hardware guide for Deep Learning,  he states that PCI lanes speed doesn't matter a lot for Deep Learning.

But on this question, the approved answer states that a CPU with the right amount of PCI lanes is required for Deep Learning. (according to me PCI lane speed and amount of PCI lanes managed by CPU are dependent)

So, I'm looking at a benchmark for Deel Learning to compare those theories/facts.

Looking at the existing benchmarks for Deep Learning, it seems that they are all only testing GPUs capacity, comparing one Graphic card to another, focusing on matrix multiplication.

I'm wondering if there is a benchmark that scores the sum of all hardware components required for deep learning (and especially the training phase): CPU, Memory, GPU, PCIe speed, hard drive, motherboard.

To list what I have tested:

DAWNBench/MLPerf Define the specifications of a DL benchmark, whatever the platform and the framework. Their code is partially incomplete.
AI Benchmark primarily focusing on mobile. Simple to use with the code available and a friendly scoring system. But even with a high precision parameter, the models are tested on small batches (I guess it's to keep it usable on mobile)
AI Matrix from Alibaba: A very heavy and complete one but testing only the GPUs.
Lambda Labs Benchmark, I didn't get it working... non-available version of Tensorflow, non-compatible with recent Cuda versions..

(I know this is related to hardware, but as I'm looking for a software library, I post here!)


title: Horovod Elasticity Adjust number of Workers in RunTime

body:
I have been using Horovod using both TensorFlow and PyTorch in docker, every thing works fine under a fixed number of containers as explained in Horovod docker

I have checked the Horovod Elastic Demos Horovod examples but they doesn't show how to change the number of workers in runtime.

What I need is to know how to change the number of workers up or down by runtime?


title: Difference betwee torch.unsqueeze and target.unsqueeze

body:
I am training a simple MLP by computing the MSE and get the following error:

UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

The following gives me the right solution
target = target.unsqueeze(1) while torch.unsqueeze(target,1) does not. The former solution is from a previous question and the latter comes from the documentation
Why does the latter fix the UserWarning message with the former doesn't?


title: Pytorch: Load and use EMLo embeddings in a model

body: I am trying to understand how to load and use the ELMo embeddings in a classical RNN/LSTM/GRU model like we do with Glove or Fasttext. I saw the documentation of AllenNLP, but I am not interested of using all things that they recommend. Its like a complete other way of building a model and I do not want to go that way. Any ideas?


title: IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2) on Transformers in PyTorch

body:
I have 14 properties and 1 label output and output class number 10, I want to run it on transformers, but I'm getting this error.When I ran this code in MLP model, I had no problem, but I am having problems here.
Here is an example from my keys value: [ 1.3870e-01, -5.7892e-02, 2.0621e-01, -6.8972e-02, 7.8414e-02, 5.7779e-02, 3.0018e-01, -6.9849e-02]
the bug is already here, on multiHeadAttetionself._scores = torch.bmm(queries, keys.transpose(1,2)) / np.sqrt(K)
https://github.com/nurkbts/error/blob/main/error.ipynb
K=64
IndexError Traceback (most recent call last)
<ipython-input-191-239442fa3bfb> in <module>()
    227         optimizer.zero_grad()
    228         print(x)
--> 229         netout = net(x.to(device))
    230         # calculate loss
    231         loss = loss_function(yhat, y)

5 frames
/content/multiHeadAttention.py in forward(self, query, key, value, mask)
     89
     90         # Scaled Dot Product
---> 91         self._scores = torch.bmm(queries, keys.transpose(1, 2)) / np.sqrt(K)
     92
     93         # Compute local map mask

IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

Here is the code:
# dataset definition
class CSVDataset(Dataset):
    # load the dataset
    def __init__(self, path):
        # load the csv file as a dataframe
        df = read_csv(path,sep=',',names=KINEMATICS_COL_NAMES,usecols=KINEMATICS_USECOLS)
        # store the inputs and outputs
        self.X = df.values[:, :-1]
        self.y = df.values[:, -1]
        # ensure input data is floats
        self.X = self.X.astype('float32')
        # label encode target and ensure the values are floats
        self.y = LabelEncoder().fit_transform(self.y)
        print(self.X.shape)
        print(self.y.shape)
                # Convert to float32
        self.X = torch.Tensor(self.X)
        self.y = torch.Tensor(self.y)

    # number of rows in the dataset
    def __len__(self):
        return len(self.X)

    # get a row at an index
    def __getitem__(self, idx):
      if torch.is_tensor(idx):
       idx = idx.tolist()
      return [self.X[idx], self.y[idx]]

    # get indexes for train and test rows
    def get_splits(self, n_test=0.33):
        # determine sizes
        test_size = round(n_test * len(self.X))
        val_size =100
        train_size = len(self.X) - test_size-100
        # calculate the split
        return random_split(self, [train_size, val_size, test_size])

# prepare the dataset
def prepare_data(path):
    # load the dataset

    dataset = CSVDataset(path)
    # calculate split
    train, val, test = dataset.get_splits()
    # prepare data loaders
    train_dl = DataLoader(train, batch_size=BATCH_SIZE,
                              shuffle=True,
                              num_workers=NUM_WORKERS,
                              pin_memory=False)
    val_dl= DataLoader(val, batch_size=BATCH_SIZE,
                            shuffle=True,
                            num_workers=NUM_WORKERS
                              )
    test_dl = DataLoader(test, batch_size=BATCH_SIZE,
                             shuffle=False,
                             num_workers=NUM_WORKERS)
    return train_dl, val_dl,test_dl

# prepare the data
path = '_B001_train.txt'
train_dl, val_dl, test_dl = prepare_data(path)
print(train_dl)
print(len(train_dl.dataset), len(val_dl.dataset),  len(test_dl.dataset))
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(f"Using device {device}")

BATCH_SIZE = 8
NUM_WORKERS = 0
LR = 2e-4
EPOCHS = 30
# Model parameters
d_model = 64 # Lattent dim #???
q = 8 # Query size
v = 8 # Value size
h = 8 # Number of heads
N = 4 # Number of encoder and decoder to stack
attention_size = 12 # Attention window size
dropout = 0.2 # Dropout rate
pe = None # Positional encoding
chunk_mode = None
d_input = 14 # From dataset
d_output = 10 # From dataset#???
#model = MLP(14)
# Load transformer with Adam optimizer and MSE loss function
sns.set()
net = Transformer(d_input, d_model, d_output, q, v, h, N, attention_size=attention_size, dropout=dropout, chunk_mode=chunk_mode, pe=pe).to(device)
optimizer = optim.Adam(net.parameters(), lr=LR)
#loss_function = OZELoss(alpha=0.3)
from torch.nn import CrossEntropyLoss
loss_function = CrossEntropyLoss
# Prepare loss history

hist_loss = np.zeros(EPOCHS)
hist_loss_val = np.zeros(EPOCHS)
for idx_epoch in range(EPOCHS):
    running_loss = 0
      for idx_batch, (x, y) in enumerate(train_dl):
        optimizer.zero_grad()
        netout = net(x.to(device)) ----> error
        loss = loss_function(yhat, y)
        loss.backward()
        optimizer.step()


title: How does torch.einsum perform this 4D tensor multiplication?

body:
I have come across a code which uses torch.einsum to compute a tensor multiplication. I am able to understand the workings for lower order tensors, but, not for the 4D tensor as below:
import torch

a = torch.rand((3, 5, 2, 10))
b = torch.rand((3, 4, 2, 10))

c = torch.einsum('nxhd,nyhd->nhxy', [a,b])

print(c.size())

# output: torch.Size([3, 2, 5, 4])

I need help regarding:

What is the operation that has been performed here (explanation for how the matrices were multiplied/transposed etc.)?
Is torch.einsum actually beneficial in this scenario?


title: Implement Pseudo labeling in Pytorchvision

body:
I have difficulty implementing pseudo labeling in pytorch.
this is the steps:

I've already got unlabeled dataset(initially all label is 0), and load with datasetfolder

for each epoch, I use my trained model to make prediction on unlabeled data

If the probability is > threshold, then I confirm the label of this unlabeled data

return the datasetfolder (filtered)

The first thing is I don't know how to use target_transform to modify the label
and I need to "filter" the other data whose probability is not > threshold
so maybe I have to delete some data? (but I still have to preserve the original data, so that next epoch can do prediction on the original datasetfolder)
I'm wondering if I can make a copy of original datasetfolder, and filter the data I want, then return it
is it possible to do without creating custom datasetfolder?


title: The best way to implement stateful LSTM/ConvLSTM in Pytorch?

body:
I am trying to boost the performance of a object detection task with sequential information, using ConvLSTM.
A typical ConvLSTM model takes a 5D tensor with shape (samples, time_steps, channels, rows, cols) as input.
as stated in this post, a long sequence of 500 images need to be split into smaller fragments in the Pytorch ConvLSTM layer. For example, it could be split into 10 fragements with each having 50 time steps.

I have two goals:

I want the network to remember the state across the 10 fragment sequences. I.e. how to pass the hidden state between the fragements?

I want to feed in the images (of the video) one by one. I.e. the long sequence of 500 images is split into 500 fragments with each one having only one image. The input should be like (all_samples, channels, rows, cols). This only make sense if the 1.goal could be achieved.

I found some good answers for Tensorflow, but I am using Pytorch.
TensorFlow: Remember LSTM state for next batch (stateful LSTM)
The best way to pass the LSTM state between batches
What is the best way to implement stateful LSTM/ConvLSTM in Pytorch?


title: CUDA version of package not importing?

body:
Firstly, I installed torch 1.1.0, and then I installed its' dependencies. So, I can import torch_scatter 1.2.0 however I get this error when importing torch_scatter.scatter_cuda:
 import torch_scatter.scatter_cuda
ModuleNotFoundError: No module named 'torch_scatter.scatter_cuda'

I have Cuda v10 installed and I have a GPU. All of the requirements for this code were installed together through pip in one go on my virtual environment.


title: RNN Implementation

body:
I am going to implement RNN using Pytorch . But , before that , I am having some difficulties in understanding the character level one-hot encoding which is asked in the question .

Please find below the question

Choose the text you want your neural network to learn, but keep in mind that your
data set must be quite large in order to learn the structure! RNNs have been trained
on highly diverse texts (novels, song lyrics, Linux Kernel, etc.) with success, so you
can get creative. As one easy option, Gutenberg Books is a source of free books where
you may download full novels in a .txt format.
We will use a character-level representation for this model. To do this, you may use
extended ASCII with 256 characters. As you read your chosen training set, you will
read in the characters one at a time into a one-hot-encoding, that is, each character
will map to a vector of ones and zeros, where the one indicates which of the characters
is present:

char → [0, 0, · · · , 1, · · · , 0, 0]
Your RNN will read in these length-256 binary vectors as input.

So , For example , I have read a novel in python. Total unique characters is 97. and total characters is somewhere around 300,000 .

So , will my input be 97 x 256 one hot encoded matrix ?

or will it be 300,000 x 256 one hot encoded matrix ?


title: why can't I reimplement my tensorflow model with pytorch?

body: I am developing a model in tensorflow and find that it is good on my specific evaluation method. But when I transfer to pytorch, I can't achieve the same results. I have checked the model architecture, the weight init method, the lr schedule, the weight decay, momentum and epsilon used in BN layer, the optimizer, and the data preprocessing. All things are the same. But I can't get the same results as in tensorflow. Anybody have met the same problem?