Implementing this paper (ROCKET), I ended up with this code, which works.
The main issue I have with it is in summary How to run multiple pytorch layers in parallel, when they are of the same graph depth?
In the paper, 10000 convolutions with different (randomized) parameters are used on the same input to create 10000 corresponding feature maps, on which simple feature extractors are used.
Implementing this simplifies to
class MyModel(pl.LightningModule):
    def __init__(self):
        ...
        num_kernels = 10000
        for _ in range(num_kernels):
            self.convs[_] = nn.Conv1d(...)

    def forward(...):
          for i, layer in enumerate(self.convs):
              res = layer(data)

This just runs a Pythonic 10000-for-loop, each iteration runs its corresponding convolution.
How do I utilize Pytorch to run those 10000 convolutions efficiently, without waiting for each one to end before dispatching the next one?
This also generalizes for any non-fully-sequential Pytorch models.


title: How can I check the value in another python file when I debug this python file?

body:
I an running a trainer.py. Now I want to check one term of my loss function. But the loss function is define in a Class in another .py
The training section of maintraining.py is as the below,
for data in tqdm(train_loader):
    trainer.train_step(data)

While the trainer.train_step is defined in another .py, which is as the below:
class DVTrainer(Trainer):
      def __init__(.....)

      def _train_step(self, data):
          loss = self._compute_loss(data)
          self.opt.zero_grad()
          loss.backward()
          self.opt.step()
      def _compute_loss(self,data):
          x, y, _ = data
          p_z_given_x = self.encoder(x)
          z = p_z_given_x.rsample()
          q_x_given_z = self.decoder(z)
          KL_loss = p_z_given_x.log_prob(z) + self.prior.log_prob(z)

Now I run the maintraining.py, but I wanna check the value and size of KL_loss when I debug the maintraing.py. I'm searching for a long time on net.But no use. Please help or try to giver some ideas how to achieve this.
Thanks in advance!


title: How to fix and find the source of the exception: Exception: process 1 terminated with exit code 1 for multiple GPU training with DDP and Pytorch

body:
I've been trying to train a DDP (distributed data parallel model) for a few weeks now. I always get a different type of error which has made it really hard to debug. Some are not informative or cryptinc. This one is one of the cryptic ones:
('master_port', '38390')
('device', device(type='cuda'))
('jobid', '-1_pid_13421')
('slurm_jobid', -1)
('slurm_array_task_id', -1)
('condor_jobid', -1)
('hostname', 'vision-02.cs.illinois.edu')
('current_time', 'Apr02_16-39-21')
('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Apr02_16-39-21_jobid_-1_pid_13421/tb'))
('gpu_name', 'GeForce GTX TITAN X')
('PID', '13421')
('nccl', 2708)
torch.cuda.device_count()=4

opts.world_size=4

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn
-> rank=3
-> mp.current_process()=<SpawnProcess name='SpawnProcess-4' parent=13421 started>
-> os.getpid()=13439
----> setting up rank=3 (with world_size=4)
---> MASTER_ADDR='127.0.0.1'
---> 38390
---> backend='nccl'
-> rank=0
-> mp.current_process()=<SpawnProcess name='SpawnProcess-1' parent=13421 started>
-> os.getpid()=13435
----> setting up rank=0 (with world_size=4)
---> MASTER_ADDR='127.0.0.1'
---> 38390
---> backend='nccl'
-> rank=2
-> mp.current_process()=<SpawnProcess name='SpawnProcess-3' parent=13421 started>
-> os.getpid()=13437
----> setting up rank=2 (with world_size=4)
---> MASTER_ADDR='127.0.0.1'
---> 38390
---> backend='nccl'
-> rank=1
-> mp.current_process()=<SpawnProcess name='SpawnProcess-2' parent=13421 started>
-> os.getpid()=13436
----> setting up rank=1 (with world_size=4)
---> MASTER_ADDR='127.0.0.1'
---> 38390
---> backend='nccl'
----> done setting up rank=3
----> done setting up rank=2
----> done setting up rank=0
----> done setting up rank=1
setup process done for rank=1
setup process done for rank=0
setup process done for rank=3
setup process done for rank=2
dataset/loader type: train: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2afa24c72c70>, val: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2afa24c72c40>, test: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2afa24c72c40>,
dataset/loader type: train: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2b520bd9dc70>, val: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2b520bd9dc40>, test: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2b520bd9dc40>,
dataset/loader type: train: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2adea3982c70>, val: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2adea3982c40>, test: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2adea3982c40>,
dataset/loader type: train: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2b37c79adc70>, val: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2b37c79adc40>, test: <data_lib.dataloader_hash_dataset.HashDatasetFromFeatFile object at 0x2b37c79adc40>,
2021-04-02 16:39:39.827202: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
Starting training...

n_epoch=0
i=0
i=0: loss=tensor(11.2723, device='cuda:0', grad_fn=<NllLossBackward>)
i=1
i=1: loss=tensor(10.9752, device='cuda:0', grad_fn=<NllLossBackward>)
i=2
i=2: loss=tensor(10.7909, device='cuda:0', grad_fn=<NllLossBackward>)
i=3
i=3: loss=tensor(10.4955, device='cuda:0', grad_fn=<NllLossBackward>)
i=4
i=4: loss=tensor(10.2056, device='cuda:0', grad_fn=<NllLossBackward>)
i=5
i=5: loss=tensor(10.0622, device='cuda:0', grad_fn=<NllLossBackward>)
i=6
i=6: loss=tensor(9.6653, device='cuda:0', grad_fn=<NllLossBackward>)
i=7
i=7: loss=tensor(9.7871, device='cuda:0', grad_fn=<NllLossBackward>)
i=8
i=8: loss=tensor(9.5899, device='cuda:0', grad_fn=<NllLossBackward>)
i=9
i=9: loss=tensor(9.7369, device='cuda:0', grad_fn=<NllLossBackward>)
i=10
i=10: loss=tensor(8.8844, device='cuda:0', grad_fn=<NllLossBackward>)
Traceback (most recent call last):
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 286, in <module>
    main_distributed()
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 197, in main_distributed
    mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 110, in join
    raise Exception(
Exception: process 1 terminated with exit code 1
JOB ENDED

I think based on the process number that it happened inside the spawn. I say that becuase I have read:

Exception: process 0 terminated with exit code 1 when use torch.multiprocessing.spawn on GPUs
https://www.reddit.com/r/pytorch/comments/j7z6l6/exception_process_2_terminated_with_exit_code_1/
https://discuss.pytorch.org/t/exception-process-0-terminated-with-exit-code-1-error-when-using-torch-multiprocessing-spawn-to-parallelize-over-multiple-gpus/90636/21

and it seems that process 0 is for the main process. Since mine says process 1 I assume it's not coming form the main process. The issue is that this error doesn't point to anything in my code making it really hard to debug.
In the past I've had SIGKILL errors which I eventually - after a lot of work - discovered it was a memory segmentation fault (somehow my program ate all the memory). I doubt this is the case because my HPC tells me my memory allocation:
(metalearningpy1.7.1c10.2) miranda9~/ML4Coq $ cat main.sh.log5240
000 (5240.000.000) 04/02 16:39:03 Job submitted from host: <130.126.112.32:9618?addrs=130.126.112.32-9618+[--1]-9618&noUDP&sock=1423_a35b_4>
...
001 (5240.000.000) 04/02 16:39:03 Job executing on host: <172.22.224.112:9618?addrs=172.22.224.112-9618+[--1]-9618&noUDP&sock=2500_5edb_3>
...
006 (5240.000.000) 04/02 16:39:12 Image size of job updated: 4572
        5  -  MemoryUsage of job (MB)
        4572  -  ResidentSetSize of job (KB)
...
006 (5240.000.000) 04/02 16:40:53 Image size of job updated: 14981540
        5  -  MemoryUsage of job (MB)
        4572  -  ResidentSetSize of job (KB)
...
005 (5240.000.000) 04/02 16:40:53 Job terminated.
        (1) Normal termination (return value 0)
                Usr 0 00:05:04, Sys 0 00:00:51  -  Run Remote Usage
                Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
                Usr 0 00:05:04, Sys 0 00:00:51  -  Total Remote Usage
                Usr 0 00:00:00, Sys 0 00:00:00  -  Total Local Usage
        0  -  Run Bytes Sent By Job
        0  -  Run Bytes Received By Job
        0  -  Total Bytes Sent By Job
        0  -  Total Bytes Received By Job
        Partitionable Resources :    Usage  Request Allocated
           Cpus                 :                 8         8
           Disk (KB)            :        2        2   1417983
           Gpus                 :                 4         4
           Memory (MB)          :        5        1         1
...

Does anyone know how to even stard debugging this, what is the source of the or how to fix this?


title: How do I debug this? RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

body:
I am training a RL agent on the "CartPole-v0" environment from Gym. This is my net:
class DiscreteNet(nn.Module):
    """For discrete action space"""
    def __init__(self, obs_size, n_actions):
        super(DiscreteNet, self).__init__()

        self.net = nn.Sequential(
            nn.Linear(obs_size, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, n_actions),
            # Use a sigmoid output do that the values for every action are between 0 and 1
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.net(x)

Now I create the network and the environment:
env = gym.make("CartPole-v0")

n_obs = env.observation_space.shape[0]
n_actions = env.action_space.n

print(f"n_obs: {n_obs}, n_actions: {n_actions}")

# The neural network
net = DiscreteNet(n_obs, n_actions).to(device)

# The optimizer to optimize the network parameters
optimizer = optim.Adam(params=net.parameters(), lr=0.01)

And this is how I train it:
Episode = namedtuple("Episode", field_names=["reward", "steps"])
EpisodeStep = namedtuple("EpisodeStep", field_names=["obs_v", "action_v", "log_prob_v"])

for batch_iter_no in range(N_EPISODES * BATCH_SIZE):

    # New episode
    episode_reward = 0
    batch = []
    episode_reward = 0.0
    episode_steps = []
    obs = env.reset()

    # Obtain a batch of episodes
    while True:
        obs_v = torch.FloatTensor([obs]).to(device)
        act_v = net(obs_v).to(device)
        distribution = Categorical(act_v)
        action_v = distribution.sample().to(device)
        log_prob_v = distribution.log_prob(action_v).to(device)
        next_obs, reward, is_done, _ = env.step(action_v.item())
        episode_reward += reward
        episode_steps.append(EpisodeStep(obs_v=obs_v, action_v=action_v, log_prob_v=log_prob_v))
        if is_done:
            batch.append(Episode(reward=episode_reward, steps=episode_steps))
            episode_reward = 0
            episode_steps = []
            next_obs = env.reset()
            if len(batch) == BATCH_SIZE:
                break
        obs = next_obs

    # Filter the batch for elite episodes
    rewards = [e.reward for e in batch]
    reward_bound = np.percentile(rewards, PERCENTILE)
    reward_mean = float(np.mean(rewards))

    for episode_iter_no, e in enumerate(batch):
        if e.reward < reward_bound:
            continue

        # Train the NN on the elite episode
        cumulated_loss = 0
        episode_len = 0
        for s in e.steps:

            # Loss is: - log_prob
            loss_v = (- s.log_prob_v).to(device)

            # Logging variables
            cumulated_loss += loss_v.item()
            episode_len += 1

            # For some reason this next line is needed
            #loss_v.requires_grad = True

            optimizer.zero_grad()
            loss_v.backward()
            optimizer.step()

        # Print the logging info after each episode
        print(f"{batch_iter_no}, {episode_iter_no}: avg_loss={cumulated_loss/episode_len:.3f},"
              f"episode_reward={e.reward}, episode_boundary={reward_bound}")

However this throws the error below. I have tried looking at other questions that had this error, but do not quite understand the error message itself. How do I know which variables are needed for computing the gradients in my code? And which one is causing this error?
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-216-320c323f8526> in <module>
     55
     56             optimizer.zero_grad()
---> 57             loss_v.backward()
     58             optimizer.step()
     59

~/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222
    223     def register_hook(self, hook):

~/anaconda3/envs/sumo_rl/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    128         retain_graph = create_graph
    129
--> 130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
    132         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 2]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


title: pytorch index_put_ gives RuntimeError: the derivative for 'indices' is not implemented

body:
This is follow up question to this question. I tried using index_put_ as suggested in the answer, however I'm getting the following error
RuntimeError: the derivative for 'indices' is not implemented

I want to backpropagate the error gradients through the indices and hence I can't detach() it. Is there any way possible to achieve this?
Example:
import torch
image = torch.tensor([[246,  50, 101], [116,   1, 113], [187, 110,  64]])
iy = torch.tensor([[1, 0, 2], [1, 0, 2], [2, 2, 2]])
ix = torch.tensor([[0, 2, 1], [1, 2, 0], [0, 1, 2]])
warped_image = torch.zeros(size=image.shape)
warped_image.index_put_((iy, ix), image, accumulate=True)


title: How to use groups parameter in PyTorch conv2d function with batch?

body:
follow by the question in How to use groups parameter in PyTorch conv2d function
May I know if the input batch size = 4, for each batch it has independent filter to conv with it,  and I modify the code as follow,
import torch
import torch.nn.functional as F

filters = torch.autograd.Variable(torch.randn(3,4,3,3))
inputs = torch.autograd.Variable(torch.randn(4,3,10,10))
out = F.conv2d(inputs, filters, padding=1, groups=3)

I have another error
RuntimeError: Given groups=3, weight of size [3, 4, 3, 3], expected input[4, 3, 10, 10] to have 12 channels, but got 3 channels instead
How to solve it?


title: PyTorch - How to use Avg 2d Pooling as a dataset transform?

body:
In Pytorch, I have a dataset of 2D images (or alternatively, 1 channel images) and I'd like to apply average 2D pooling as a transform. How do I do this? The following does not work:
    omniglot_dataset = torchvision.datasets.Omniglot(
        root=data_dir,
        download=True,
        transform=torchvision.transforms.Compose([
            torchvision.transforms.ToTensor(),
            torchvision.transforms.CenterCrop((80, 80)),
            # torchvision.transforms.Resize((10, 10))
            torch.nn.functional.avg_pool2d(kernel_size=3, strides=1),
        ])
    )


title: problem with torch_scatter: undefined symbol

body:
When running import torch_scatter on a GPU, I met the following error:
OSError: /scratch/ssd001/home/.local/lib/python3.6/site-packages/torch_scatter/_version_cuda.so: undefined symbol: _ZN3c106detail12infer_schema20make_function_schemaENS_8ArrayRefINS1_11ArgumentDefEEES4_

Does anyone know how to fix this OSError? Thank you! My python version is 3.6 with CUDA 10.2 and pytorch 1.7.1.


title: question about the attention calculation in Pytorch seq2seq tutorial: discrepancy with original Badahnau or Luong paper

body:
I am studying the attention recent. I have some doubt about they calculate the attention is Pytorch NLP attention tutorial: https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html.
In the tutorial, they calculate the score or weight using the decoder’s input and decoder's hidden state. However I find out neither Luong or Badahnau do that why. Instead both use the decoder hidden state and the ENCODER output the calculate the weight.
Why Pytorch tutorial do that way?


title: How to make new weight parameter in pytorch?

body:
In this SS_wrapper class, I have to calculate new weight parameter.
In the reset_weights(self) function, I created a new weight (weight to be used in the backward) via torch.nn.init.uniform_(self.fixed_fb_weights). But, I don't know how to do this with forward weight sign and multiplication. I'm curious how to make self.sign * self.fixed_fb_weights do by putting them inside self.fixed_fb_weights and worked.
Now, the result of multiplying the sign is not shown in self.fixed_fb_weights, but only torch.nn.init.uniform_. How to fix it?


title: Deep learning model test accuracy unstable

body:
I am trying to train and test a pytorch GCN model that is supposed to identify person. But the test accuracy is quite jumpy like it gives 49% at 23 epoch then goes below near 45% at 41 epoch. So it's not increasing all the time though loss seems to decrease at every epoch.
My question is not about implementation errors rather I want to know why this happens. I don't think there is something wrong in my coding as I saw SOTA architecture has this type of behavior as well. The author just picked the best result and published saying that their models gives that result.
Is it normal for the accuracy to be jumpy (up-down) and am I just to take the best ever weights that produce that?


title: pytorch Dataloader - if input data returns multiple training instances

body:
Problem
I have the following problem:
I want to use pytorchs DataLoader (in a similar way like here) but my setup varies a bit:
In my datafolder I have images (lets call them image_total of different street situations and I want to use cropped images (called image_crop_[idx] around persons that are close enough to the camera. So it can happen that some images give me one or more cropped images while others give me zero images as they do not show any person or they are to far away.
As I have a lot of images I want to make the implementation as efficient as possible.
My hope is that it is possible to use something like this:
I want to load the image_total and check if useful crops are in it. If so I extract the cropped images and get a list like [image_crop_0, image_crop_1, image_crop_2,...]
Now my question: Is this possible to be compatible with pytorchs DataLoader? The problem I see is that ````getitem```-method of my class would return zero to arbitrary instances. I want to use a constant batch-size for training.
Considerations

maybe DataLoader supports this (and I did not find it)
I have to work with a buffer or something similar
the fallback would be to pre process the data, but this would not be the most efficient solution


title: torch model evaluate from string

body:
I do evaluation of my torch model loading a batch from text files:
def batchify(data, bsz):
    nbatch = data.size(0)
    data = data.narrow(0, 0, nbatch * bsz)
    data = data.view(bsz, -1)
    return data

def load_file(path, vocab, direction):
    lines = open(path).readlines()
    data = list(''.join(lines))
    idx = vocab['char'].map(data)
    if direction == 'backward': idx = idx[::-1]
    return torch.tensor(idx)

def load_data(path, vocab, direction):
    data = load_file(path, vocab, direction)
    yield data

This works ok:
eval_file_or_dir = os.path.join(BASE_PATH,'shakespeare.txt')
data = load_data(eval_file, vocab, direction)
if isinstance(data, GeneratorType):
    data = list(data)
    data = data[0]
batches = batchify(data, batch_size)

and I get torch.Size([100, 6])
Now, I want to load data from a string, so I wrote
def load_text(text, vocab, direction):
    buf = io.StringIO(text)
    lines = buf.readlines()
    data = list(''.join(lines))
    idx = vocab['char'].map(data)
    if direction == 'backward': idx = idx[::-1]
    yield torch.tensor(idx)

but it does not work as expected:
data = load_text(text, vocab, direction)
if isinstance(data, GeneratorType):
    data = list(data)
    data = data[0]
batches = batchify(data, batch_size)
print(batches, batches.size())

and I get a empty tensor:
tensor([], size=(100, 0), dtype=torch.int64) torch.Size([100, 0])


title: BertForTokenClassification Has Extra Output

body: I am using PyTorch's BertForTokenClassification pretrained model to do custom word tagging (not NER or POS, but essentially the same). There are 20 different possible tags (using BIO scheme): 9 B's, 9 I's, and an O. Despite there being 19 possible tags, the feed-forward layer that is added on top of BERT has 20 tags. I have used other datasets, too, and the result is the same: there is always one more output than the number of classes. Can anyone tell me why this is?


title: How to use tf.broadcast_to() in a Keras model?

body:
I am creating a keras model using its functional API, I want to use the tensorflow broadcast operation, as (unsqueeze(-1).expand_as(x)) in Pytorch:
x1 = tf.broadcast_to(x1, shape)
Where x1 is a tensor of shape (None, 44, 1, 1), and shape = (None, 44, 44, 1).
However, this operation is not possible since in the keras functional API, we need to work with Input()which generates symbolic tensors.
So, I get the error:
ValueError: Tried to convert 'shape' to a tensor and failed. Error: Cannot convert a partially known TensorShape to a Tensor: (None, 44, 44, 1)
Is there a proper way to use the broadcast operation for a symbolic tensor inside a keras functional model?


title: TypeError: img should be PIL Image. Got  even though using latest pytorch version

body:
I had to create a new pytorch environment in Anaconda, I created my code and it worked fine in my old environment. I then created the new environment with the same version of pytorch and cuda 10.1 (but then updated to cuda 11, same as the old one. WHen i try to run the same code i get the error:
TypeError: img should be PIL Image. Got <class 'torch.Tensor'>
When trying to apply any transformation to my tensors, for example the following code gives me an error:
def randRoll(batch, deg):
rotator = torchvision.transforms.RandomRotation(deg)
batch = rotator(batch)
return batch

Nothing has changed and I can't understand why i would get this.
Any suggestions?


title: Loading Pytorch model doesn't seem to work

body:
I want to load a pretrained model from PyTorch. Specifically, I want to run the SAT-Speaker-with-emotion-grounding (431MB) model from this repo. However, I don't seem to be able to load it. When I download the model and run the script below, I get a dictionary and not the model.
Loading the model:
model_emo = torch_load_model('best_model.pt', map_location=torch.device('cpu'))

Running the model
model_emo(image)

The error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-e80ccbe8b6ed> in <module>
----> 1 model_emo(image)

TypeError: 'dict' object is not callable

Now, the docs say that I should instantiate the model class and then load the checkpoint data. However, I don't know what model class this belongs to, and the documents don't say. Does anyone have any advice on how to proceed with this issue? Thanks.


title: How can one define a single parameter that is shared among layers of a neural network?

body:
Suppose we have the following fully connected network:
class FC(nn.Module):
  def __init__(self, imgsz=28, num_classes=10):
      super(FC, self).__init__()
      self.imgsz = imgsz
      self.fc1 = nn.Linear(imgsz*imgsz, 300)
      self.fc2 = nn.Linear(300, 100)
      self.fc3 = nn.Linear(100, num_classes)
      self.relu = nn.ReLU(inplace=True)

  def forward(self, x):
      x = x.view(-1, self.imgsz*self.imgsz)
      x = self.relu(self.fc1(x))
      x = self.relu(self.fc2(x))
      x = self.fc3(x)

      return x

How can we define a single parameter using nn.Parameter() or any other methods that is shared among all layers?


title: How to deal with the unlabeled nodes in Pytorch Geometric?

body:
I have a dataset on my own, and the dataset contains two classes, let's say 0 and 1. Besides, there is a large part of nodes which class is unlabeled. My goal is to predict these unlabeled nodes using GCN. But I am confused about how to deal with these unlabeled nodes in Pytorch Geometric.
As far as I can think about, I can label the nodes into 3 classes, 0, 1 and unknown. But if I do it this way, that means I am trying to classify the dataset into three classes? (But that's not what I want since unknown is not a class).
And another way to deal with these node is to ignore them, simply run PyG on the labeled node. But in this way, it seems that these unlabeled node(with feature) is useless in the dataset?


title: Use Adam optimizer for LSTM network vs LBGFS

body:
I have modified pytorch tutorial on LSTM (sine-wave prediction: given [0:N] sine-values -> [N:2N] values) to use Adam optimizer instead of LBFGS optimizer. However, the model does not train well and cannot predict sine-wave correctly. Since in most cases we use Adam optimizer for RNN training, I wonder how this issue can be resolved. I also wonder if the code segment regarding sequence-in-sequence-out (done with a loop: for input_t in input.split(1, dim=1)), can be done by a pytorch module or function.
from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import matplotlib
#matplotlib.use('Agg')
import matplotlib.pyplot as plt

class Sequence(nn.Module):
    def __init__(self):
        super(Sequence, self).__init__()
        self.lstm1 = nn.LSTMCell(1, 51)
        self.lstm2 = nn.LSTMCell(51, 51)
        self.linear = nn.Linear(51, 1)

    def forward(self, input, future = 0):
        outputs = []
        h_t = torch.zeros(input.size(0), 51, dtype=torch.double)
        c_t = torch.zeros(input.size(0), 51, dtype=torch.double)
        h_t2 = torch.zeros(input.size(0), 51, dtype=torch.double)
        c_t2 = torch.zeros(input.size(0), 51, dtype=torch.double)

        for input_t in input.split(1, dim=1):
            h_t, c_t = self.lstm1(input_t, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
            output = self.linear(h_t2)
            outputs += [output]
        for i in range(future):# if we should predict the future
            h_t, c_t = self.lstm1(output, (h_t, c_t))
            h_t2, c_t2 = self.lstm2(h_t, (h_t2, c_t2))
            output = self.linear(h_t2)
            outputs += [output]
        outputs = torch.cat(outputs, dim=1)
        return outputs

if __name__ == '__main__':
    # set random seed to 0
    np.random.seed(0)
    torch.manual_seed(0)
    # load data and make training set
    data = torch.load('traindata.pt')
    input = torch.from_numpy(data[3:, :-1])
    target = torch.from_numpy(data[3:, 1:])
    test_input = torch.from_numpy(data[:3, :-1])
    test_target = torch.from_numpy(data[:3, 1:])
    print("input.size", input.size())
    print("target.size", target.size())
    # build the model
    seq = Sequence()
    seq.double()
    criterion = nn.MSELoss()
    # use LBFGS as optimizer since we can load the whole data to train
    optimizer = optim.Adam(seq.parameters(), lr=0.005)
    #begin to train
    for i in range(15):
        print('STEP: ', i)
        seq.train()
        def run1step():
            optimizer.zero_grad()
            out = seq(input)
            loss = criterion(out, target)
            print('train loss:', loss.item())
            loss.backward()
            return loss
        run1step()
        optimizer.step()
        # begin to predict, no need to track gradient here
        seq.eval()
        with torch.no_grad():
            future = 1000
            pred = seq(test_input, future=future)
            loss = criterion(pred[:, :-future], test_target)
            print('test  loss:', loss.item())
            y = pred.detach().numpy()
        # draw the result
        def draw(yi, color):
            plt.figure(figsize=(30,10))
            plt.title('Predict future values for time sequences\n(Dashlines are predicted values)', fontsize=30)
            plt.xlabel('x', fontsize=20)
            plt.ylabel('y', fontsize=20)
            plt.xticks(fontsize=20)
            plt.yticks(fontsize=20)
            plt.plot(np.arange(input.size(1)), yi[:input.size(1)], color, linewidth = 2.0)
            plt.plot(np.arange(input.size(1), input.size(1) + future), yi[input.size(1):], color + ':', linewidth = 2.0)
            plt.show()
        if i == 14:
          draw(y[0], 'r')
          draw(y[1], 'g')
          draw(y[2], 'b')
          plt.savefig('predict_LSTM%d.pdf'%i)
          #plt.close()


title: Difference in using LSTM for fixed sized input and variable size input

body:
I wonder how I can train (natural) LSTM model where I just feed a single time step one by one in one loop using all data points, and not just of a sample with a fixed time lenght of history data points.
And, what might be the difference and why is it standard to use a fixed lenght whereas the using all point step by step is the more  intuitive way?
Beside my theoretical question, is there an good example using a loop for single time steps instead of feeding the whole window to a LSTM?


title: How to set constraints on weights based on other weights in the same layer [Pytorch]

body: As a simple example, take a neural network with 4 input nodes and 1 output node. I would like to take the 4 corresponding weights and set the constraint W1 < W2 < W3 < W4. Is this possible in Pytorch?


title: PyTorch: two binary masks union?

body:
I have two binary masks of shape (batch_size, width, heigh) that I want to create a binary mask which indicates the union of elements between the two.
To find the intersection, I can use torch.where(A == B, 1, 0), but how can I find the union?


title: How to use torch custom dataset with fastai data loaders

body:
I created a custom torch dataset and specified 2 required methods: __getitem__ and __len__. Then I created two torch data loaders:
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=2)
val_loader = torch.utils.data.DataLoader(val_dataset, batch_size=32, shuffle=True, num_workers=2)

and I wanted to use them with fastai, so I tried to run:
from fastai.vision.data import DataLoaders
dls = vision.data.DataLoaders(train_loader, valid_loader)
from fastai.vision.learner import cnn_learner, error_rate
learner_original = cnn_learner(dls, models.resnet34, metrics=error_rate, pretrained=True)

However, this gives me an error:

AttributeError: 'DataLoader' object has no attribute 'after_batch'

What's wrong with my setup? Should torch dataset have another attribute to make it compatible with fastai DataLoaders?


When I Processed video and its audio，I encountered an error:

Original Traceback (most recent call last):
  File "/home/yzx/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/yzx/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yzx/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yzx/lunwen/PseudoBinaural_CVPR2021-master/data/Augment_dataset.py", line 66, in __getitem__
    data_ret, data_ret_sep = self._get_pseudo_item(index)
  File "/home/yzx/lunwen/PseudoBinaural_CVPR2021-master/data/Pseudo_dataset.py", line 263, in _get_pseudo_item
    stereo = self.construct_stereo_ambi(pst_sources)
  File "/home/yzx/lunwen/PseudoBinaural_CVPR2021-master/data/Pseudo_dataset.py", line 120, in construct_stereo_ambi
    signals = np.stack([src.signal for src in pst_sources], axis=1) # signals shape: [Len, n_signals]
  File "", line 5, in stack
  File "/home/yzx/anaconda3/envs/pytorch/lib/python3.8/site-packages/numpy/core/shape_base.py", line 423, in stack
    raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack

The list_samlple_file below is the content of train.txt. Every line has the same form as the first line "/home/yzx/lunwen/datasets/FAIR-PLAY/binaural_audios/000383.wav,/home/yzx/lunwen/datasets/FAIR-PLAY/frames/000383". The former path is the path of each audio,and the latter is the folder of the frames of each video.I extracted 10 frames per second in each 10 seconds video.They are splited to 'audio_file' and 'img_folder' in Psedo_dataset.py.I checked this path is true.But I don't know why the error generate.I need your help.
The content of clss Augment_dataset.py is:
class AugmentDataset(StereoDataset, PseudoDataset):
    def __init__(self, opt, list_sample_file):
        self.opt = opt
        PseudoDataset.__init__(self, opt, list_sample_file)
        StereoDataset.__init__(self, opt)

        if "MUSIC" in self.opt.datalist:
            dup_times = 1
            print("dup_times:", dup_times)
        else:
            dup_times = 2
            print("dup_times:", dup_times)
        self.total_samples *= dup_times # in order to align with the length of FAIR-Play dataset

        random.shuffle(self.audios)
        random.shuffle(self.total_samples)

    def __getitem__(self, index):
        if random.uniform(0,1) < self.opt.pseudo_ratio:
            data_ret, data_ret_sep = self._get_pseudo_item(index)
        else:
            data_ret = self._get_stereo_item(index)
            _, data_ret_sep = self._get_pseudo_item(index)
        data_ret.update(data_ret_sep)

        return data_ret

    def __len__(self):
        return min(len(self.audios), len(self.total_samples))

    def name(self):
        return 'AugmentDataset'

The content of Psedo_dataset.py is:
def audio_normalize(samples, desired_rms = 0.1, eps = 1e-4):
    rms = np.maximum(eps, np.sqrt(np.mean(samples**2)))
    samples = samples * (desired_rms / rms)
    return rms / desired_rms, samples

def generate_spectrogram(audio):
    spectro = librosa.core.stft(audio, n_fft=512, hop_length=160, win_length=400, center=True)
    real = np.expand_dims(np.real(spectro), axis=0)
    imag = np.expand_dims(np.imag(spectro), axis=0)
    spectro_two_channel = np.concatenate((real, imag), axis=0)
    return spectro_two_channel

def process_image(image, augment, square=False):
    if square:
        iH, iW = 240, 240
        H, W = 224, 224
    else:
        iH, iW = 240, 480
        H, W = 224, 448
    image = mmcv.imresize(image, (iW,iH))
    h,w,_ = image.shape
    w_offset = w - W
    h_offset = h - H
    left = random.randrange(0, w_offset + 1)
    upper = random.randrange(0, h_offset + 1)
    image = mmcv.imcrop(image, np.array([left, upper, left+W-1, upper+H-1]))

    if augment:
        enhancer = ImageEnhance.Brightness(Image.fromarray(image))
        image = enhancer.enhance(random.random()*0.6 + 0.7)
        enhancer = ImageEnhance.Color(image)
        image = enhancer.enhance(random.random()*0.6 + 0.7)
    return image

class PseudoDataset(data.Dataset):
    def __init__(self, opt, list_sample_file):
        super().__init__()

        self.opt = opt
        self.total_samples = mmcv.list_from_file(list_sample_file)

        normalize = transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
        )
        vision_transform_list = [transforms.ToTensor(), normalize]
        self.vision_transform = transforms.Compose(vision_transform_list)

        random.seed(self.opt.seed)

        # load background, just one large-sizeimage
        self.bkg_img = mmcv.imread('./data/bkg.png')

        # build binauralizer
        hrtf_dir = "./data/subject03"
        # binauralizer = SourceBinauralizer(use_hrtfs=True, cipic_dir=hrtf_dir)
        self.hrir_db = CIPIC_HRIR(hrtf_dir)
        # encode to ambisonics, and then stereo
        speakers_phi = (2. * np.arange(2*4) / float(2*4) - 1.) * np.pi
        self.speakers_pos = [Position(phi, 0, 1, 'polar') for phi in speakers_phi]
        self.sph_mat = spherical_harmonics_matrix(self.speakers_pos, max_order=1) # shape: [N_array_speakers, 4]

        # parameter of loading audio
        self.exp_audio_len = int(self.opt.audio_length * self.opt.audio_sampling_rate)
        #print("Now load {} box info".format(self.opt.mode))
        #self.box_info = mmcv.load('./dataset/ASMR/scripts/results/{}_box_info.pkl'.format(self.opt.mode))

        if opt.fov == '1/3':
            self.fov = 1/3.
        elif opt.fov == '1/2':
            self.fov = 1/2.
        elif opt.fov == '5/6':
            self.fov = 5/6.
        elif opt.fov == '1':
            self.fov = 1.
        else:
            self.fov = 2/3.

        #self.categories = ['acoustic_guitar', 'banjo', 'bass', 'cello', 'drum', 'harp', 'piano', 'trumpet', 'ukelele']

    def construct_stereo_direct(self, pst_sources):
        stereo = np.zeros((2, self.exp_audio_len))
        for src in pst_sources:
            left_hrir, right_hrir = self.hrir_db.get_closest(src.position)[1:]
            left_signal = np.convolve(src.signal, np.flip(left_hrir, axis=0), 'valid')
            right_signal = np.convolve(src.signal, np.flip(right_hrir, axis=0), 'valid')

            n_valid, i_start = left_signal.shape[0], left_hrir.shape[0] - 1
            stereo[0, i_start:(i_start + n_valid)] += left_signal
            stereo[1, i_start:(i_start + n_valid)] += right_signal

        return stereo

    def construct_stereo_ambi(self, pst_sources):
        # encode to ambisonics
        Y = spherical_harmonics_matrix([src.position for src in pst_sources], max_order=1) # Y shape: [n_signals, 4]
        signals = np.stack([src.signal for src in pst_sources], axis=1) # signals shape: [Len, n_signals]
        ambisonic = np.dot(signals, Y) # shape: [Len, 4]

        array_speakers_sound = np.dot(ambisonic, self.sph_mat.T)
        #array_speakers_sound = np.dot(ambisonic, np.linalg.pinv(self.sph_mat))
        array_sources = [PositionalSource(array_speakers_sound[:, i], speaker_pos, \
            self.opt.audio_sampling_rate) for i, speaker_pos in enumerate(self.speakers_pos)]

        return self.construct_stereo_direct(array_sources)

    def construct_stereo_ambi_direct(self, pst_sources):
        # encode to ambisonics
        Y = spherical_harmonics_matrix([src.position for src in pst_sources], max_order=1)
        signals = np.stack([src.signal for src in pst_sources], axis=1)
        ambisonic = np.dot(signals, Y) # shape: [Len, 4]

        stereo = np.stack((
            ambisonic[:, 0] / 2 + ambisonic[:, 1] / 2,
            ambisonic[:, 0] / 2 - ambisonic[:, 1] / 2
        ))

        return stereo

    def _get_pseudo_item(self, index):
        # ensure the number of audios in a scene
        N = np.random.choice([1,2,3], p=[0.4, 0.5, 0.1])
        chosen_samples = [self.total_samples[index]]
        # avoid repeat sample
        for _ in range(1, N):
            while True:
                new_sample = random.choice(self.total_samples)
                if new_sample not in chosen_samples:
                    chosen_samples.append(new_sample)
                    break

        audio_margin = 0
        init_H = 360
        init_W = 640
        pst_sources = []

        if self.opt.not_use_background:
            cur_bkg_img = np.zeros((init_H, init_W, 3)).astype(np.uint8)
        else:
            # crop background img, exp_shape: [init_H, init_W, 3]
            bkg_start_x = np.random.randint(low=0, high=self.bkg_img.shape[1] - init_W)
            bkg_start_y = np.random.randint(low=0, high=self.bkg_img.shape[0] - init_H)
            cur_bkg_img = mmcv.imcrop(self.bkg_img.copy(),
                np.array([bkg_start_x, bkg_start_y, bkg_start_x+init_W-1, bkg_start_y+init_H-1]))
            #H_bkg, W_bkg, _ = cur_bkg_img.shape

        corner_record = []
        patch_size_record = []
        center_x_record = []
        audio_list = []
        patch_list = []
        actual_N = 0
        #load audio
        for idx, chosen_sample in enumerate(chosen_samples):
            audio_file, img_folder = chosen_sample.split(',')
            # audio part
            audio, audio_rate = librosa.load(audio_file, sr=self.opt.audio_sampling_rate, mono=True)
            #randomly get a start time for the audio segment from the original clip
            audio_len = len(audio) / audio_rate
            assert audio_len - self.opt.audio_length - audio_margin > audio_margin
            audio_start_time = random.uniform(audio_margin, audio_len - self.opt.audio_length - audio_margin)
            audio_end_time = audio_start_time + self.opt.audio_length
            audio_start = int(audio_start_time * self.opt.audio_sampling_rate)
            audio_end = audio_start + self.exp_audio_len
            audio = audio[audio_start:audio_end]
            if self.opt.audio_normal:
                normalizer0, audio = audio_normalize(audio)

            # video part
            # load img **patches**, copy bkg_img and construct a new image
            # load accurate frame
            cur_img_list = natsort.natsorted(glob(osp.join(img_folder, '*.jpg')))
            # get the closest frame to the audio segment
            frame_idx = (audio_start_time + audio_end_time) / 2 * 10
            frame_idx = int(np.clip(frame_idx, 0, len(cur_img_list) - 1))
            img_file = cur_img_list[frame_idx]
            img_patch = mmcv.imread(img_file)

            if self.opt.patch_resize:
                h_patch, w_patch, _= img_patch.shape
                resize_ratio = min(1/normalizer0, init_H / h_patch, init_W / 2 / w_patch)
                img_patch = mmcv.imrescale(img_patch, resize_ratio * random.uniform(0.8, 1))
            H_new, W_new, _ = img_patch.shape

            # just consider the overlap in the horizontal axis
            occupy_matrix = np.ones((init_W))
            # avoid cross border in x dim
            occupy_matrix[:(-W_new + 1)] = 0
            # avoid overlap
            for last_corner_x, W_last in zip(corner_record, patch_size_record):
                occupy_x = max(0, last_corner_x - W_new)
                occupy_matrix[occupy_x : last_corner_x + W_last] = 1

            # random sample position for this mono audio
            free_x_positions = np.where(occupy_matrix == 0)[0]
            if len(free_x_positions) < 2:
                break
            actual_N += 1
            corner_x = random.choice(free_x_positions)
            corner_record.append(corner_x)
            patch_size_record.append(W_new)
            corner_y = random.randint(0, init_H - H_new)

            center_y = corner_y + H_new // 2
            center_x = corner_x + W_new // 2
            center_x_record.append(center_x)
            azimuth = (init_W // 2 - center_x) / init_W * pi * self.fov
            elevation = (init_H // 2 - center_y) / init_H * pi / 2

            if self.opt.visualize_data:
                output_dir = 'others/dataset_visual/{:d}_{:d}'.format(N, index)
                if not osp.exists(output_dir):
                    os.mkdir(output_dir)
                if librosa.__version__ >= '0.8.0':
                    import soundfile as sf
                    sf.write(osp.join(output_dir, '{:d}.wav'.format(idx)), audio.transpose(), audio_rate)
                else:
                    librosa.output.write_wav(osp.join(output_dir, '{:d}.wav'.format(idx)), audio, sr=audio_rate)
            audio_list.append(audio)
            patch_list.append(img_file)
            pst_sources.append(PositionalSource(audio, Position(azimuth, elevation, 3, 'polar'), audio_rate))
            pdb.set_trace()

            if self.opt.blending:
                center = (center_x, center_y)
                mask = 255 * np.ones(img_patch.shape, img_patch.dtype)
                cur_bkg_img = cv2.seamlessClone(img_patch, cur_bkg_img, mask, center, cv2.NORMAL_CLONE)
            else:
                patch_in_start_x = corner_x
                patch_in_start_y = corner_y
                assert patch_in_start_x >= 0
                assert patch_in_start_y >= 0
                cur_bkg_img[patch_in_start_y : patch_in_start_y + H_new, patch_in_start_x : patch_in_start_x + W_new] = img_patch

        if self.opt.stereo_mode == 'direct':
            #print("use direct")
            stereo = self.construct_stereo_direct(pst_sources)
        elif self.opt.stereo_mode == 'ambisonic':
            #print("use ambisonic")
            stereo = self.construct_stereo_ambi(pst_sources)
        elif self.opt.stereo_mode == 'ambidirect':
            #print("use ambidirect")
            stereo = self.construct_stereo_ambi_direct(pst_sources)
        else:
            raise ValueError("please choose right stereo mode")

        normalizer, _ = audio_normalize(stereo[0] + stereo[1])
        stereo = stereo / normalizer
        audio_channel1, audio_channel2 = stereo
        frame = cur_bkg_img
        if self.opt.visualize_data:
            output_dir = 'others/dataset_visual/{:d}_{:d}'.format(N, index)
            if librosa.__version__ >= '0.8.0':
                import soundfile as sf
                sf.write(osp.join(output_dir, 'input_binaural.wav'), stereo.transpose(), audio_rate)
            else:
                librosa.output.write_wav(osp.join(output_dir, 'input_binaural.wav'), stereo, sr=audio_rate)
            mmcv.imwrite(frame, osp.join(output_dir, 'reference.jpg'))
        frame = process_image(frame, self.opt.enable_data_augmentation)
        frame = self.vision_transform(frame)

        #passing the spectrogram of the difference
        audio_diff_spec = torch.FloatTensor(generate_spectrogram(audio_channel1 - audio_channel2))
        audio_mix_spec = torch.FloatTensor(generate_spectrogram(audio_channel1 + audio_channel2))

        data_ret = {'frame': frame, 'audio_diff_spec':audio_diff_spec, 'audio_mix_spec':audio_mix_spec}

        # incorporate separation part
        assert len(patch_list) == len(audio_list)
        left_channel = np.zeros(self.exp_audio_len).astype(np.float32)
        right_channel = np.zeros(self.exp_audio_len).astype(np.float32)
        if len(audio_list) >= 2:
            for cur_audio, center_x in zip(audio_list, center_x_record):
                if center_x < init_W // 3:
                    left_channel += cur_audio
                elif center_x > init_W // 3:
                    right_channel += cur_audio
                else:
                    left_channel += cur_audio
                    right_channel += cur_audio

            _, left_channel = audio_normalize(left_channel)
            _, right_channel = audio_normalize(right_channel)
        else:
            left_channel = audio_channel1
            right_channel = audio_channel2

        if self.opt.visualize_data:
            output_dir = 'others/dataset_visual/{:d}_{:d}'.format(N, index)
            if librosa.__version__ >= '0.8.0':
                import soundfile as sf
                sf.write(osp.join(output_dir, 'gt_left.wav'), left_channel, audio_rate)
                sf.write(osp.join(output_dir, 'gt_right.wav'), right_channel, audio_rate)
            else:
                librosa.output.write_wav(osp.join(output_dir, 'gt_left.wav'), left_channel, audio_rate)
                librosa.output.write_wav(osp.join(output_dir, 'gt_right.wav'), right_channel, audio_rate)

        sep_mix_spec = audio_mix_spec
        sep_diff_spec = torch.FloatTensor(generate_spectrogram(left_channel - right_channel))
        frame_sep_list = frame

        if self.opt.mode == 'train':
            data_ret_sep = {'frame_sep': frame_sep_list, 'sep_diff_spec': sep_diff_spec, 'sep_mix_spec': sep_mix_spec}
        else:
            data_ret_sep = {'frame_sep': frame_sep_list, 'sep_diff_spec': sep_diff_spec, 'sep_mix_spec': sep_mix_spec, 'left_audio': left_channel, 'right_audio': right_channel}

        return data_ret, data_ret_sep

    def __getitem__(self, index):
        data_ret, data_ret_sep = self._get_pseudo_item(index)
        if self.opt.dataset_mode == 'Pseudo_stereo':
            return data_ret
        elif self.opt.dataset_mode == 'Pseudo_sep':
            return data_ret_sep
        else:
            data_ret.update(data_ret_sep)
            return data_ret

    def __len__(self):
        return len(self.total_samples)

    def name(self):
        return 'PseudoDataset'

    def initialize(self, opt):
        pass


title: How to include correlation among parameter vectors in the cost function?

body:
I have the following model:
class Model(nn.Module):
    def __init__(self, dim_in, lambda_=.3):
        super(FeatExtractorGR, self).__init__()

       '''The linear transform layer'''
        self.lt = nn.Linear(in_features = dim_in, out_features = 10, bias=True)

    '''The encoder'''
    self.feature_extractor = \
        nn.Sequential(
            #nn.Linear(in_features = 10, out_features = 30, bias=True),
            nn.BatchNorm1d(10),
            nn.ReLU(),
            nn.Linear(in_features = 10, out_features = 20, bias=True),
            nn.BatchNorm1d(20),
            nn.ReLU(),
            nn.Linear(in_features = 20, out_features = 10, bias=True),
            nn.BatchNorm1d(10),
            nn.ReLU()
        )

def forward(self, x):
    transformed = self.lt(x)
    return self.feature_extractor(transformed)

I want to force the weight vectors of the linear transformation layer to be uncorrelated. I tried to include the dot products among the vectors in the cost function:
        params=list(model.lt.parameters())[0]
        dotprod=torch.tensordot(params, params, dims=([1],[1])).abs().fill_diagonal_(0).sum()/2

        loss = other_losses + dotprod * weight

But this is not working, even with really high weight. The weight vectors from the lt layer are still highly correlated. I have also tried to remove other_losses, but no effect. What am I doing wrong?


title: Use other source to download from than pip in PythonVirtualenvOperator

body:
Say I'm using the PythonVirtualenvOperator and have PyTorch as a requirement. When calling "pip freeze" I get
#requirements.txt
.
.
torch==1.8.1+cpu

and defining my task as
#tasks.py
from airflow.operators.python import PythonVirtualenvOperator

t1= PythonVirtualenvOperator(
        task_id = "test",
        python_version = "3.7",
        python_callable = test_func,
        requirements = ["torch==1.8.1+cpu"]
    )

throws the ERROR: Could not find a version that satisfies the requirement torch==1.8.1+cpu.
In the documentation from PyTorch we install it by pip3 install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html i.e downloading it from their webpage and not from pip (if I understand it correctly), which might be why pip fails in the venv. Thus I would like to make the venv (created by airflow for the PythonVirtualOperator) to download torch from the link specified above, instead of pip.
Is that doable? And is there a difference between torch==1.8.1+cpu and just torch==1.8.1 when using the cpu i.e does it make a difference if I just remove the +cpu?


title: How to extract tensors to numpy arrays or lists from a larger pytorch tensor

body:
I have a list of pytorch tensors as shown below:
data = [[tensor([0, 0, 0]), tensor([1, 2, 3])],
        [tensor([0, 0, 0]), tensor([4, 5, 6])]]

Now this is just a sample data, the actual one is quite large but the structure is similar.
Question: I want to extract the tensor([1, 2, 3]), tensor([4, 5, 6]) i.e., the index 1 tensors from data to either a numpy array or a list in flattened form.
Expected Output:
out = array([1, 2, 3, 4, 5, 6])

OR
out = [1, 2, 3, 4, 5, 6]

I have tried several ways one including map function like:
map(lambda x: x[1].numpy(), data)

This gives:
[array([1, 2, 3]),
 array([4, 5, 6])]

And I'm unable to get the desired result with any other method I'm using. Any help would be appreciated.
Thanks in advance:)


title: easyOCR allocates GPU even when on gpu=False

body:
I need to use multiple (my) TF CNN models for a rather complex task. I also need to use easyOCR (PyTorch) every now and then. However easyOCR usage is pretty rare and the task is very small in comparison to TF models inference. Therefore I use gpu=False in easyocr.Reader constructor. Nevertheless, as soon as easyocr predicts anything, GPU is allocated for for pyTorch (This is a known bug, I already checked easyOCR's github issues) and any TF model throws error:
Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so tr
y looking to see if a warning log message was printed above.

If I predict with any TF model first, the easyocr model throws:
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278

I have found a workaround, but it seems rather dangerous to put something like this into production.
Is there a more safe way of achieving this?


title: How to use two Pytorch models together

body: Is it possible to have two Pytorch models and run them at the same time or combine them? For example, lets say I have two Pytorch models, one that detects cars and another that detects people. Can I use both models at the same time to detect objects from my webcam?


title: How to construct CNN with 400 nodes hidden layer using PyTorch?

body:
I would like to create Convolution layer with 400 nodes using PyTorch like below with condition as fully connected linear layer with 400 hidden neurons/ output is 10, flatten image to a vector for input, and use ReLU function.
When I print out x.shape it returns like torch.Size([1024, 300]) and  torch.Size([1024, 10]), and my very first layer is torch.Size([100, 3, 32, 32]). I am confused how to construct this simple CNN and what I am missing.
class MyNet(nn.Module):
 def __init__(self):
  super(MyNet, self).__init__()
  self.relu = nn.ReLU()
  self.fc1 = torch.nn.Linear(400, 10)

  def forward(self, x):
   x = x.view(-1, 400)
   print(x.shape)
   x = self.relu(self.fc1(x))
   print(x.shape)

   return x


title: Pytorch：How to balance the backpropagation calculation with multiple gpus

body:
In order to balance calculation of loss backward, I not only calculate loss in forward() but also realese the loss backward.Here is my realease way.Is there any wrong?

 def forward(self, x, y=None, criterion=None, gpu_nums=1):
    x = self.task(x)
    if not y == None:
        loss = criterion(x, y)
        if gpu_nums > 1：
            loss /= gpu_nums
        loss.backward()
        return x, loss
    return x

Is my implementation correct?If not, hope you can provide the right way.This is meaningful for everyone.


title: Replace all indices in tensor within a range with 1s

body:
def generate_mask(data : list, max_seq_len : int):
    """
    Generates a mask for data where each element is expected to be max_seq_len length after padding
    Args:
    data : The data being forwarded through LSTM after being converted to a tensor
    max_seq_len : The length of the names after being padded
    """
    batch_sz = len(data)
    ret = torch.zeros(1,batch_sz, max_seq_len, dtype=torch.bool)
    for i in range(batch_sz):
        name = data[i]

        for letter_idx in range(len(name)):
            ret[0][i][letter_idx] = 1

    return ret

I have this code for generating a mask and I really hate how I'm doing it. Essentially as you can see I'm just going through every name and turning each index from 0 to name length to 1, I'd prefer a more elegant way to do this.


title: Can I duplicate parameter tensor whose values are the same across duplicates while training?

body:
So let's say I have an input vector of size 100. Specifically, the input vector is size [50 + 50]. Then I want to create a parameter tensor of size [50 x 2] that is applied to the first 50 inputs and second 50 inputs. such that we get a vector of size [2 + 2] afterward. I want the first 50 inputs and last 50 inputs of the input vector to be multiplied by the same [50 x 2] parameter tensor and that this remains true while parameter updating. Can you "duplicate" a parameter tensor like so or is there a way to program this? I'm not sure if the ".clone()" function does what I want..

Thanks!


title: How can I get one array to return only the masked values define by another array with Numpy / PyTorch?

body:
I have a mask, which has a shape of: [64, 2895] and an array pred which has a shape of [64, 2895, 161].

mask is binary with only 0s and 1s. What I want to do is reduce pred so that it maintains 64 batches, and along the 2895, wherever there is a 1 in the mask for each batch, return the related pred.

So as a simplified example, if:

mask = [[1, 0, 0],
        [1, 1, 0],
        [0, 0, 1]]
pred = [[[0.12, 0.23, 0.45, 0.56, 0.57],
         [0.91, 0.98, 0.97, 0.96, 0.95],
         [0.24, 0.46, 0.68, 0.80, 0.15]],

        [[1.12, 1.23, 1.45, 1.56, 1.57],
         [1.91, 1.98, 1.97, 1.96, 1.95],
         [1.24, 1.46, 1.68, 1.80, 1.15]],

        [[2.12, 2.23, 2.45, 2.56, 2.57],
         [2.91, 2.98, 2.97, 2.96, 2.95],
         [2.24, 2.46, 2.68, 2.80, 2.15]]]

What I want is:

[[[0.12, 0.23, 0.45, 0.56, 0.57]],

 [[1.12, 1.23, 1.45, 1.56, 1.57],
  [1.91, 1.98, 1.97, 1.96, 1.95]],

 [[2.24, 2.46, 2.68, 2.80, 2.15]]]

I realize that there are different dimensions, I hope that that's possible. If not, then fill in the missing dimensions with 0. Either numpy or pytorch would be helpful. Thank you.


title: how does the neural netwok definition in pytorch use pyton classes

body:
in order to understand how this code works, I have written a small reproducer. How does the self.hidden variable use a variable x in the forward method?

enter code class Network(nn.Module):
def __init__(self):
    super().__init__()

    # Inputs to hidden layer linear transformation
    self.hidden = nn.Linear(784, 256)
    # Output layer, 10 units - one for each digit
    self.output = nn.Linear(256, 10)

    # Define sigmoid activation and softmax output
    self.sigmoid = nn.Sigmoid()
    self.softmax = nn.Softmax(dim=1)

def forward(self, x):
    # Pass the input tensor through each of our operations
    x = self.hidden(x)
    x = self.sigmoid(x)
    x = self.output(x)
    x = self.softmax(x)

    return x


title: Using torch to plot neural network as a function

body:
I've built and trained a neural network using pytorch, which is to be used as a numerical simulation of a function.

It is now viewed as a function of 3 variables:

t as time in the 1-dim euclidean space;
x as location in the 3-dim euclidean space;
v as velocity in the 3-dim euclidean space.

I want to plot this function with x and v fixed as a function of t. Can anyone help me with an idea or some code? I'll be much obliged.


title: How to initialise the h_n and h_c in LSTM?

body: I try to use LSTM do to some classifications, and I see some examples where h_n and h_c are set to 0, can I initialise these two parameters with other values?


title: How to use permute on this Input and Target?

body:
I am having errors on my semantic segmentation masks with 5 classes + 1 (background).

How do I use permute to avoid this?

Target size (torch.Size([4, 1, 320, 480, 6])) must be the same as input size (torch.Size([4, 6, 320, 480]))


title: Shifting tensor over dimension

body:
Lets say I have a tensor of A of shape [batch_size X length X 1024]
I want to do the following :
for the i element in the batch i want to shift the (embedding 1024) of all 'length' elements embedding by their position .
for example the vector A[0 , 0 , : ] should stay the same, and A[0 , 1 , :] should be shifted (or rolled) by 1 , and A[0 , 15 , :] should be shifted by 15.
this is for all the elements in the batch.
so far i did it with for loops, but its not efficient
below is my code with for loops :

    x = # [batchsize , length , 1024]
    new_embedding = []
    llist = []
    batch_size = x.shape[0]
    seq_len    = x.shape[1]
    for sample in range(batch_size):
     for token in range(seq_len):
      orig = x[sample , token , : ]
      new_embedding.append(torch.roll(orig , token , 0))
     llist.append(torch.stack(new_embedding , 0))
     new_embedding = []
    x = torch.stack(llist , 0)


title: conversion of LSTM cell from pytroch to keras model

body:
I have written the code for LSTM cell in pytorch and same model I want to construct in keras , via s subclassing(with __ init__, def call method).Can anyone help me out to solve this problem?

Pytorch code :

class MyModel(torch.nn.Module):

def __init__(self,hidden_size):
    super(MyModel,self).__init__()
    self.lstm = torch.nn.LSTMCell(10,hidden_size,bias = False)
    self.hidden_dim  = hidden_size
def forward(self,inputs):
    hx = torch.zeros(1, self.hidden_dim)
    cx = torch.zeros(1, self.hidden_dim)
    hx, cx = self.lstm(inputs,(hx,cx))
    x = hx
    return x,(hx,cx)

hidden_size = 512

lstm = MyModel(hidden_size)

print(lstm.state_dict())

input = torch.randn(1,10)
x,(hx,cx) = lstm(input)


title: Convert from Torch-Tensor string to Tensor?

body:
I have the following
import torch
a="""tensor([-0.4610, -0.6222,  0.5374, -0.8090,  0.2586,  0.6906, -0.9288, -0.5442,
        -0.7199,  0.6775,  0.6667, -0.4475, -0.1938,  0.3532, -0.3552, -0.5396,
         0.8704, -0.2893,  0.4406,  0.0168,  0.2506,  0.6775, -0.6763,  0.6242],
       device='cuda:0')"""
tensor_obj_i_want=convert_to_tensor(a)

How to convert this string-object to a Tensor using torch? In other words is there a good function "convert_to_tensor" from the library


title: pytorch/gradient computation/inplace operation

body:
I get the above mentioned error and really can't find the in_place operation. This is the code I want to run. For clarification: The list "u" contains neural nets, however I want the last to always return 1 s.t. the assert works out.
def generate_stopping_time_factors_from_path(self, x_input):
    local_N = x_input.shape[1]
    U = torch.empty(local_N)
    U[0] = torch.zeros(1)
    x = torch.empty(self.d, local_N)
    # x = torch.from_numpy(x_input) doesn't work for some reason

    h = torch.empty(self.N + 1)
    h[self.N] = 1

    for n in range(local_N):
        sum = torch.sum(U[0:n])
        x[:, n] = torch.tensor(x_input[:, n], requires_grad=True)
        if n < self.N:
            # all entries in "u" are nets
            h[n] = self.u[n](x[:, n])
        U[n] = h[n] * (torch.ones(1) - sum)

    assert torch.sum(U).item() == 1

    return U

However when I tested it a bit further I found that of the 2 codes below the second doesn't work with the same error, while the first one runs just fine (obviously it computes nonsense, but it does compute)
def generate_stopping_time_factors_from_path(self, x_input):
    return self.u[0](x[:, 0])

def generate_stopping_time_factors_from_path(self, x_input):
    h = torch.empty(self.N + 1)
    h[0] = self.u[0](x[:, 0])
    return h[0]

Is inserting a value in an empty tensor really counted as an in place operation? If so, how can I rewerite my code to account for that? I need to save the output of my net locally as the output of the next net uses it as a factor.


title: pytorch: Add a number based on index

body:
Result I want to achieve:
I've got a tensor A = torch.zeros(3, 3); of which the contents are arbitrary.
I then also have a tensor add_indices = torch.tensor([2, 0, 1]) which I want to add to A in order to get this result:
A = torch.tensor([[0., 0., 1.],
                  [1., 0., 0.],
                  [0., 1., 0.]])

The way I currently do it:
I've already got a (bad) solution for this using indexing, which works. It goes as follows:
A = torch.zeros(3, 3)
A[torch.range(0, 2, dtype=torch.long), add_indices.long()] += 1

The question
However, I find myself slamming the table in Raymond Hettinger style, thinking "There must be a better way!"
I'd love some input on how to do this properly.


title: A potentially dangerous place in pytorch code, which leads to producing NaNs

body:
I am trying to implement Bidirectional LSTM in PyTorch, and the implementation of forward method looks as follows:
def forward(self, inputs):
    word_embeddings = self.embedding.forward(inputs)

    output = torch.zeros((inputs.shape[0], inputs.shape[1], self.tagset_size))

    #init hidden states for LSTM
    hidden_state_L = []
    hidden_state_R = []

    hidden_state_L.append(torch.zeros(inputs.shape[1], self.lstm_hidden_dim))
    hidden_state_R.append(torch.zeros(inputs.shape[1], self.lstm_hidden_dim))

    if next(self.parameters()).is_cuda:
      output = output.cuda()
      hidden_state_L[0] = hidden_state_L[0].cuda()
      hidden_state_R[0] = hidden_state_R[0].cuda()

    for i, word in enumerate(word_embeddings):
      hidden_state_L.append(self.tanh.forward(self.fc_xh_L.forward(word) + self.fc_hh_L.forward(hidden_state_L[-1])))

    for i, word in enumerate(reversed(word_embeddings)):
      hidden_state_R.append(self.tanh.forward(self.fc_xh_R.forward(word) + self.fc_hh_R.forward(hidden_state_R[-1])))

    hidden_state_L.pop()
    hidden_state_R = hidden_state_R[1:][::-1]

    for idx, (h_L, h_R) in enumerate(zip(hidden_state_L, hidden_state_R)):
      output[idx - 1] = self.fc_hy.forward(torch.cat([h_L, h_R], dim = 1))

    return output

In my implementation the hidden states are stored in lists, and then from the list I get activations. The training process makes an epoch and a half, reaching an accuracy of approximately 80%, and then the neural network loss becomes NaN and the training fails. Where is a potentially risky place, or maybe one can suggest more sensible way, than these lists? Or there may be some caveats with the computations on the device?
I would be grateful for possible suggestions


title: How can I trace ATen function calls in pytorch?

body: I'm running MobileNetv2 in Pytorch (https://pytorch.org/hub/pytorch_vision_mobilenet_v2/), and I'd like to inspect which function calls within the ATen library are being executed. Is there any way to trace these?


title: Implemented model network but neither training error nor val error decreasing

body:
Since I'm novice to Pytorch, this question might be a very trivial one, but I'd like to ask for your help about how to solve this one.
I've implemented one network from a paper and used all hyper parameters and all layers described in the paper.
But when it starts training, even though I set the learning rate decay as 0.001, the errors didn't go down. Training errors goes around 3.3~3.4 and test errors around 3.5~3.6 during 100 epochs..!
I could change the hyperparameters to improve the model, but since the paper told exact numbers, I'd like to see whether there's an error in the training code that I've implemented.
The code below is the code that I used for training.
from torch.utils.data.sampler import SubsetRandomSampler
import torch.nn.functional as F
import torch.optim as optim
import torch.nn as nn
import json
import torch
import math
import time
import os

model = nn.Sequential(Baseline(), Classification(40)).to(device)

optimizer = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999))
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.5)

batch = 32

train_path = '/content/mtrain'
train_data = os.listdir(train_path)

test_path = '/content/mtest'
test_data = os.listdir(test_path)

train_loader = torch.utils.data.DataLoader(train_data, batch, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch, shuffle=True)

train_loss, val_loss = [], []
epochs = 100

now = time.time()
print('training start!')
for epoch in range(epochs):

  running_loss = 0.0

  for bidx, trainb32 in enumerate(train_loader):

    bpts, blabel = [], []

    for i, data in enumerate(trainb32):
      path = os.path.join(train_path, data)

      with open(path, 'r') as f:
        jdata = json.load(f)

      label = jdata['label']
      pts = jdata['pts']

      bpts.append(pts)
      blabel.append(label)

    bpts = torch.tensor(bpts).transpose(1, 2).to(device)
    blabel = torch.tensor(blabel).to(device)

    input = data_aug(bpts).to(device)

    optimizer.zero_grad()

    y_pred, feat_stn, glob_feat = model(input)
    # print(f'global_feat is {global_feat}')
    loss = F.nll_loss(y_pred, blabel) + 0.001 * regularizer(feat_stn)

    loss.backward()
    optimizer.step()
    running_loss += loss.item()

    if bidx % 10 == 9:
      vrunning_loss = 0
      vacc = 0
      model.eval()
      with torch.no_grad():

        # val batch
        for vbidx, testb32 in enumerate(test_loader):
          bpts, blabel = [], []

          for j, data in enumerate(testb32):
            path = os.path.join(test_path, data)

            with open(path, 'r') as f:
              jdata = json.load(f)

            label = jdata['label']
            pts = jdata['pts']

            bpts.append(pts)
            blabel.append(label)

          bpts = torch.tensor(bpts).transpose(1, 2).to(device)
          blabel = torch.tensor(blabel).to(device)

          input = data_aug(bpts).to(device)
          vy_pred, vfeat_stn, vglob_feat = model(input)
          # print(f'global_feat is {vglob_feat}')
          vloss = F.nll_loss(vy_pred, blabel) + 0.001 * regularizer(vfeat_stn)
          _, vy_max = torch.max(vy_pred, dim=1)
          vy_acc = torch.sum(vy_max == blabel) / batch

          vacc += vy_acc
          vrunning_loss += vloss

      # print every training 10th batch
      train_loss.append(running_loss / len(train_loader))
      val_loss.append(vrunning_loss / len(test_loader))

      print(f"Epoch {epoch+1}/{epochs} {bidx}/{len(train_loader)}.. "
            f"Train loss: {running_loss / 10:.3f}.."
            f"Val loss: {vrunning_loss / len(test_loader):.3f}.."
            f"Val Accuracy: {vacc/len(test_loader):.3f}.."
            f"Time: {time.time() - now}")
      now = time.time()
      running_loss = 0
      model.train()

print(f'training finish! training time is {time.time() - now}')
print(model.parameters())

savePath = '/content/modelpath.pth'
torch.save(model.state_dict(), '/content/modelpath.pth')

Sorry for the basic question, but if there's no error in this training code, it would be very pleasure to let me know and if there is, please give any hint to solve..
I've implemented pointNet code and the full code is available at https://github.com/RaraKim/PointNet/blob/master/PointNet_pytorch.ipynb
Thank you!


title: How to evaluate loss only on elements satisfying a condition pytorch

body:
I'm working on a regression problem in pytorch. I get good results on my evaluation set, but I want to make sure it's not because I have many small elements and less large ones. Therefore, I would like to check whether I get similar loss for the large elements (eg. elements > 0.01). I use mse loss.
Can anyone pls suggest a way of doing so?
Thanks!


title: Pytorch Buffers causing Problems during Automatic Differentiation

body:
I want to implement batch normalization on my own in pytorch, however I have issues with the buffers to store a running mean and variance. This is my code:
class BatchNorm(nn.Module):

    def __init__(self, num_features, momentum=0.1, eps=1e-5):
        super(BatchNorm, self).__init__()
        self.num_features = num_features
        self.momentum = momentum
        self.eps = eps
        self.alpha = nn.Parameter(torch.ones(num_features, requires_grad=True))
        self.beta = nn.Parameter(torch.zeros(num_features, requires_grad=True))
        self.register_buffer('mu', torch.zeros(num_features))
        self.register_buffer('var', torch.ones(num_features))

    def forward(self, x):
        mu, var = self.update_running_stats(x)
        normalizer = torch.sqrt(var + self.eps)
        x = (x - mu) / normalizer # element-wise standardization
        x = self.alpha * x + self.beta # scale and shift
        return x

    def update_running_stats(self, x):
        batch_mu = x.sum(0) / x.size(0)
        batch_var = ((x - batch_mu)**2).sum(0) / x.size(0)
        self.mu = (1. - self.momentum) * self.mu + self.momentum * batch_mu
        self.var = (1. - self.momentum) * self.var + self.momentum * batch_var
        return self.mu, self.var

When I run my code like this I (with calling backward() on the loss only once), I get the following error: "RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time".
The problem is fixed when I make the following changes in the running_stats() method:
    mu = (1. - self.momentum) * self.mu + self.momentum * batch_mu
    var = (1. - self.momentum) * self.var + self.momentum * batch_var
    return mu, var

But like this the buffers don't get updated and that's the whole point of using them. How can I solve this and what is the issue?


title: Is there how I can treat the error Is a directory?

body:
#Defining the directories to the file
train_image_path = "/content/drive/My Drive/Skin_Disease/train_images"#Path(".../Skin_Disease/train_images")
train_annotation_path = "/content/drive/My Drive/Skin_Disease/train"#Path(".../Skin_Disease/train")

test_image_path = Path("./Skin_Disease/test_images")
test_annotation_path = Path("./Skin_Disease/test")

valid_image_path = Path("./Skin_Disease/valid_images")
valid_annotation_path = Path("./Skin_disease/valid")

#walking through the training directory to get list files

def filelist(root, file_type):
    return [os.path.join(directory_path, f) for directory_path, directory_name,
            files in os.walk(root) for f in files if f.endswith(file_type)]

#creating a dataframe to view the read files
def generate_train_df (anno_path):
    annotations = filelist(anno_path, '.xml')
    anno_list = []
    for annotation in annotations:
        root = ET.parse(anno_path).getroot()
        anno = {}
        anno["filename"] = Path(str(train_image_path) + '/' + root.find("./filename").text)
        anno["width"] = root.find("./size/width").text
        anno["height"] = root.find("./size/height").text
        anno["xmin"] = int(root.find("./object/bndbox/xmin").text)
        anno["ymin"] = int(root.find("./object/bndbox/ymin").text)
        anno["xmax"] = int(root.find("./object/bndbox/xmax").text)
        anno["ymax"] = int(root.find("./object/bndbox/ymax").text)
        anno_list.append(anno)
        #print(anno)
    return pd.DataFrame(anno_list)

df_train = generate_train_df(train_annotation_path)
print(df_train.shape)
df_train.head()
#print(df_train)

The error am getting:
IsADirectoryError                         Traceback (most recent call last)
<ipython-input-36-b80f473b0929> in <module>()
----> 1 df_train = generate_train_df(train_annotation_path)
      2 print(df_train.shape)
      3 df_train.head()
      4 #print(df_train)

2 frames
/usr/lib/python3.6/xml/etree/ElementTree.py in parse(self, source, parser)
    584         close_source = False
    585         if not hasattr(source, "read"):
--> 586             source = open(source, "rb")
    587             close_source = True
    588         try:

IsADirectoryError: [Errno 21] Is a directory: '/content/drive/My Drive/Skin_Disease/train'

Am trying to do Bounding box prediction using pytorch. My annotation is in xml format and they are placed inside a folder. In the above code am trying to walk through  the folder and am picking the elements in the annotation I need and appending it to a list in order to view in a dataframe but am getting the above error. I think the error is a result of the path but am referencing it well but maybe it's not the accepted path I don't know what to do about it.


title: error: model is not define ,after calling from directory

body:
I instantiated my models and saved inside the directory
#instantiation of lstm unit
net1=LSTM_net(input_size=512,hidden_size=512, output_size=101,cnn_net=resnet)
model_save_name = 'net1_spatial_stream.pt'
torch.save(net1.state_dict(), MODELS_PATH_SPATIAL)

when I load a model from a directory
model_save_name = 'net1_spatial_stream.pt'
net1.load_state_dict(torch.load(MODELS_PATH_SPATIAL))
net1=net1.cuda()#pass to gpu

shows error
 1 model_save_name = 'net1_spatial_stream.pt'
 ----> 2 net1.load_state_dict(torch.load(MODELS_PATH_SPATIAL))
  3 net1=net1.cuda()#pass to gpu

 NameError: name 'net1' is not defined


title: How to download the pretrained dataset of huggingface RagRetriever to a custom directory

body:
I'm playing with a RAG example from facebook (huggingface) https://huggingface.co/facebook/rag-token-nq#usage.
Here a very nice explanation of it: https://ai.facebook.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/
The code is very simple but the dataset it downloads in this step is a little big (75GB):
retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=True)

It downloads the dataset in /root/.cache/huggingface/datasets/, something that I'd like to change if possible. This is the output of that line of code is:
Downloading and preparing dataset wiki_dpr/psgs_w100.nq.no_index (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/wiki_dpr/

My question is: how I can change the folder to where download the dataset used by RagRetriever.from_pretrained (the 75GB one) to another one different to root/.cache/huggingface/datasets/ .
Thanks!.


title: What is a subspace of a dimension in pytorch?

body:
The documentation of torch.Tensor.view says:

each new view dimension must either be a subspace of an original dimension, or only span across original dimensions ...

https://pytorch.org/docs/stable/tensors.html?highlight=view#torch.Tensor.view
What is a subspace of a dimension?


title: can't install dependency during python heroku deployment

body:
click==7.1.2
Flask==1.1.2
future==0.18.2
gunicorn==20.0.4
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1,
numpy==1.19.2,
Pillow==7.2.0,
torch==1.5.0+cpu,
torchvision==0.6.0+cpu,
Werkzeug==1.0.1,

these are the required dependency for my web app although there ar two dependency which does not install directly via pip install torch and torchvision package but rather via
pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
but while deploying to heroku we have to create a requirements.txt file so is there a way to explicitly mention the source in the requirements.txt file.
Any help will be really appreciated. Thank you.


title: How do I get my simple neural network to output values closer to the labels?

body:
I'm pretty new to reinforcement learning and deep Q learning. I'm trying to learn by making a tic tac toe DQN as a starter project. I have some experience with Q learning, so made a Q table and am trying to get simple NN to model the Q table.
However, the NN is not converging onto the Q table (with whatever parameters I try).
The data and code are here: https://github.com/smpurkis/pytorch_playground/tree/master/tic_tac_toe
I thought I was going crazy, so decided to train for 25 epochs on a single input and output. As the label and output should be almost exactly the same then.
I found the same problem. So am completely lost as to why the network isn't converging exactly.
Any thought, or help would be greatly appreciated!
Code is below.
import warnings
from torch.utils.data import Dataset, DataLoader

import pytorch_lightning as pl
import torch
from torch import nn
import torch.nn.functional as F
import pandas as pd
import numpy as np
import random
warnings.filterwarnings("ignore")

class TicTacToeModel(pl.LightningModule):
    def __init__(self, board_size=3, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.learning_rate = 0.001

        # basic network, should be enough to approximate Q-table. Have tried much more complex models, same result
        self.input_layer = nn.Linear(board_size**2, board_size**2)
        # self.hidden_layer = nn.Linear(board_size**2, board_size**2)
        self.output_layer = nn.Linear(board_size**2, board_size**2)

    def forward(self, t):
        # rely to add non-linearity
        t = F.tanh(self.input_layer(t))
        # t = F.tanh(self.hidden_layer(t))
        t = self.output_layer(t)
        return t

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        # standard loss for regression
        loss_fn = torch.nn.MSELoss()
        loss = loss_fn(y_hat, y)

        return {"loss": loss}

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)
        return [optimizer]

    def train_dataloader(self):
        self.static_q_table = StaticQTable()
        train_dataloader = DataLoader(dataset=self.static_q_table, batch_size=1, shuffle=True, num_workers=1)
        return train_dataloader

class StaticQTable(Dataset):
    def __init__(self):
        super().__init__()
        self.q = np.load("10000_games.npy", allow_pickle=True)
        self.q = self.q[np.random.randint(self.q.shape[0], size=1), :]
        self.db = pd.DataFrame(self.q, columns=["state", "q_values"])

    def __len__(self):
        return self.q.shape[0]

    def __getitem__(self, index):
        state, q_values = torch.Tensor(self.q[index][0]), torch.Tensor(self.q[index][1])

        if not int(torch.max(torch.abs(q_values))) == 0:
            q_values = torch.div(q_values, torch.max(torch.abs(q_values)))

        return [state, q_values]

    def get_q_values(self, state):
        return self.db.loc[self.db["state"] == state, "q_values"].values[0]

if __name__ == "__main__":
    # example state, -1 = O, 1 = X
    # state = torch.Tensor([0., 0., 0., 0., 0., 0., -1., 0., 1.])

    net = TicTacToeModel()
    trainer = pl.Trainer(min_epochs=1, max_epochs=25, gpus=1)
    trainer.fit(net)
    torch.save(net.state_dict(), "static_model.pth")
    # net.load_state_dict(torch.load("static_model.pth"))

    net.eval()
    state = torch.Tensor(net.static_q_table.q[0][0])
    q_values = torch.Tensor(net.static_q_table.q[0][1])
    if not int(torch.max(torch.abs(q_values))) == 0:
        q_values = torch.div(q_values, torch.max(torch.abs(q_values)))
    print(state, q_values)
    output = net(state)
    print(output)
    # output I got, in my last try
    # board state: tensor([ 0., -1.,  1.,  0.,  1.,  0., -1., -1.,  1.])
    # labels: tensor([1.0000, 0.0000, 0.0000, 0.8100, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000])
    # NN output: tensor([ 0.1497, -0.0335,  0.0308, -0.1341,  0.0103,  0.0310, -0.1358, -0.0607, -0.0380], grad_fn=<AddBackward0>)


title: Longer LSTM Prediction

body:
I am using an LSTM to take 5 sequences as input to predict another 5. I want to know how to predict more than 5 timesteps. I assume its got something to do with the hidden_dim but I can't figure it out.
Here is my code
class LSTM(nn.Module):
    def __init__(self, seq_len=5, n_features=256, n_hidden=256, n_layers=1, output_size=1):
        super().__init__()
        self.n_features = n_features
        self.seq_len = seq_len
        self.n_hidden = n_hidden
        self.n_layers = n_layers

        self.l_lstm = nn.LSTM(input_size=self.n_features, hidden_size=self.n_hidden, num_layers=self.n_layers, batch_first=True)

    def init_hidden(self, batch_size):
        hidden_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(device)
        cell_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(device)
        self.hidden = (hidden_state, cell_state)

    def forward(self, x):

        lstm_out, self.hidden = self.l_lstm(x,self.hidden)

        return lstm_out

If anyone knows how to extend the prediction range or could suggest a better way of writing LSTM, I would really appreciate it.


title: What is the correct input to LSTM?

body:
I have tensors of varying length . These tensors are data for different time period. My aim is to get final output of the lstm.
torch.randn(4)-Time1
torch.randn(3,4)-Time2
torch.randn(4,4)-Time3

These are my  data, what is the input to LSTM from here?  , my aim is to get the final output from the lstm
For example, this is what I did
out_position = self.linear_out_position(features)
    _, (hn, _) = self.lstm(out_position)
        output = self.ffn(hn)

however i am getting result for each tensor, how can i only get the final result? Please I need your help


title: Pytorch: creating repeated nn.Linear layers using list[] failed on GPU

body:
I'm trying to create 10 Linear layers in my model. It fails when using cuda I'm using a list to create these layers. It works with CPU.
Creating layers individually seems no problem.
Also working with setattr() and getattr() works. Don't understand why the list is not working.
class Cifar10Model(common.BaseClassificationModel):

def __init__(self, input_size, hidden_size, output_size):
    super().__init__()
    self.input_size = input_size
    self.fc1 = nn.Linear(input_size, hidden_size) #fully connection layer

    self.fc_layers = []
    self.num_fcs = 10
    for i in range(self.num_fcs):
         self.fc_layers.append(nn.Linear(hidden_size, hidden_size)) #fully connection layer
#        setattr(self, "fc_%d" % i, nn.Linear(hidden_size, hidden_size))

    self.fc_out = nn.Linear(hidden_size, output_size) #fully connection layer

    #define loss function
    # for cross_entropy, the label should be int/long: to indicate which one our predict is.
    self.loss_fn = torch.nn.functional.cross_entropy
    #self.fc1.reset_parameters()
    #self.fc2.reset_parameters()

#image_tensor is a batch
def forward(self, image_tensor):
    batch_size = image_tensor.size(0)   #size of highest dimision
    image_tensor = image_tensor.view(batch_size, -1) #-1 can work with any batch size

    out = self.fc1(image_tensor)
    out = F.relu(out)     #activation

    for i in range(self.num_fcs):
        #out = getattr(self, "fc_%d" % i)(out)
        out = self.fc_layers[i](out)
        out = F.relu(out)     #activation

    out = self.fc_out(out)

    return out

The error is RuntimeError: CUDA error: an illegal memory access was encountered during training.
Maybe there is another better way? I would like to create 100 layers.


title: How to infer inputs of a neural network via MCMC?

body:
I know I can use gradient descent to infer input [parameters] of a neural network model but for some reason I need to infer those parameters via MCMC. To give you more context, the input has low number of parameters (e.g. 10) and they do not come from a specific distribution (e.g. Gaussian, Uniform etc) but are bounded in [0, 1]. During test time, the neural network model produces an image given an input. My goal is to infer the input parameters conditioned on an output. I am pretty new to Pyro and MCMC but I looked at an example here but cannot figure out how to translate what the example is doing to my problem because pyro.sample seems to require using a specific distribution (maybe I can assume Uniform for my parameters?) and it uses SVI and not MCMC. In addition, the VAE model used in the example has a definition that is different that the model I am using and I'm not sure how to modify the example to make it work in my case. So I wonder, how can I condition my model on an [expected] output and infer the input parameters of my model using Pyro and via MCMC?
Here's a simplified version of the neural network model I'm using:
class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
        .
        . # define model layers
        .

    def forward(self, x):
        .
        . # do transposed convolutional operations on x and get predicted_means, predicted_sigma and reconsted_image
        .

        elbo = torch.sum(Normal(predicted_means, predicted_sigma).log_prob(reconsted_image), dim=[1,2,3])

        return elbo

    def reconstruct(self, x):

        # This is the function I use during test time and MCMC needs to use this function

        .
        . # do transposed convolutional operations on x and get mu
        .

        return torch.clamp(mu, 0, 1)


title: Reasons for an LSTM getting stuck at one value (always predicting same value for a sequence)?

body:
I tried to use an LSTM in pytorch to generate new songs (respectively generating sequences of notes) I use 100 midi file note sequences as training data but everytime, the model ends up only predicting a sequence of always the same value. Are there any obvious design mistakes i made?
Example (0-128 are the different notes, I just hardcoded 76 as the first note for sampling a new song):
epoch: 9
loss: 3.723158359527588
song: [76, 71, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, 62]
INPUT_DIM = 128
EMBEDDING_DIM = 12
LSTM_NUM_LAYERS = 2
LSTM_HIDDEN_DIM = 8
OUTPUT_DIM = INPUT_DIM

log_softmax = nn.LogSoftmax(dim=1)

class MidiLSTM(nn.Module):
    def __init__(self):
        super(MidiLSTM, self).__init__()

        self.embeddings = nn.Embedding(INPUT_DIM, EMBEDDING_DIM)
        self.lstm = nn.LSTM(EMBEDDING_DIM, LSTM_HIDDEN_DIM, num_layers=LSTM_NUM_LAYERS)
        self.lin = nn.Linear(LSTM_HIDDEN_DIM, OUTPUT_DIM)
        self.to(device)

    def forward(self, song):
        embeds = self.embeddings(song)
        hidden_all, _ = self.lstm(embeds.view(len(song), 1, -1))
        output_space = self.lin(hidden_all.view(len(song), -1))[:-1]
        scores = output_space
        return scores

    def predict(self, song):
        embeds = self.embeddings(song)
        hidden_all, _ = self.lstm(embeds.view(len(song), 1, -1))
        output_space = self.lin(hidden_all.view(len(song), -1))[-1:]
        scores = output_space
        return torch.argmax(scores)

I used following lossfunc and optimizer:
loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)


title: error message for loading the pre-trained model

body:
I was trying to use the following code segment to leverage a pre-trained model.
ENCODER_NAME = 'se_resnext50_32x4d'

class HuBMAPModel(nn.Module):
    def __init__(self):
        super(HuBMAPModel, self).__init__()
        self.model = Unet(encoder_name = ENCODER_NAME,
                          encoder_weights = None,
                          classes = 1,
                          activation = None)

    def forward(self, images):
        img_masks = self.model(images)
        return img_masks

state_dict = torch.load(PATH_TRAINED_MODEL)
model = HuBMAPModel()
model.load_state_dict(state_dict)

However, running the code gave error messages such as the following. What can be the cause for this kind of error message? Is that because the pre-trained model was not save right or because the way I load the model?
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-26f8c9ef9b94> in <module>
     15 state_dict = torch.load(PATH_TRAINED_MODEL)
     16 model = HuBMAPModel()
---> 17 model.load_state_dict(state_dict)

~\Anaconda3\envs\tamulab\lib\site-packages\torch\nn\modules\module.py in load_state_dict(self, state_dict, strict)
    828         if len(error_msgs) > 0:
    829             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 830                                self.__class__.__name__, "\n\t".join(error_msgs)))
    831         return _IncompatibleKeys(missing_keys, unexpected_keys)
    832

RuntimeError: Error(s) in loading state_dict for HuBMAPModel:
    Missing key(s) in state_dict: "model.encoder.layer0.conv1.weight", "model.encoder.layer0.bn1.weight", "model.encoder.layer0.bn1.bias", "model.encoder.layer0.bn1.running_mean", "model.encoder.layer0.bn1.running_var", "model.encoder.layer1.0.conv1.weight", "model.encoder.layer1.0.bn1.weight", "model.encoder.layer1.0.bn1.bias", "model.encoder.layer1.0.bn1.running_mean", "model.encoder.layer1.0.bn1.running_var", "model.encoder.layer1.0.conv2.weight", "model.encoder.layer1.0.bn2.weight", "model.encoder.layer1.0.bn2.bias", "model.encoder.layer1.0.bn2.running_mean",


title: pytorch: fast way to get the indices of the largest element of a matrix

body:
If I have a tensor (n*n matrix) [[1,2,3],[3,4,5],[5,6,7]], how can i get the indices of the largest element ([2,2]) using pytorch efficiently?
I tried to do t.max(matrix, 2). But this only returns the largest element instead of its index. Is there a simple way to do so?
Thanks!!


title: How to custome the gradient of some variables in Tensorflow 2.0

body:
Assume I have the code
import tensorflow as tf
class Net(tf.keras.layers.Layer):
    # Set layers.
    def __init__(self, input_dim, output_dim):
        super(Net, self).__init__()
        self.weight = self.add_weight('weiht', [input_dim, output_dim])
        self.bias = self.add_weight('bia', [output_dim])

    # Set forward pass.
    def call(self, x):
        x = tf.matmul(x, self.weight) + self.bias
        return x

    def update_weight(self):
        pass

    def update_bias(self):
        pass

x = tf.constant([[1.0,2.,3., 4.,5.],[6.,7.,8.,9.,10.]])
y = tf.constant([4.,6.])

model = Net(5, 2)
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)
with tf.GradientTape() as tape:
    y_pred = model(x)
    loss = tf.nn.l2_loss(y-y_pred)

grads = tape.gradient(loss, model.trainable_variables)
print(grads)
optimizer.apply_gradients(zip(grads, model.trainable_variables))

The print(grads) gives
[<tf.Tensor: shape=(5, 2), dtype=float32, numpy=
array([[ -34.14073 , -109.76845 ],
       [ -45.0742  , -136.88483 ],
       [ -56.007668, -164.00119 ],
       [ -66.94114 , -191.11757 ],
       [ -77.8746  , -218.23393 ]], dtype=float32)>, <tf.Tensor: shape=(2,), dtype=float32, numpy=array([-10.933468, -27.116375], dtype=float32)>]

In this way, the Tensorflow 2, will automatically update the gradients of weight and bias.
My question is that: May I modify the gradients based on the results of tensorflow?  For example, I want to implement self.weight.grad = tf.math.sigmoid(self.weight.grad) + 10 and self.bias=cos(self.bias)
It is easy to implement in PyTorch since we can access the gradient self.weight.grad.data. How to do the same things in tensorflow2?
Thanks,


title: Training & Validation loss and dataset size

body:
I'm new on Neural Networks and I am doing a project that has to define a NN and train it. I've defined a NN of 2 hidden layers with 17 inputs and 17 output. The NN has 21 inputs and 3 outputs.
I have a data set of labels of 10 million, and a dataset of samples of another 10 million. My first issue is about the size of the validation set and the training set. I'm using PyTorch and batches, and of what I've read, the batches shouldn't be larger. But I don't know how many approximately should be the size of the sets.
I've tried with larger and small numbers, but I cannot find a correlation that shows me if I'm right choosing a large set o small set in one of them (apart from the time that requires to process a very large set).
My second issue is about the Training and Validation loss, which I've read that can tell me if I'm overfitting or underfitting depending on if it is bigger or smaller. The perfect should be the same value for both, and it also depends on the epochs. But I am not able to tune the network parameters like batch size, learning rate or choosing how much data should I use in the training and validation. If 80% of the set (8 million), it takes hours to finish it, and I'm afraid that if I choose a smaller dataset, it won't learn.
If anything is badly explained, please feel free to ask me for more information. As I said, the data is given, and I only have to define the network and train it with PyTorch.
Thanks!


title: How to make tensordot operations after permutation

body:
I have 2 tensors, A and B:
A = torch.randn([32,128,64,12],dtype=torch.float64)
B = torch.randn([64,12,64,12],dtype=torch.float64)
C = torch.tensordot(A,B,([2,3],[0,1]))
D = C.permute(0,2,1,3) # shape:[32,64,128,12]

tensor D comes from the operations "tensordot -> permute". How can I implement a new operation f() to make the tensordot operation after f() like:
A_2 = f(A)
B_2 = f(B)
D = torch.tensordot(A_2,B_2)