title: pytorch Dataloader error "Too many open files" when yielding an int

body:
I'm trying to implement a custom IterableDataset in which I read words from a file, get theirs unique id, gather them and return them batched.
import os
import torch
import tqdm
from torch.utils.data import IterableDataset, DataLoader
import vocab  # C++ class bound to python with pybind11

class MyIterableDataset(IterableDataset):
    def __init__(self, file_path, v, num_workers=4):
        super(MyIterableDataset).__init__()
        self.file_path = file_path
        self.file_size = os.stat(file_path).st_size
        self.v = v  # vocab object, bound from a C++ class with pybind11

        chunk_size = self.file_size // num_workers
        start = 0
        end = chunk_size
        bonus = self.file_size - chunk_size * num_workers
        if (bonus > 0):
            end = chunk_size + 1
            bonus -= 1
        self.endpoints = [(start, end)]
        for i in range(1, num_workers):
            start = end
            if (bonus > 0):
                end += chunk_size + 1
                bonus -= 1
            else:
                end += chunk_size
            self.endpoints.append((start, end))

    def read_word(self, f):
        ch = ''
        word = ""
        while True:
            ch = f.read(1)
            if not ch:
                return ''
            if (str.isspace(ch)):
                if len(word) > 0:
                    break
                if (ch == '\n'):
                    return "\n"
                else:
                    continue
            word += ch
        return word

    def parse_file(self, start, words_to_read, id):
        words_read = 0
        f = open(self.file_path, "r")
        f.seek(start, 0)
        if id > 0:
            while True:
                ch = f.read(1)
                if not ch or str.isspace(ch):
                    break
                start += 1
            f.seek(start, 0)
        while True:
            word = self.read_word(f)
            if word and word != "\n":
                wid = self.v.word2id(word)
                if wid != -1:
                    words_read += 1
                    yield wid  # if I yield 'word' instead, everything works. You can also yield 1 and you get the error
            if words_read >= words_to_read or not word:
                break
        f.close()

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        words_to_read = self.v.get_train_words() // worker_info.num_workers
        start, end = self.endpoints[worker_info.id]
        return self.parse_file(start, words_to_read, worker_info.id)

Upon running a DataLoader over my dataset with
num_workers = 7
v = vocab.Vocab("./text8")  # Vocab is a C++ class bound to python with pybind11
ds = MyIterableDataset(file_path=file_path, v=v, num_workers=num_workers)
wids = [j for _, j in tqdm.tqdm(enumerate(DataLoader(ds, num_workers=num_workers, batch_size=10)))]

whenever I yield the word id I get the following error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-04575fb9c982> in <module>
      2
      3 t0 = time.time()
----> 4 tokens = [j for _, j in tqdm.tqdm(enumerate(DataLoader(ds, num_workers=num_workers, batch_size=10)))]
      5 print()
      6 print(time.time() - t0)

<ipython-input-18-04575fb9c982> in <listcomp>(.0)
      2
      3 t0 = time.time()
----> 4 tokens = [j for _, j in tqdm.tqdm(enumerate(DataLoader(ds, num_workers=num_workers, batch_size=10)))]
      5 print()
      6 print(time.time() - t0)

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/tqdm/std.py in __iter__(self)
   1165
   1166         try:
-> 1167             for obj in iterable:
   1168                 yield obj
   1169                 # Update and possibly print the progressbar.

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_data(self)
   1066
   1067             assert not self._shutdown and self._tasks_outstanding > 0
-> 1068             idx, data = self._get_data()
   1069             self._tasks_outstanding -= 1
   1070             if self._dataset_kind == _DatasetKind.Iterable:

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _get_data(self)
   1032         else:
   1033             while True:
-> 1034                 success, data = self._try_get_data()
   1035                 if success:
   1036                     return data

~/miniconda3/envs/word2gm/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _try_get_data(self, timeout)
    897             except OSError as e:
    898                 if e.errno == errno.EMFILE:
--> 899                     raise RuntimeError(
    900                         "Too many open files. Communication with the"
    901                         " workers is no longer possible. Please increase the"

RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code

while if I yield the word everything works!
Can someone help me understand why this is happening in the first place?


title: Slow execution time for CUDA initialization in Azure Batch VM

body:
I have an issue of slow initialization time for running some CUDA program in one of the VM for Azure Batch.
After some troubleshooting, I made a simple test running this call as shown in the below code.
#include <stdio.h>
#include <cuda.h>
#include <cuda_runtime_api.h>
#include <time.h>

clock_t start, end;
double cpu_time_used;

int main()
{
    CUresult result;
    printf("CUDA version %d \n", CUDA_VERSION);
    start = clock();
    result = cuInit(0);
    if (result != CUDA_SUCCESS) {
        printf("cuInit failed with error code %d: %s\n", result, cudaGetErrorString(result));
        return 1;
    }
    end = clock();
    cpu_time_used = ((double) (end - start)) / CLOCKS_PER_SEC;
    printf("cuInit took %f seconds to execute \n", cpu_time_used);
    return 0;
}

It takes about 1.9 seconds in average.
Some specs:

NVidia driver: 460.32.03
CUDAToolkit: 10.2
Azure Batch: nc6, Tesla K80

As a comparisons, the same code was running on my desktop (Windows) as well another custom Azure VM (nc6, not Azure Batch) giving similar result of 0.03 seconds. (cudatoolkit 10.2)
-- update 1 --
Calling CUDA initialization via Torch also shows a significant lag (for the first call) as shown from this test:
run: 0, import torch: 0.430, cuda_available: 4.921
run: 1, import torch: 0.000, cuda_available: 0.000
run: 2, import torch: 0.000, cuda_available: 0.000
run: 3, import torch: 0.000, cuda_available: 0.000
max time for import torch: 0.43 s,  max time for cuda_available: 4.92 s
torch.version 1.7.1+cu101 torch.version.cuda: 10.1

The import torch code is:
import torch

and the cuda_available code is:
torch.cuda.is_available()

My question is the time taken by the Azure Batch for CUDA initialization normal behavior ?


title: Score being minimised by pytorch NN with the Cartpole problem

body:
I am trying to solve the CartPole problem in openAI gym by training a simple 2 layer NN in pytorch. The method used is DQN yet results converge on a maximum score of around 8 or 9 and is not seen to improve over time or with training. Instead the score gets lower with training. How this might be improved/what is wrong in the code that is making it do this? Below is the code used:

import gym
import random
import math
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import torch.nn.functional as F
from collections import namedtuple
import numpy as np

class network(nn.Module):
    def __init__(self):
        nn.Module.__init__(self)
        # network takes 4 inputs (state, action, next_state, reward), hidden layer then has
        # 256 inputs and the network has 2 outputs (the q value of going left or right)
        # in this network the index of the output references the action.
        self.l1 = nn.Linear(4, 256)
        self.l2 = nn.Linear(256, 2)

def forward(self, x):
    # forward function defines how the model will run
    x = F.relu(self.l1(x))
    x = self.l2(x)
    return (x)

class replay_memory():
    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []

def save(self, transition):
    # saves all transitions for the environment in a tensor
    self.memory.append(transition)
    if len(self.memory) > self.capacity:
        del self.memory[0]

def sample(self, batch_size):
    # generates a random sample from the memory
    return random.sample(self.memory, batch_size)

def __len__(self):
    return len(self.memory)

class agent():
    def __init__(self, env, model):
        self.epsilon = 1  # exploration rate
        self.epsilon_min = 0.001 # smallest exploration value
        self.epsilon_decay = 0.995 # rate at which exploration occurs
        self.learning_rate = 0.001

def act(self, state, model):
        # define actions, random or optimal based on exploration rate DOES NOT ACCOUNT FOR THE DECAY
    if random.uniform(0, 1) <= self.epsilon:
        action = torch.LongTensor([[random.randrange(2)]])
        action_np = (action.numpy())[0][0]
    else:

        action = model(Variable(torch.FloatTensor([state])).type(torch.FloatTensor)).max(1)[1].view(1,1)
        action_np = (action.numpy())[0][0]

    if self.epsilon > self.epsilon_min:
        self.epsilon *= self.epsilon_decay
    else:
        self.epsilon = self.epsilon_min

    return action, action_np

def trained_act(self, episodes, network, env):
    for e in range (episodes):
        state = env.reset()
        for t in range (200):
            action, action_np = agent.act(state, network)
            next_state, reward, done, info = env.step(action_np)
            env.render()
            if done:
                break
        print(t)

        env.close()

def learn(batch_size, gamma, memory, optimizer):
    BATCH_SIZE = batch_size
    if len(memory) < BATCH_SIZE:
        return
# random transition batch is taken from experience replay memory.
transitions = memory.sample(BATCH_SIZE)
batch_state, batch_action, batch_reward, batch_next_state = zip(*transitions)

batch_state = Variable(torch.cat(batch_state))
batch_action = Variable(torch.cat(batch_action))
batch_reward = Variable(torch.cat(batch_reward))
batch_next_state = Variable(torch.cat(batch_next_state))

current_q_values = network.forward(batch_state).gather(1, batch_action.unsqueeze(-1))

max_next_q_values = network.forward(batch_next_state).detach().max(1)[0]
expected_q_values = batch_reward + (gamma * max_next_q_values)

# loss is measured from error between current and newly expected Q values
loss = F.smooth_l1_loss(expected_q_values, current_q_values)

# backpropagation of loss to NN
optimizer.zero_grad()
loss.backward()
optimizer.step()

return loss

env = gym.make('CartPole-v0')
env.reset()
network = network()
agent = agent(env, network)
batch_size = 50
episode = 500
T = 200
gamma = 0.95
memory = replay_memory(100)
optimizer = optim.SGD(network.parameters(), 0.001)
l = []
s = []

for e in range (episode):
    state = env.reset()
    for t in range (T):
        action, action_np = agent.act(state, network)
        next_state, reward, done, info = env.step(action_np)
        if done:
            reward = -2

    transition = torch.FloatTensor([state]),torch.LongTensor([action]), torch.FloatTensor([reward]),torch.FloatTensor([next_state])

    memory.save(transition)
    state = next_state

    loss = learn(batch_size, gamma, memory, optimizer)
    l.append(loss)
    if done:
        print('Loss = {}, Episode = {}, finsited after {} steps'.format(loss, e, t))
        s.append(t)
        break


title: what is the biggest bottleneck in maskrcnn_benchmark repo?

body:
I am working on a repo that make use of the maskrcnn_benchmark repo. I have extensively, explored the bench-marking repo extensively for the cause of its slower performance on a cpu with respect to enter link description here.

In order to create a benchmark for the individual forward passes I have put a time counter for each part and it gives me the time required to calculate each component. I have had a tough time exactly pinpointing as to the slowest component of the entire architecture.I believe it to be BottleneckWithFixedBatchNorm class in the maskrcnn_benchmark/modeling/backbone/resnet.py file.

I will really appreciate any help in localisation of the biggest bottle neck in this architecture.


title: Is prepend() a built-in function or macro of CMake?

body:
I was trying to read the source code of Pytorch, and got stuck with its CMake files.

As you can see in the file:utils.cmake:

##############################################################################
# Macro to update cached options.
macro (caffe2_update_option variable value)
  if(CAFFE2_CMAKE_BUILDING_WITH_MAIN_REPO)
    get_property(__help_string CACHE ${variable} PROPERTY HELPSTRING)
    set(${variable} ${value} CACHE BOOL ${__help_string} FORCE)
  else()
    set(${variable} ${value})
  endif()
endmacro()

##############################################################################
# Add an interface library definition that is dependent on the source.
#
# It's probably easiest to explain why this macro exists, by describing
# what things would look like if we didn't have this macro.
#
# Let's suppose we want to statically link against torch.  We've defined
# a library in cmake called torch, and we might think that we just
# target_link_libraries(my-app PUBLIC torch).  This will result in a
# linker argument 'libtorch.a' getting passed to the linker.
#
# Unfortunately, this link command is wrong!  We have static
# initializers in libtorch.a that would get improperly pruned by
# the default link settings.  What we actually need is for you
# to do -Wl,--whole-archive,libtorch.a -Wl,--no-whole-archive to ensure
# that we keep all symbols, even if they are (seemingly) not used.
#
# What caffe2_interface_library does is create an interface library
# that indirectly depends on the real library, but sets up the link
# arguments so that you get all of the extra link settings you need.
# The result is not a "real" library, and so we have to manually
# copy over necessary properties from the original target.
#
# (The discussion above is about static libraries, but a similar
# situation occurs for dynamic libraries: if no symbols are used from
# a dynamic library, it will be pruned unless you are --no-as-needed)
macro(caffe2_interface_library SRC DST)
  add_library(${DST} INTERFACE)
  add_dependencies(${DST} ${SRC})
  # Depending on the nature of the source library as well as the compiler,
  # determine the needed compilation flags.
  get_target_property(__src_target_type ${SRC} TYPE)
  # Depending on the type of the source library, we will set up the
  # link command for the specific SRC library.
  if (${__src_target_type} STREQUAL "STATIC_LIBRARY")
    # In the case of static library, we will need to add whole-static flags.
    if(APPLE)
      target_link_libraries(
          ${DST} INTERFACE -Wl,-force_load,$<TARGET_FILE:${SRC}>)
    elseif(MSVC)
      # In MSVC, we will add whole archive in default.
      target_link_libraries(
          ${DST} INTERFACE -WHOLEARCHIVE:$<TARGET_FILE:${SRC}>)
    else()
      # Assume everything else is like gcc
      target_link_libraries(${DST} INTERFACE
          "-Wl,--whole-archive,$<TARGET_FILE:${SRC}> -Wl,--no-whole-archive")
    endif()
    # Link all interface link libraries of the src target as well.
    # For static library, we need to explicitly depend on all the libraries
    # that are the dependent library of the source library. Note that we cannot
    # use the populated INTERFACE_LINK_LIBRARIES property, because if one of the
    # dependent library is not a target, cmake creates a $<LINK_ONLY:src> wrapper
    # and then one is not able to find target "src". For more discussions, check
    #   https://gitlab.kitware.com/cmake/cmake/issues/15415
    #   https://cmake.org/pipermail/cmake-developers/2013-May/019019.html
    # Specifically the following quote
    #
    # """
    # For STATIC libraries we can define that the PUBLIC/PRIVATE/INTERFACE keys
    # are ignored for linking and that it always populates both LINK_LIBRARIES
    # LINK_INTERFACE_LIBRARIES.  Note that for STATIC libraries the
    # LINK_LIBRARIES property will not be used for anything except build-order
    # dependencies.
    # """
    target_link_libraries(${DST} INTERFACE
        $<TARGET_PROPERTY:${SRC},LINK_LIBRARIES>)
  elseif(${__src_target_type} STREQUAL "SHARED_LIBRARY")
    if("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU")
      target_link_libraries(${DST} INTERFACE
          "-Wl,--no-as-needed,$<TARGET_FILE:${SRC}> -Wl,--as-needed")
    else()
      target_link_libraries(${DST} INTERFACE ${SRC})
    endif()
    # Link all interface link libraries of the src target as well.
    # For shared libraries, we can simply depend on the INTERFACE_LINK_LIBRARIES
    # property of the target.
    target_link_libraries(${DST} INTERFACE
        $<TARGET_PROPERTY:${SRC},INTERFACE_LINK_LIBRARIES>)
  else()
    message(FATAL_ERROR
        "You made a CMake build file error: target " ${SRC}
        " must be of type either STATIC_LIBRARY or SHARED_LIBRARY. However, "
        "I got " ${__src_target_type} ".")
  endif()
  # For all other interface properties, manually inherit from the source target.
  set_target_properties(${DST} PROPERTIES
    INTERFACE_COMPILE_DEFINITIONS
    $<TARGET_PROPERTY:${SRC},INTERFACE_COMPILE_DEFINITIONS>
    INTERFACE_COMPILE_OPTIONS
    $<TARGET_PROPERTY:${SRC},INTERFACE_COMPILE_OPTIONS>
    INTERFACE_INCLUDE_DIRECTORIES
    $<TARGET_PROPERTY:${SRC},INTERFACE_INCLUDE_DIRECTORIES>
    INTERFACE_SYSTEM_INCLUDE_DIRECTORIES
    $<TARGET_PROPERTY:${SRC},INTERFACE_SYSTEM_INCLUDE_DIRECTORIES>)
endmacro()

##############################################################################
# Creating a Caffe2 binary target with sources specified with relative path.
# Usage:
#   caffe2_binary_target(target_name_or_src <src1> [<src2>] [<src3>] ...)
# If only target_name_or_src is specified, this target is build with one single
# source file and the target name is autogen from the filename. Otherwise, the
# target name is given by the first argument and the rest are the source files
# to build the target.
function(caffe2_binary_target target_name_or_src)
  # https://cmake.org/cmake/help/latest/command/function.html
  # Checking that ARGC is greater than # is the only way to ensure
  # that ARGV# was passed to the function as an extra argument.
  if (ARGC GREATER 1)
    set(__target ${target_name_or_src})
    prepend(__srcs "${CMAKE_CURRENT_SOURCE_DIR}/" "${ARGN}")
  else()
    get_filename_component(__target ${target_name_or_src} NAME_WE)
    prepend(__srcs "${CMAKE_CURRENT_SOURCE_DIR}/" "${target_name_or_src}")
  endif()
  add_executable(${__target} ${__srcs})
  target_link_libraries(${__target} ${Caffe2_MAIN_LIBS})
  # If we have Caffe2_MODULES defined, we will also link with the modules.
  if (DEFINED Caffe2_MODULES)
    target_link_libraries(${__target} ${Caffe2_MODULES})
  endif()
  if (USE_TBB)
    target_include_directories(${__target} PUBLIC ${TBB_ROOT_DIR}/include)
  endif()
  install(TARGETS ${__target} DESTINATION bin)
endfunction()

function(caffe2_hip_binary_target target_name_or_src)
  if (ARGC GREATER 1)
    set(__target ${target_name_or_src})
    prepend(__srcs "${CMAKE_CURRENT_SOURCE_DIR}/" "${ARGN}")
  else()
    get_filename_component(__target ${target_name_or_src} NAME_WE)
    prepend(__srcs "${CMAKE_CURRENT_SOURCE_DIR}/" "${target_name_or_src}")
  endif()

  caffe2_binary_target(${target_name_or_src})

  target_compile_options(${__target} PRIVATE ${HIP_CXX_FLAGS})
  target_include_directories(${__target} PRIVATE ${Caffe2_HIP_INCLUDE})
endfunction()

##############################################################################
# Multiplex between loading executables for CUDA versus HIP (AMD Software Stack).
# Usage:
#   torch_cuda_based_add_executable(cuda_target)
#
macro(torch_cuda_based_add_executable cuda_target)
  IF (USE_ROCM)
    hip_add_executable(${cuda_target} ${ARGN})
  ELSEIF(USE_CUDA)
    cuda_add_executable(${cuda_target} ${ARGN})
  ELSE()

  ENDIF()
endmacro()

##############################################################################
# Multiplex between adding libraries for CUDA versus HIP (AMD Software Stack).
# Usage:
#   torch_cuda_based_add_library(cuda_target)
#
macro(torch_cuda_based_add_library cuda_target)
  IF (USE_ROCM)
    hip_add_library(${cuda_target} ${ARGN})
  ELSEIF(USE_CUDA)
    cuda_add_library(${cuda_target} ${ARGN})
  ELSE()
  ENDIF()
endmacro()

##############################################################################
# Get the NVCC arch flags specified by TORCH_CUDA_ARCH_LIST and CUDA_ARCH_NAME.
# Usage:
#   torch_cuda_get_nvcc_gencode_flag(variable_to_store_flags)
#
macro(torch_cuda_get_nvcc_gencode_flag store_var)
  # setting nvcc arch flags
  if ((NOT EXISTS ${TORCH_CUDA_ARCH_LIST}) AND (DEFINED ENV{TORCH_CUDA_ARCH_LIST}))
    message(WARNING
        "In the future we will require one to explicitly pass "
        "TORCH_CUDA_ARCH_LIST to cmake instead of implicitly setting it as an "
        "env variable. This will become a FATAL_ERROR in future version of "
        "pytorch.")
    set(TORCH_CUDA_ARCH_LIST $ENV{TORCH_CUDA_ARCH_LIST})
  endif()
  if (EXISTS ${CUDA_ARCH_NAME})
    message(WARNING
        "CUDA_ARCH_NAME is no longer used. Use TORCH_CUDA_ARCH_LIST instead. "
        "Right now, CUDA_ARCH_NAME is ${CUDA_ARCH_NAME} and "
        "TORCH_CUDA_ARCH_LIST is ${TORCH_CUDA_ARCH_LIST}.")
    set(TORCH_CUDA_ARCH_LIST TORCH_CUDA_ARCH_LIST ${CUDA_ARCH_NAME})
  endif()

  # Invoke cuda_select_nvcc_arch_flags from proper cmake FindCUDA.
  cuda_select_nvcc_arch_flags(${store_var} ${TORCH_CUDA_ARCH_LIST})
endmacro()

##############################################################################
# Add standard compile options.
# Usage:
#   torch_compile_options(lib_name)
function(torch_compile_options libname)
  set_property(TARGET ${libname} PROPERTY CXX_STANDARD 14)

  if (NOT INTERN_BUILD_MOBILE OR NOT BUILD_CAFFE2_MOBILE)
    # until they can be unified, keep these lists synced with setup.py
    if(MSVC)

      if (MSVC_Z7_OVERRIDE)
        set(MSVC_DEBINFO_OPTION "/Z7")
      else()
        set(MSVC_DEBINFO_OPTION "/Zi")
      endif()

      target_compile_options(${libname} PUBLIC
        ${MSVC_RUNTIME_LIBRARY_OPTION}
        ${MSVC_DEBINFO_OPTION}
        /EHa
        /DNOMINMAX
        /wd4267
        /wd4251
        /wd4522
        /wd4522
        /wd4838
        /wd4305
        /wd4244
        /wd4190
        /wd4101
        /wd4996
        /wd4275
        /bigobj
        )
    else()
      target_compile_options(${libname} PUBLIC
        #    -std=c++14
        -Wall
        -Wextra
        -Wno-unused-parameter
        -Wno-missing-field-initializers
        -Wno-write-strings
        -Wno-unknown-pragmas
        # Clang has an unfixed bug leading to spurious missing braces
        # warnings, see https://bugs.llvm.org/show_bug.cgi?id=21629
        -Wno-missing-braces
        )

      if(NOT APPLE)
        target_compile_options(${libname} PRIVATE
          # Considered to be flaky.  See the discussion at
          # https://github.com/pytorch/pytorch/pull/9608
          -Wno-maybe-uninitialized)
      endif()

    endif()

    if (MSVC)
    elseif (WERROR)
      target_compile_options(${libname} PRIVATE -Werror -Wno-strict-overflow)
    endif()
  endif()

  if (NOT WIN32 AND NOT USE_ASAN)
    # Enable hidden visibility by default to make it easier to debug issues with
    # TORCH_API annotations. Hidden visibility with selective default visibility
    # behaves close enough to Windows' dllimport/dllexport.
    #
    # Unfortunately, hidden visibility messes up some ubsan warnings because
    # templated classes crossing library boundary get duplicated (but identical)
    # definitions. It's easier to just disable it.
    target_compile_options(${libname} PRIVATE "-fvisibility=hidden")
  endif()

  # Use -O2 for release builds (-O3 doesn't improve perf, and -Os results in perf regression)
  target_compile_options(${libname} PRIVATE "$<$<OR:$<CONFIG:Release>,$<CONFIG:RelWithDebInfo>>:-O2>")

  # ---[ Check if warnings should be errors.
  # TODO: Dedupe with WERROR check above
  if (WERROR)
    target_compile_options(${libname} PRIVATE -Werror)
  endif()
endfunction()

##############################################################################
# Set standard target properties.
# Usage:
#   torch_set_target_props(lib_name)
function(torch_set_target_props libname)
  if(MSVC AND AT_MKL_MT)
    set_target_properties(${libname} PROPERTIES LINK_FLAGS_RELEASE "/NODEFAULTLIB:${VCOMP_LIB}")
    set_target_properties(${libname} PROPERTIES LINK_FLAGS_DEBUG "/NODEFAULTLIB:${VCOMP_LIB}")
    set_target_properties(${libname} PROPERTIES STATIC_LIBRARY_FLAGS "/NODEFAULTLIB:${VCOMP_LIB}")
  endif()
endfunction()

In the line 125, it uses a function or macro named

prepend(__srcs "${CMAKE_CURRENT_SOURCE_DIR}/" "${ARGN}").

As you may have noticed that such file is just a CMake module file and without any external import at a glance. Therefore, I guess it might be a built-in function or macro of CMake.

However, I also looked up to the CMake Manual and search for details of prepend() and the search result shows that there is no keyword matching with prepend as built-in functions or macros.

Here is my question: is prepend a legal built-in function or macro in CMake? Even though I can understand what kind of operation this piece of code is trying to do (it tries to prepend ${CMAKE_CURRENT_SOURCE_DIR}/ to every item in ${ARGN}), I just want to further confirm whether such usage is still effective (or recommended) in current latest CMake system.

BTW, I am not an adept in CMake scripting so I am not whether my guess is correct. Longing for your replies and generous help.


title: How to use ITK to convert PNG into tensor for PyTorch

body:
I'm trying to run a C++ PyTorch framework and ran into the following problem.

I successfully scripted a model and is now ready to run. Now I have to feed in a png image to the model.

I found someone with simmilar issue on the internet, his idea was to use ITK module to read in a PNG file and convert it to a array, then make it to Tensor.

PNG -> RGBPixel[] -> tensor

So the following is what I am trying out right now.

using PixelTyupe = itk::RGBPixel<unsinged char>;
const unsigned int Dimension = 3;
typedef itk::Image<PixelType, Dimension>      ImageType;
typedef itk::ImageFileReader<ImageType>       ReaderType;
typedef itk::ImageRegionIterator<ImageType>   IteratorType;

typename ImageType::RegionType region = itk_img->GetLargestPossibleRegion();
const typename ImageType::SizeType size = region.GetSize();

int len = size[0] * size[1] * size[2]; // This ends up 1920 * 1080 * 1
PixelType rowdata[len];
int count = 0;
IteratorType iter(itk_img, itk_img->GetRequestedRegion());

// convert itk to array
for (iter.GoToBegin(); !iter.IsAtEnd(); ++iter) {
   rowdata[count] = iter.Get();
   count++;
} // count = 1920 * 1080

// convert array to tensor
tensor_img = torch::from_blob(rowdata, {3, (int)size[0], (int)size[1]}, torch::kShort). clone(); // Segmenation Fault

When I try to print log data, it holds three numbers like 84 85 83, so I suppose the PNG file is successfully read.

However, I can't get the last part of the code to work. What I need is 3:1920:1080 tensor, but I don't think the three RGBPixel value is understood properly by the function.

And apart from that, I don't see why the dimension is set to 3.

I would appreciate any kind of help.


title: The affine transformation parameters obtained by cv2.getaffinetransform are not applicable to pytorch's F.affine_grid API

body:
I am training a network similar to Spatial Transformer Networks, which is used to estimate affine transformation parameters according to the incoming heatmap. The labels are the parameters of affine transformation through the coordinates of the target box. So I tried cv2.getaffinetransform to get the parameters of the corresponding affine transformation, but it seemed that the final result was not ideal.

This code is to display the bbox and (minx,miny,maxx,maxy) is the coordinates of the bbox

miny=127
minx=191
maxx=351
maxy=351
print(miny,minx,maxx,maxy)
fig, ax = plt.subplots(ncols=1, nrows=1, figsize=(6, 6))
ax.imshow(image)
rect=patches.Rectangle((minx, miny), maxx - minx, maxy - miny,
                              fill=False, edgecolor='red', linewidth=2)
ax.add_patch(rect)

origin image with bbox

The goal is to extract the region of the bbox by affine transformation

And I get the parameters through the coordinates,because the shift value in M that cv2 gets is the pixel value, and the shift in affine_grid is supposed to be scale value, so I deal with shift value in this way

points1=np.float32([[191,127],[191,351],[351,351]])
points2=np.float32([[0,0],[0,448],[448,448]])
M=cv2.getAffineTransform(points2,points1)
M[0][2]=M[0][2]/448
M[1][2]=M[1][2]/448

Finally, I use the steps of affine transformation in Spatial Transformer Networks to transform：

theta=torch.from_numpy(M).float()
grid=F.affine_grid(theta.unsqueeze(0),tensor_image.shape)
grid=grid.cuda()
output=F.grid_sample(tensor_image,grid)

But the result is not good

affine image1

Someone told me that maybe the cv2 image coordinates are different from the pytorch image coordinates,So I switch the axes of the points and calculate the affine transformation parameters again

points1=np.float32([[127,191],[351,191],[351,351]])
points2=np.float32([[0,0],[448,0],[448,448]])
M=cv2.getAffineTransform(points2,points1)
M[0][2]=M[0][2]/448
M[1][2]=M[1][2]/448

But it  is also not good:

affine image2


title: How to use TPU Pod in Polyaxon

body:
Did anyone manage to use TPU Pod (eg. 32 v2 TPU cores) in experiments on GKE in Polyaxon experiment?
I have a lot of problems when trying to do that. I'm using Pytorch.

Eg., you need to know the TPU Pod name to start training with TPUs (https://github.com/pytorch/xla#start-distributed-training), which you don't know before running the experiment because TPU is created after the experiment is scheduled.

But, even if I find out name somehow after the experiment starts (which I did), I have a problem when connecting to TPU Pod (ssh permissions error) because the user in the experiment container doesn't have permissions to connect to TPU Pod ...

When I start TPU Pod manually (without Polyaxon), I'm able to connect with TPU Pod (ssh succeeds) because I have the correct user - my user on GCP (this ssh is called from some instance in instance group created for this purpose). I have followed these instructions for manually starting a TPU Pod: https://cloud.google.com/tpu/docs/tutorials/pytorch-pod#create-instance-template

EDIT

This is the experiment configuration I have used for starting TPU Pod in Polyaxon:

version: 1

kind: experiment

environment:
  resources:
    tpu:
      limits: 32
      requests: 32
  node_selector:
    polyaxon: tpu

build:
  image: gcr.io/tpu-pytorch/xla:r1.5

run:
  cmd:
    - TPU_NAME=FETCHING_TPU_NAME_COMMAND
    - python -m torch_xla.distributed.xla_dist
         --tpu=$TPU_NAME
         --conda-env=pytorch
         --env XLA_USE_BF16=1
         --
         python /pytorch/xla/test/test_train_mp_imagenet.py
         --fake_data

Also, on the Polyaxon settings page, under Hardware Accelerators settings, I have set pytorch-1.5 for K8S:TPU_TF_VERSION. I'm using Polyaxon 0.6.0.

FETCHING_TPU_NAME_COMMAND is just a call to some python k8s API for fetching the name of TPU Pod (it would be nice that this could be fetched more easily because the name is needed when starting the Pytorch code which uses TPU Pod). Eg. this: https://github.com/polyaxon/polyaxon-k8s/blob/master/polyaxon_k8s/manager.py

Actually, inside FETCHING_TPU_NAME_COMMAND I store the fetched TPU Pod name in some file and then read it, but I put it this way in the above example to make it more clear what I'm doing.

When running python -m torch_xla.distributed.xla_dist ..., it will cause the ssh permission error because in the container is the user that doesn't have permission to connect to TPU Pod.

So I wanted to create a user in the container which will have permissions to connect to TPU. I tried two things, but failed:

I have tried to create a user using my GCP uid and gid inside the container during the build, but then that build caused all Polyaxon pods to stuck, and then it takes some time for GKE to restart those pods ...
Then I tried to set the uid and gid in Polyaxon deployment configuration (described here: https://legacy-docs.polyaxon.com/configuration/security-context/#enable-security-context), but it was the same: now Polyaxon alone tried to create a user with given uid and gid (it can be seen from build log) and it stuck, again causing all Polyaxon pods to stuck ...

I have created an instance group with 4 instances because of 32 tpu cores (as it says in the instructions here https://cloud.google.com/tpu/docs/tutorials/pytorch-pod#create-instance-template).

EDIT 2

I tried again manually starting a pod which uses TPUs, but now with creating  a new user, becuase the root user (which is default in the container) was causing ssh permission denied error.

Here is the job .yaml config:

apiVersion: batch/v1
kind: Job
metadata:
  name: pytorch-tpu-train-mnist
spec:
  template:
    metadata:
      annotations:
        # The runtime version that the TPU will run with.
        # Note: It's called "tf-version" for historical reasons.
        tf-version.cloud-tpus.google.com: "pytorch-1.5"
    spec:
      restartPolicy: Never
      volumes:
      # Increase size of tmpfs /dev/shm to avoid OOM.
      - name: dshm
        emptyDir:
          medium: Memory
      containers:
      - name: mnist-pytorch-tpu
        # This is the image we publish nightly with our package pre-installed.
        image: gcr.io/tpu-pytorch/xla:r1.5
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        command: [
            'bash', '-c',
            'USERNAME=antonio_juric && USER_UID=5001 && USER_GID=$USER_UID && groupadd --gid $USER_GID $USERNAME && useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME && apt-get update && apt-get install -y sudo && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME && chmod 0440 /etc/sudoers.d/$USERNAME && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* && echo PATH="$PATH" >> /etc/environment && su - $USERNAME -c "sudo chown $USERNAME /root && source /etc/environment && python -m torch_xla.distributed.xla_dist --tpu=tpu-test-manual --conda-env=pytorch --env XLA_USE_BF16=1 -- python /pytorch/xla/test/test_train_mp_imagenet.py --fake_data"'
        ]
        resources:
          requests:
            memory: 30Gi
            cpu: 10
      nodeSelector:
        polyaxon: tpu

The nodeSelector: polyaxon: tpu is label instance group used for this purpose (size of that instance group is fixed to 4 due to 32 tpu cores, no autoscaling). No tpus was requested in resources section because I created TPU vm manually so that I could know the name of the TPU Pod (tpu-test-manual is the name); otherwise, if tpus were requested in resource section, it will trigger creation of TPU vm with random name ... In the container execution command, I first create the new user named antonio_juric. It turned out that uid and gid doesn't need to be your GCP account uid and gid, so I put 5001. Then I switch from root user to new user and run the command. Path sourcing is due to calling right python (from anaconda package, that python has pytorch xla installed ...).

This manages to ssh to TPU Pod, so there is no ssh permission anymore. But, new problem is that executing bash scripts on the instances from the instance group fails, eg.:

2020-06-17 10:53:57 10.128.0.54 [2] bash: /tmp/326-remote/dist_training_ptxla_2.sh: Permission denied

I see that this file is created inside by file: https://github.com/pytorch/xla/blob/master/torch_xla/distributed/xla_dist.py Not sure why it now fails with permission denied. If the process creates that file (I see that it is created when I ssh manually into a vm from instance group), why it then wouldn't have permission to run it ...?


title: Pytorch Faster R-CNN size mismatch errors in testing

body:
there!
When running test_net.py in pytorch1.0 Faster R-CNN and demo.py on coco dataset with faster_rcnn_1_10_9771.pth(the pretrained resnet101 model on coco dataset provided by jwyang), I encounter the same errors below :
Called with args:
Namespace(batch_size=1, cfg_file='cfgs/res101.yml', checkepoch=10, checkpoint=9771, checksession=1, class_agnostic=True, cuda=True, dataset='coco', image_dir='/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/images', load_dir='/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/models', mGPUs=True, net='res101', parallel_type=0, set_cfgs=None, vis=True, webcam_num=-1)
/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/lib/model/utils/config.py:374: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  yaml_cfg = edict(yaml.load(f))
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
 'ANCHOR_SCALES': [4, 8, 16, 32],
 'CROP_RESIZE_WITH_MAX_POOL': False,
 'CUDA': False,
 'DATA_DIR': '/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/data',
 'DEDUP_BOXES': 0.0625,
 'EPS': 1e-14,
 'EXP_DIR': 'res101',
 'FEAT_STRIDE': [16],
 'GPU_ID': 0,
 'MATLAB': 'matlab',
 'MAX_NUM_GT_BOXES': 20,
 'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
               'FIXED_LAYERS': 5,
               'REGU_DEPTH': False,
               'WEIGHT_DECAY': 4e-05},
 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
 'POOLING_MODE': 'align',
 'POOLING_SIZE': 7,
 'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
 'RNG_SEED': 3,
 'ROOT_DIR': '/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0',
 'TEST': {'BBOX_REG': True,
          'HAS_RPN': True,
          'MAX_SIZE': 1000,
          'MODE': 'nms',
          'NMS': 0.3,
          'PROPOSAL_METHOD': 'gt',
          'RPN_MIN_SIZE': 16,
          'RPN_NMS_THRESH': 0.7,
          'RPN_POST_NMS_TOP_N': 300,
          'RPN_PRE_NMS_TOP_N': 6000,
          'RPN_TOP_N': 5000,
          'SCALES': [600],
          'SVM': False},
 'TRAIN': {'ASPECT_GROUPING': False,
           'BATCH_SIZE': 128,
           'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
           'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
           'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
           'BBOX_NORMALIZE_TARGETS': True,
           'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
           'BBOX_REG': True,
           'BBOX_THRESH': 0.5,
           'BG_THRESH_HI': 0.5,
           'BG_THRESH_LO': 0.0,
           'BIAS_DECAY': False,
           'BN_TRAIN': False,
           'DISPLAY': 20,
           'DOUBLE_BIAS': False,
           'FG_FRACTION': 0.25,
           'FG_THRESH': 0.5,
           'GAMMA': 0.1,
           'HAS_RPN': True,
           'IMS_PER_BATCH': 1,
           'LEARNING_RATE': 0.001,
           'MAX_SIZE': 1000,
           'MOMENTUM': 0.9,
           'PROPOSAL_METHOD': 'gt',
           'RPN_BATCHSIZE': 256,
           'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
           'RPN_CLOBBER_POSITIVES': False,
           'RPN_FG_FRACTION': 0.5,
           'RPN_MIN_SIZE': 8,
           'RPN_NEGATIVE_OVERLAP': 0.3,
           'RPN_NMS_THRESH': 0.7,
           'RPN_POSITIVE_OVERLAP': 0.7,
           'RPN_POSITIVE_WEIGHT': -1.0,
           'RPN_POST_NMS_TOP_N': 2000,
           'RPN_PRE_NMS_TOP_N': 12000,
           'SCALES': [600],
           'SNAPSHOT_ITERS': 5000,
           'SNAPSHOT_KEPT': 3,
           'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
           'STEPSIZE': [30000],
           'SUMMARY_INTERVAL': 180,
           'TRIM_HEIGHT': 600,
           'TRIM_WIDTH': 600,
           'TRUNCATED': False,
           'USE_ALL_GT': True,
           'USE_FLIPPED': True,
           'USE_GT': False,
           'WEIGHT_DECAY': 0.0001},
 'USE_GPU_NMS': True}
load checkpoint /home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/models/res101/coco/faster_rcnn_1_10_9771.pth
Traceback (most recent call last):
  File "/home/ubuntu/users/fasterrcnn/faster-rcnn.pytorch-pytorch-1.0/demo.py", line 205, in <module>
    fasterRCNN.load_state_dict(checkpoint['model'])
  File "/home/ubuntu/users/anaconda3/envs/fasterrcnn/lib/python3.6/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for resnet:
    **size mismatch for RCNN_bbox_pred.weight:** copying a param with shape torch.Size([324, 2048]) from checkpoint, the shape in current model is torch.Size([4, 2048]).
    **size mismatch for RCNN_bbox_pred.bias**: copying a param with shape torch.Size([324]) from checkpoint, the shape in current model is torch.Size([4]).

Process finished with exit code 1

And here is my envs:
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
blas                      1.0                         mkl    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
bzip2                     1.0.8                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ca-certificates           2020.1.1                      0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cairo                     1.14.12              h8948797_3    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
certifi                   2020.4.5.1               py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
cffi                      1.14.0           py36he30daa8_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
cuda100                   1.0                           0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
cycler                    0.10.0                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
cython                    0.29.17          py36he6710b0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
dbus                      1.13.14              hb2f20db_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
easydict                  1.9                        py_0    https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge
expat                     2.2.6                he6710b0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
faster-rcnn               0.1                       dev_0    <develop>
ffmpeg                    4.0                  hcdf2ecd_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
fontconfig                2.13.0               h9420a91_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
freeglut                  3.0.0                hf484d3e_5    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
freetype                  2.9.1                h8a8886c_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
glib                      2.63.1               h3eb4bd4_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
graphite2                 1.3.13               h23475e2_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
gst-plugins-base          1.14.0               hbbd80ab_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
gstreamer                 1.14.0               hb31296c_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
harfbuzz                  1.8.8                hffaf4a1_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
hdf5                      1.10.2               hba1933b_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
icu                       58.2                 he6710b0_3    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
intel-openmp              2020.1                      217    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
jasper                    2.0.14               h07fcdf6_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jpeg                      9b                   h024ee3a_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
kiwisolver                1.2.0            py36hfd86e86_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
ld_impl_linux-64          2.33.1               h53a641e_7    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libedit                   3.1.20181209         hc058e9b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libffi                    3.3                  he6710b0_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libgcc-ng                 9.1.0                hdf63c60_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libgfortran-ng            7.3.0                hdf63c60_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libglu                    9.0.0                hf484d3e_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libopencv                 3.4.2                hb342d67_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libopus                   1.3.1                h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libpng                    1.6.37               hbc83047_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libprotobuf               3.11.4               hd408876_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx-ng              9.1.0                hdf63c60_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libtiff                   4.1.0                h2733197_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libuuid                   1.0.3                h1bed415_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libvpx                    1.7.0                h439df22_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libxcb                    1.13                 h1bed415_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
libxml2                   2.9.9                hea5a465_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
matplotlib                3.1.3                    py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
matplotlib-base           3.1.3            py36hef1b27d_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl                       2020.1                      217    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl-service               2.3.0            py36he904b0f_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl_fft                   1.0.15           py36ha843d7b_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
mkl_random                1.1.1            py36h0573a6f_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
ncurses                   6.2                  he6710b0_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
ninja                     1.9.0            py36hfd86e86_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
numpy                     1.18.1           py36h4f9e942_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
numpy-base                1.18.1           py36hde5b4d6_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
olefile                   0.46                     py36_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
opencv                    3.4.2            py36h6fd60c2_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
openssl                   1.1.1g               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pcre                      8.43                 he6710b0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pillow                    7.1.2            py36hb39fc2d_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pip                       20.0.2                   py36_3    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pixman                    0.38.0               h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
protobuf                  3.11.4           py36he6710b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
py-opencv                 3.4.2            py36hb342d67_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
pycparser                 2.20                       py_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pyparsing                 2.4.7                      py_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pyqt                      5.9.2            py36h05f1152_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
python                    3.6.10               h7579374_2    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
python-dateutil           2.8.1                      py_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
pytorch                   1.0.0           py3.6_cuda10.0.130_cudnn7.4.1_1  [cuda100]  https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
pyyaml                    5.3.1            py36h7b6447c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
qt                        5.9.7                h5867ecd_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
readline                  8.0                  h7b6447c_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
scipy                     1.2.1            py36h7c811a0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
setuptools                46.4.0                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
sip                       4.19.8           py36hf484d3e_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
six                       1.14.0                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
sqlite                    3.31.1               h62c20be_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
tensorboardx              2.0                        py_0    conda-forge
tk                        8.6.8                hbc83047_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
torchvision               0.2.0            py36h17b6947_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch
tornado                   6.0.4            py36h7b6447c_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
wheel                     0.34.2                   py36_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
xz                        5.2.5                h7b6447c_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
yaml                      0.1.7                had09818_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zlib                      1.2.11               h7b6447c_3    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
zstd                      1.3.7                h0b5b093_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main

I tried the most useful way described here:
solution1
to __C.ANCHOR_SCALES = [4,8,16,32].

Also tried modifying the classes of pascal voc dataset to the classes of coco dataset
coco_classes = np.asarray(["__background__",
                             "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat",
                             "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep",
                             "cow", "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee",
                             "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard",
                             "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich",
                             "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed",
                             "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven",
                             "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier",
                             "toothbrush"])

But still don't work for me.
Anyone help?


title: Larger batch size didn't bring shorter training time in semantic segmentation

body:
I'm new to semantic segmentation and distribute dataparallel of PyTorch.
And I run this pspnet code on my machine (2 x P100).

To my knowledge, larger batch size usually brings shorter training time. But it didn't happen this time.

On one GPU, the training time with batch_size=8 is actually the same as (even a little longer) than batch_size=2. Specifically, the training time for 10 iterations with bs=8 is 4 times of that bs=2.

Did I make some mistake here?


title: pytorch albumentations augmentation p value?

body:
augmented_images(raw.image_id.unique()[1230], albumentations.HorizontalFlip(p=1))

for augmented_image what is the p=1 mean? is value difference make angle different?
if its not it how should I make various angle different horizontal augmentation?


title: Runtime error in pytoch-lightning based multimodal task

body:
def _build_model(self):

    lang_module = torch.nn.Linear(
            in_features=self.embedding_dim,
            out_features=self.language_feature_dim
    )

    vision_module = torchvision.models.resnet152(
        pretrained=True
    )
    vision_module.fc = torch.nn.Linear(
            in_features=2048,
            out_features=self.vision_feature_dim
    )

Hello, I was working on multimodal classification work based on Facebook hateful meme challenge.
I was trying to change the baseline code as stated above with a pretrained BERT model with the code below
    lang_module = BertForSequenceClassification.from_pretrained(
        "bert-base-uncased",
        num_labels = 2,
        output_attentions = False,
        output_hidden_states = False,
    )
    lang_module.fc = torch.nn.Linear(
        in_features=self.embedding_dim, #150
        out_features=self.language_feature_dim #300
    )

But I am getting a RTE which says
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got
torch.cuda.FloatTensor instead (while checking arguments for embedding)

with a note before showing RTE:
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

So I just want to check how it works with BERT pretrained model for the text classification.
Any help will be really appreciated.


title: Why does .items() work for the multiplication between sequence and `float`

body:
I got two dicts self.theta & self.beta whose values are all fp and a fp value eta, when I tried to modify it as below:
self.theta[0]=self.theta[0]-eta*self.beta[0]

It returned TypeError: can't multiply sequence by non-int of type 'float'.
If I put it into a for loop of the dict, it worked, but I cannot separately manipulate with a single key such as [0] above, have to finish the modification in the for loop in this case.
for key, value in self.theta.items():
      self.theta[key]=self.theta[key]-eta*self.beta[key]

I wonder is it possible to directly modify the self.theta[0] without getting the error? The numpy array seems like a workaround but I need to change the code a lot so I am thinking about solely torch methods.
p.s. I know the list cannot multiply with fp, but how come the for loop iterates the .items() works? Full scripts below:
beta = defaultdict(list)
theta = defaultdict(list)
layer = [500,300,100,3]
eta = 0.1
i = 0
for l in layer[0:-1]:
    # print(l)
    theta[l] = (torch.randn(l, layer[i+1]))/math.sqrt(l)
    beta[l] = torch.Tensor(l, layer[i+1])
    i += 1

theta[0] = theta[0] - eta * beta[0]
# TypeError: can't multiply sequence by non-int of type 'float'

for key, value in theta.items():
    theta[key] = theta[key] - eta * beta[key]
    print('theta:', theta[key])
# This is okay


title: How would I apply a nn.conv1d manually, given an input matrix and weight matrix?

body:
I am trying to understand how a nn.conv1d processes an input for a specific example related to audio processing in a WaveNet model.
I have input data of shape (1,1,8820), which passes through an input layer (1,16,1), to output a shape of (1,16,8820).
That part I understand, because you can just multiply the two matrices. The next layer is a conv1d, kernel size=3, input channels=16, output channels=16, so the state dict shows a matrix with shape (16,16,3) for the weights. When the input of (1,16,8820) goes through that layer, the result is another (1,16,8820).
What multiplication steps occur within the layer to apply the weights to the audio data? In other words, if I wanted to apply the layer(forward calculations only) using only the input matrix, the state_dict matrix, and numpy, how would I do that?
This example is using the nn.conv1d layer from Pytorch. Also, if the same layer had a dilation=2, how would that change the operations?


title: PyTorch CUDA Out of memory

body:
I am trying to run a modeling  script using pytorch.I tried batch processing where I am sending only 30 videos at a time to my model.
 File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 988, in forward
    return F.prelu(input, self.weight)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1319, in prelu
    return torch.prelu(input, weight)
**RuntimeError: CUDA out of memory. Tried to allocate 2.19 GiB (GPU 0; 11.17 GiB total capacity; 7.43 GiB already allocated; 535.31 MiB free; 10.58 GiB reserved in total by PyTorch)
^M 83%|████████▎ | 5/6 [00:39<00:07,  7.82s/it]**

I am trying to understand what this error is indicating? Any advice on what could be done?


title: Why there are orders of magnitude in the evaluation of torch.sin() and numpy.sin() at integer multiples of Pi?

body:
Why are trigonometric functions of Pytorch and Numpy when evaluated at integer multiples of Pi result in such huge discrepancy in the orders of magnitude?
>>> torch.sin(torch.ones(1)*2*np.pi)
tensor([1.7485e-07])
>>> np.sin(np.ones(1)*2*np.pi)
array([-2.4492936e-16])


title: matching PyTorch tensor dimensions

body:
I am having some issues with regards to the dimensionality of my tensors in my training function at present. I am using the MNIST dataset, so 10 possible targets, and originally wrote the prototype code using a training batch size of 10, which was in retrospect not the wisest choice. It gave poor results during some earlier tests, and increasing the amount of training iterations saw no benefit. Upon trying to then increase the batch size, I realised that what I had written was not that general, and I was likely never training it on the proper data. Below is my training function:
def Train(tLoops, Lrate):
    for _ in range(tLoops):
        tempData = train_data.view(batch_size_train, 1, 1, -1)
        output = net(tempData)
        trainTarget = train_targets
        criterion = nn.MSELoss()
        print("target:", trainTarget.size())
        print("Output:", output.size())
        loss = criterion(output, trainTarget.float())
        # print("error is:", loss)
        net.zero_grad()  # zeroes the gradient buffers of all parameters
        loss.backward()
        for j in net.parameters():
            j.data.sub_(j.grad.data * Lrate)

of which the print functions return
target: torch.Size([100])
Output: torch.Size([100, 1, 1, 10])

before the error message on the line where loss is calculated;
RuntimeError: The size of tensor a (10) must match the size of tensor b (100) at non-singleton dimension 3

The first print, target, is a 1-dimensional list of the respective ground truth values for each image. Output contains the output of the neural net for each of those 100 samples, so a 10 x 100 list, however from skimming and reshaping the data from 28 x 28 to 1 x 784 earlier, I seem to have extra dimensions unnecessarily. Does PyTorch provide a way to remove these? I couldn't find anything in the documentation, or is there something else that could be my issue?


title: Dividing 3D tensor into Batches (Pytorch)

body:
I have a tensor of shape (50,50,2). That is filled with values of a grid in (x,y). Like this.
r =0.5
#Sample 3D space
min = -10
max = 10
complexnum = 50j

#Sample 3D space
X,Y  = np.mgrid[min:max:complexnum,min:max:complexnum,] * r

T = np.sin(3*X) * np.sin(3*Y)

XY = np.concatenate((X.reshape((X.shape[0],X.shape[0],1)),Y.reshape((X.shape[0],X.shape[0],1))),2)
XY = torch.from_numpy(XY.astype(np.float32)).to(device)
T = T.reshape((T.shape[0],T.shape[0],1))
T = torch.from_numpy(T.astype(np.float32)).to(device)

I created a Neural Network that can take the XY as input and T as an output. In that is creating a function f(x,y). Now I need to create a way to train that NN using batches. The total amount of samples should be 2500 = 50 x 50. I want to divide the XY and T tensor into batches. I have tried DataLoader and creating a data set class with no luck. Is there a way I can achieve this? Are my Tensor shapes correct?
For some context, this is the custom data set I created to give to the DataLoader :
class CustomTrainDataset(Dataset):
def __init__(self, x_tensor, y_tensor):
    self.xdomain = x_tensor
    self.ydomain = y_tensor

def __getitem__(self, index):

  # I flatten them to find the ith sample
  augxy = torch.from_numpy(np.vstack([self.xdomain[:,:,0].cpu().numpy().ravel(), self.xdomain[:,:,1].cpu().numpy().ravel()]).transpose().astype(np.float32)).to(device)

   augw= torch.from_numpy(np.vstack([self.ydomain.cpu().detach().numpy().ravel()]).transpose().astype(np.float32)).to(device)
    xy =  augxy[index]

    l = torch.where((self.xdomain[:,:,0] == xy[0]) )
    m = torch.where((self.xdomain[:,:,1]== xy[1]))

    xindex = m[0][0]
    yindex = l[1][0]

    return (self.xdomain[index],self.ydomain[index])

def __len__(self):
    return len(self.xdomain) * len(self.xdomain)

I understand I have to change __ len __ and __ getitem __. I thought by giving the a ([1,2],[1,1]) as the ith sample, and 50x50 as the total length would give me a nice batch. But instead, it gives me an array of size(2500, 2) divided into batches.
Sorry for this very basic question, I have been struggling to work with higher dimensions.


title: Runtime error while trying dynamic quantization on transformer model

body:
I am trying to do dynamic quantization(quantizes the weights and the activations) on a pytorch pre-trained model from huggingface library. I have referred this link and found dynamic quantization the most suitable. I will be using the quantized model on a CPU.
Link to hugginface model here.
torch version: 1.6.0 (installed via pip)
Pre-trained models
tokenizer = AutoTokenizer.from_pretrained("microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext")
model = AutoModel.from_pretrained("microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext")

Dynamic quantization
quantized_model = torch.quantization.quantize_dynamic(
    model, qconfig_spec={torch.nn.Linear}, dtype=torch.qint8
)

print(quantized_model)

Error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-df2355c17e0b> in <module>
      1 quantized_model = torch.quantization.quantize_dynamic(
----> 2     model, qconfig_spec={torch.nn.Linear}, dtype=torch.qint8
      3 )
      4
      5 print(quantized_model)

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in quantize_dynamic(model, qconfig_spec, dtype, mapping, inplace)
    283     model.eval()
    284     propagate_qconfig_(model, qconfig_spec)
--> 285     convert(model, mapping, inplace=True)
    286     _remove_qconfig(model)
    287     return model

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in convert(module, mapping, inplace)
    363     for name, mod in module.named_children():
    364         if type(mod) not in SWAPPABLE_MODULES:
--> 365             convert(mod, mapping, inplace=True)
    366         reassign[name] = swap_module(mod, mapping)
    367

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in convert(module, mapping, inplace)
    363     for name, mod in module.named_children():
    364         if type(mod) not in SWAPPABLE_MODULES:
--> 365             convert(mod, mapping, inplace=True)
    366         reassign[name] = swap_module(mod, mapping)
    367

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in convert(module, mapping, inplace)
    363     for name, mod in module.named_children():
    364         if type(mod) not in SWAPPABLE_MODULES:
--> 365             convert(mod, mapping, inplace=True)
    366         reassign[name] = swap_module(mod, mapping)
    367

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in convert(module, mapping, inplace)
    363     for name, mod in module.named_children():
    364         if type(mod) not in SWAPPABLE_MODULES:
--> 365             convert(mod, mapping, inplace=True)
    366         reassign[name] = swap_module(mod, mapping)
    367

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in convert(module, mapping, inplace)
    363     for name, mod in module.named_children():
    364         if type(mod) not in SWAPPABLE_MODULES:
--> 365             convert(mod, mapping, inplace=True)
    366         reassign[name] = swap_module(mod, mapping)
    367

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in convert(module, mapping, inplace)
    364         if type(mod) not in SWAPPABLE_MODULES:
    365             convert(mod, mapping, inplace=True)
--> 366         reassign[name] = swap_module(mod, mapping)
    367
    368     for key, value in reassign.items():

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/quantization/quantize.py in swap_module(mod, mapping)
    393             )
    394             device = next(iter(devices)) if len(devices) > 0 else None
--> 395             new_mod = mapping[type(mod)].from_float(mod)
    396             if device:
    397                 new_mod.to(device)

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/nn/quantized/dynamic/modules/linear.py in from_float(cls, mod)
    101         else:
    102             raise RuntimeError('Unsupported dtype specified for dynamic quantized Linear!')
--> 103         qlinear = Linear(mod.in_features, mod.out_features, dtype=dtype)
    104         qlinear.set_weight_bias(qweight, mod.bias)
    105         return qlinear

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/nn/quantized/dynamic/modules/linear.py in __init__(self, in_features, out_features, bias_, dtype)
     33
     34     def __init__(self, in_features, out_features, bias_=True, dtype=torch.qint8):
---> 35         super(Linear, self).__init__(in_features, out_features, bias_, dtype=dtype)
     36         # We don't muck around with buffers or attributes or anything here
     37         # to keep the module simple. *everything* is simply a Python attribute.

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/nn/quantized/modules/linear.py in __init__(self, in_features, out_features, bias_, dtype)
    150             raise RuntimeError('Unsupported dtype specified for quantized Linear!')
    151
--> 152         self._packed_params = LinearPackedParams(dtype)
    153         self._packed_params.set_weight_bias(qweight, bias)
    154         self.scale = 1.0

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/nn/quantized/modules/linear.py in __init__(self, dtype)
     18         elif self.dtype == torch.float16:
     19             wq = torch.zeros([1, 1], dtype=torch.float)
---> 20         self.set_weight_bias(wq, None)
     21
     22     @torch.jit.export

~/.virtualenvs/python3/lib64/python3.6/site-packages/torch/nn/quantized/modules/linear.py in set_weight_bias(self, weight, bias)
     24         # type: (torch.Tensor, Optional[torch.Tensor]) -> None
     25         if self.dtype == torch.qint8:
---> 26             self._packed_params = torch.ops.quantized.linear_prepack(weight, bias)
     27         elif self.dtype == torch.float16:
     28             self._packed_params = torch.ops.quantized.linear_prepack_fp16(weight, bias)

RuntimeError: Didn't find engine for operation quantized::linear_prepack NoQEngine


title: tensor type attributes in bert model returned as string

body:
I am new to nlp and i want to build a bert model for sentiment Analysis so i am following this tuto
https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/
but i am getting the error bellow
bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)

last_hidden_state, pooled_output = bert_model(
   input_ids=encoding['input_ids'],
   attention_mask=encoding['attention_mask']
)
last_hidden_state.shape
pooled_output.shape

When i want to execute last_hidden_state.shape I get an error:
'str' object has no attribute 'shape'
why does it return last_hidden_state and pooled_output as str and not tensors.
Thank you.


title: Pretraining bert model with additional numerical features

body:
I need a suggestion of using an existing bert model which is pre-trained for sentence classification.
the existing model accepts text in form of: “ClC1=CC=CC(Cl)=C1C=O.ClC1=CC=CC(Cl)” which is a chemical reaction for example. Now to enhance the model where I can use continuous values such as time and temperature as features to it to retrain the model on my dataset.
my idea is to use an ensemble approach, where I take the last layer output values of the existing bert model, create another model to just accept the continuous variables and concat them in an ensemble approach similar to this link: Combining Trained Models in PyTorch
Any alternate approaches or suggestions? or links to use as a resource for this implementation
I use pytorch and huggingface transformers library
Thanks
Rahul Raj Devaraja


title: all pairwise dot product pytorch

body:
Is there a built in function to calculate efficiently all pairwaise dot products of two tensors in Pytorch?
e.g.
input - tensor A (shape NxD)
tensor B (shape NxD)
output - tensor C (shape NxN) such that C_i,j = torch.dot(A_i, B_j) ?


title: RuntimeError: CUDA out of memory

body:
I got this Error:
RuntimeError: CUDA out of memory

GPU 0; 1.95 GiB total capacity; 1.23 GiB already allocated 1.27 GiB reserved in total by PyTorch
But it is not out of memory, it seems (to me) that the PyTorch allocates the wrong size of memory. I did change the batch size to 1, kill all apps that use the memory then reboot, and none worked.
This is how I run it, please let me know what info I need to fix it, or where should I check? Thank you.
python train.py --img 416 --batch 16 --epochs 1 \\
--data '../data.yaml' --cfg ./models/yolov4-csp.yaml \\
--weights '' --name yolov4-csp-results  --cache

Using CUDA device0 _CudaDeviceProperties(name='Quadro P620', total_memory=2000MB)

    Namespace(adam=False, batch_size=16, bucket='', cache_images=True, cfg='./models/yolov4-csp.yaml', data='../data.yaml', device='', epochs=1, evolve=False, global_rank=-1, hyp='data/hyp.scratch.yaml', img_size=[416, 416], local_rank=-1, logdir='runs/', multi_scale=False, name='yolov4-csp-results', noautoanchor=False, nosave=False, notest=False, rect=False, resume=False, single_cls=False, sync_bn=False, total_batch_size=16, weights='', world_size=1)
    Start Tensorboard with "tensorboard --logdir runs/", view at http://localhost:6006/
    Hyperparameters {'lr0': 0.01, 'momentum': 0.937, 'weight_decay': 0.0005, 'giou': 0.05, 'cls': 0.5, 'cls_pw': 1.0, 'obj': 1.0, 'obj_pw': 1.0, 'iou_t': 0.2, 'anchor_t': 4.0, 'fl_gamma': 0.0, 'hsv_h': 0.015, 'hsv_s': 0.7, 'hsv_v': 0.4, 'degrees': 0.0, 'translate': 0.5, 'scale': 0.5, 'shear': 0.0, 'perspective': 0.0, 'flipud': 0.0, 'fliplr': 0.5, 'mixup': 0.0}
    Overriding ./models/yolov4-csp.yaml nc=80 with nc=1

                     from  n    params  module                                  arguments
      0                -1  1       928  models.common.Conv                      [3, 32, 3, 1]
      1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]
      2                -1  1     20672  models.common.Bottleneck                [64, 64]
      3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]
      4                -1  1    119936  models.common.BottleneckCSP             [128, 128, 2]
      5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]
      6                -1  1   1463552  models.common.BottleneckCSP             [256, 256, 8]
      7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]
      8                -1  1   5843456  models.common.BottleneckCSP             [512, 512, 8]
      9                -1  1   4720640  models.common.Conv                      [512, 1024, 3, 2]
     10                -1  1  12858368  models.common.BottleneckCSP             [1024, 1024, 4]
     11                -1  1   7610368  models.common.SPPCSP                    [1024, 512, 1]
     12                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]
     13                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
     14                 8  1    131584  models.common.Conv                      [512, 256, 1, 1]
     15          [-1, -2]  1         0  models.common.Concat                    [1]
     16                -1  1   1642496  models.common.BottleneckCSP2            [512, 256, 2]
     17                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]
     18                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
     19                 6  1     33024  models.common.Conv                      [256, 128, 1, 1]
     20          [-1, -2]  1         0  models.common.Concat                    [1]
     21                -1  1    411648  models.common.BottleneckCSP2            [256, 128, 2]
     22                -1  1    295424  models.common.Conv                      [128, 256, 3, 1]
     23                -2  1    295424  models.common.Conv                      [128, 256, 3, 2]
     24          [-1, 16]  1         0  models.common.Concat                    [1]
     25                -1  1   1642496  models.common.BottleneckCSP2            [512, 256, 2]
     26                -1  1   1180672  models.common.Conv                      [256, 512, 3, 1]
     27                -2  1   1180672  models.common.Conv                      [256, 512, 3, 2]
     28          [-1, 11]  1         0  models.common.Concat                    [1]
     29                -1  1   6561792  models.common.BottleneckCSP2            [1024, 512, 2]
     30                -1  1   4720640  models.common.Conv                      [512, 1024, 3, 1]
     31      [22, 26, 30]  1     32310  models.yolo.Detect                      [1, [[12, 16, 19, 36, 40, 28], [36, 75, 76, 55, 72, 146], [142, 110, 192, 243, 459, 401]], [256, 512, 1024]]
    Model Summary: 334 layers, 5.24994e+07 parameters, 5.24994e+07 gradients

    Optimizer groups: 111 .bias, 115 conv.weight, 108 other
    Scanning labels ../train/labels.cache (78 found, 0 missing, 0 empty, 0 duplicate, for 78 images): 100%|█| 78/78 [00:00<0
    Caching images (0.0GB):   3%|█▌                                                          | 2/78 [00:00<00:03, 19.31it/Caching images (0.0GB):  54%|███████████████████████████████▏                          |Caching images (0.0GB): 100%|█████████████████████████████████████████████ █████████████| 78/78 [00:00<00:00, 305.27it/s]
    Scanning labels ../valid/labels.cache (15 found, 0 missing, 0 empty, 0 duplicate, for 15 images): 100%|█| 15/15 [00:00<0
    Caching images (0.0GB): 100%|█████████████████████████████████████████████]█████████████| 15/15 [00:00<00:00, 333.01it/s]

    Analyzing anchors... anchors/target = 4.64, Best Possible Recall (BPR) = 1.0000
    Image sizes 416 train, 416 test
    Using 8 dataloader workers
    Starting training for 1 epochs...

         Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
      0%|                                                                                | 0/5 [00:04<?, ?it/s]
    Traceback (most recent call last):
      File "train.py", line 443, in <module>
        train(hyp, opt, device, tb_writer)
      File "train.py", line 256, in train
        pred = model(imgs)
      File "/home/ctdi/anaconda3/envs/scaled-yolov4.03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
        result = self.forward(*input, **kwargs)
      File "/home/ctdi/content/ScaledYOLOv4/models/yolo.py", line 109, in forward
        return self.forward_once(x, profile)  # single-scale inference, train
      File "/home/ctdi/content/ScaledYOLOv4/models/yolo.py", line 129, in forward_once
        x = m(x)  # run
      File "/home/ctdi/anaconda3/envs/scaled-yolov4.03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
        result = self.forward(*input, **kwargs)
      File "/home/ctdi/content/ScaledYOLOv4/models/common.py", line 47, in forward
        return x + self.cv2(self.cv1(x)) if self.add else self.cv2(self.cv1(x))
      File "/home/ctdi/anaconda3/envs/scaled-yolov4.03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
        result = self.forward(*input, **kwargs)
      File "/home/ctdi/content/ScaledYOLOv4/models/common.py", line 31, in forward
        return self.act(self.bn(self.conv(x)))
      File "/home/ctdi/anaconda3/envs/scaled-yolov4.03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
        result = self.forward(*input, **kwargs)
      File "/home/ctdi/anaconda3/envs/scaled-yolov4.03/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 136, in forward
        self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
      File "/home/ctdi/anaconda3/envs/scaled-yolov4.03/lib/python3.6/site-packages/torch/nn/functional.py", line 2059, in batch_norm
        training, momentum, eps, torch.backends.cudnn.enabled
    RuntimeError: CUDA out of memory. Tried to allocate 44.00 MiB (GPU 0; 1.95 GiB total capacity; 1.23 GiB already allocated; 26.94 MiB free; 1.27 GiB reserved in total by PyTorch)


title: How to find r2score of my PyTorch model for regression

body:
I have a UNet model. I'm trying for a regression model since, in my output, I have different floating values for each pixel. In order to check the r2score, I tried to put the below code in the model class, training_step, validation_step, and test_step.
from pytorch_lightning.metrics.functional import r2score
r2 = r2score(pred, y)
self.log('r2:',r2)
But it's giving the following error

ValueError: Expected both prediction and target to be 1D or 2D tensors, but recevied tensors with dimension torch.Size([50, 1, 32, 32])

How can I check my model fit?


title: Getting negative (inverted) image in Pytorch

body: I want to get the negative of an image straight from the data loader and feed it as a tensor. Are there any libraries that I can use? I have tried torch transforms and didn't find any.


title: How yolo v1 handle negative predicted values?

body:
Let suppose if model generate negative w1 and h1, how can yolo handle this? It will produce a nan value and any other value added to it becomes nan.
So is there any idea, how yolo handle it?


title: PyTorch vectorise sum lookup quantity into buckets

body:
Using PyTorch, I have figured out the following code for calculating totals of an item's property by some "bucket index":

DATASET_SIZE = 10
NUM_BUCKETS = 4
bucket_assignment = torch.tensor([0,1,2,3,0,1,2,3,0,1], dtype = torch.long)
values_to_add = torch.tensor([1,2,3,4,5,6,7,8,9,10], dtype = torch.float)
buckets = torch.zeros(NUM_BUCKETS, dtype = torch.float)
buckets.index_add_(0, bucket_assignment, values_to_add)

# Buckets is now tensor([15., 18., 10., 12.])

In my case this is specifically to check allocation bounds on a problem, and later code checks that no bucket is under- or over- allocated.

I would like to check multiple different possible assignments at once (and later pick a best option, code not shown). I thought I could do this by adding another dimension to bucket_assignment plus to buckets and have each row be a different set of assignments. However, this does not work as intended, because the second argument of index_add_ must be a simple vector, I cannot pass in any higher rank tensor.

E.g.

BATCH_SIZE = 2
DATASET_SIZE = 5
NUM_BUCKETS = 3
bucket_assignment = torch.tensor([[0,1,2,0,1], [1,1,1,2,1]], dtype = torch.long)
values_to_add = torch.tensor([1,2,3,4,5], dtype = torch.float)
buckets = torch.zeros(BATCH_SIZE, NUM_BUCKETS, dtype = torch.float)
buckets.index_add_(0, bucket_assignment, values_to_add)

I would like to get this result:

tensor([[5., 7., 3.], [ 0., 11.,  4.]])

Instead, I get an error:

RuntimeError: invalid argument 3: Index is supposed to be a vector at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:733

That's not unexpected due to the limitations of .index_add, but I don't know how to progress.

I am not sure what other approach would allow me to solve this problem in PyTorch - is there some other torch method I could use that would allow me to achieve the same thing. The main goal here is vectorisation and avoiding loops in Python, as in reality the batch sizes are large and I will be taking advantage of GPU acceleration.


title: Training using pytorch Distributed Data Parallel with world_size 4 on a multi-gpu machine continues even when only two GPU processes started

body: I want to run a test to see how the synchronization works. I assume that at the end of each batch, DDP would wait for the processes on the world_size GPUs to reach the synchronization point like a backward pass to synchronize gradients.  I used a 4-GPU machine and use environment variable CUDA_VISIBLE_DEVICES to make sure only 2 GPU processes can be started.  If only 2 GPU processes started, I assume that at the end of the first batch, the synchronization on the existing 2 GPUS would wait on the other two and time out as the other two never started. What I observed is that the training continued with only 2 GPU processes, even though the world size is 4. How to explain this? Is my understanding not correct?


title: Compiling Pytorch Models to CoreML so they run on the Apple ANE

body:
I benchmarked 2 different Resnet50 Models - the Apple CoreML model, available on the Apple website, and a pretrained Torchvision Resnet50 model which I converted using ONNX (Opset9) and CoreMLTools (iOS Version 13).

I tested both models on a brand new iPhone XR.

Inference Times:

Apple Resnet50: CPU Inference 100ms, GPU Inference 60ms, ANE Inference 15ms
Torchvision Resnet50: CPU Inference 100ms, GPU Inference 60ms, ANE Inference 60ms.

As you can see, the ANE Inference for the Apple version is 4x faster.

Memory Usage:

Apple Resnet50: 46 MB CPU, 92 MB GPU, 35 MB ANE
Torchvision Resnet50: 48 MB CPU, 91 MB GPU, 91 MB ANE

The Apple version uses 3x less memory than the Torchvision one.

I am concluding the Torchvision Model doesn't run on the ANE.

I was wondering if anyone knew how the Apple Repo models were compiled in order to run on the ANE? From the model description, the Apple Models were originally written in Keras, but as far as I know the Keras implementation isn't different from the Pytorch Torchvision one, so I am not sure where the difference comes from.

Thanks!


title: How to write multiple training settings in PyTorch Lightning

body:
I want to iteratively quantize my model. That means that after the normal training loop which is usually implemented in training_step I would like to iteratively quantize some of the parameters and retrain the model for a couple of steps.

I don't really see how this would be done in Lightning. I could add a new instance to the class but then I would have to write the training loop myself again?


title: PyTorch TRAINING A CLASSIFIER tutorial error during CUDA run

body:
I am trying to run PyTorch TRAINING A CLASSIFIER tutorial code with CUDA.
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py
The code is below. The code runs fine for CPU, but gives the following error with GPU-CUDA run:

"Process finished with exit code -1073741819 (0xC0000005)
The error happens during execution of the loss.backward()

I will be very grateful to solution.

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.optim as optim

# Let’s first define our device as the first visible cuda device if we have CUDA available:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# device = torch.device("cpu")

# Assuming that we are on a CUDA machine, this should print a CUDA device:

print(device)
# The output of torchvision datasets are PILImage images of range [0, 1]. We transform them to Tensors of normalized range [-1, 1]. .. note:
#
# If running on Windows and you get a BrokenPipeError, try setting
# the num_worker of torch.utils.data.DataLoader() to 0.

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

# Use CIFAR10
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=0)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=0)

classes1 =trainset.classes
classes =tuple(classes1)
# https://www.w3schools.com/python/python_tuples.asp
# https://www.geeksforgeeks.org/python-convert-a-list-into-a-tuple/
# classes = ('plane', 'car', 'bird', 'cat',
#            'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# functions to show an image

def imshow(img):
    img = img / 2 + 0.5     # unnormalize. The output of torchvision datasets
    # are PILImage images of range [0, 1]. We transform
    # them to Tensors of normalized range [-1, 1]. ..
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4))) # Batch size is 4, Therefore we have 4 images.

# Define a Convolutional Neural Network

# Copy the neural network from the Neural Networks section before and modify it to take 3-channel images (instead of 1-channel images as it was defined).
# Important - this is the way to upgrade my BBOB net.

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
net.to(device)

# Define a Loss function and optimizer
# Let’s use a Classification Cross-Entropy loss and SGD with momentum.

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

# Train the network
# This is when things start to get interesting. We simply have to loop over our data iterator, and feed the inputs to the network and optimize.
# zero the parameter gradients
# optimizer.zero_grad()

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs = inputs.to(device)
        labels = labels.to(device)
        # inputs, labels = data[0].to(device), data[1].to(device)
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

# Let’s quickly save our trained model:

PATH = './cifar_net.pth'
torch.save(net.state_dict(), PATH)

# Test the network on the test data
dataiter = iter(testloader)
images, labels = dataiter.next()

# print images
imshow(torchvision.utils.make_grid(images))
print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

# Next, let’s load back in our saved model (note: saving and re-loading the model wasn’t necessary here, we only did it to illustrate how to do so):

net = Net()
net.load_state_dict(torch.load(PATH))

# Okay, now let us see what the neural network thinks these examples above are:
outputs = net(images)

# The outputs are energies for the 10 classes. The higher the energy for a class, the more the network thinks that the image is of the particular class. So, let’s get the index of the highest energy:

_, predicted = torch.max(outputs, 1)

print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
                              for j in range(4)))

correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (100 * correct / total))

# Hmmm, what are the classes that performed well, and the classes that did not perform well:

class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs, 1)
        c = (predicted == labels).squeeze()
        for i in range(4):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1

for i in range(10):
    print('Accuracy of %5s : %2d %%' % (
        classes[i], 100 * class_correct[i] / class_total[i]))


title: tensor index manipulation with "..."

body:
Hi I'm new to Pytorch and torch tensors. I'm reading yolo_v3 code and encounter this question. I think it relates to tensor indexing with ..., but it's difficult to search ... by google, so I figure to ask it here. The code is:

        prediction = (
            x.view(num_samples, self.num_anchors, self.num_classes + 5, grid_size, grid_size)
            .permute(0, 1, 3, 4, 2)
            .contiguous()
        )
        print (prediction.shape)
        # Get outputs
        x = torch.sigmoid(prediction[..., 0])  # Center x
        y = torch.sigmoid(prediction[..., 1])  # Center y
        w = prediction[..., 2]  # Width
        h = prediction[..., 3]  # Height
        pred_conf = torch.sigmoid(prediction[..., 4])  # Conf
        pred_cls = torch.sigmoid(prediction[..., 5:])  # Cls pred.

My understanding is that the prediction will be a tensor with shape of [batch, anchor, x_grid, y_grid, class]. But what does prediction[..., x] do (x=0,1,2,3,4,5)? Is it similar as numpy indexing of [:, x]? If so the calculation of x, y, w, h, pred_conf and pred_cls don't make sense.


title: How to avoid iterating over Dataloader while resuming training in Huggingface Trainer class?

body:
I'm currently using Huggingface's Trainer class to train Distillbert for a regression problem using a custom loss function. I'm using their checkpoints to resume training due to the ephemeral nature of compute / unexpected errors.
The issue I'm facing is that each time I resume training from a checkpoint as per their Trainer class via the model_path in the Trainer.train() method, I noticed that the class iterates over the dataloader until it reaches the iteration count as saved in the checkpoint (see the lines from the Trainer class that match the issue).
This might usually not be a issue, but due to the nature of my dataloader's collate function and the size of the dataset, iterating for such a duration without any training is pretty expensive and slows down the overall training.
I planned on utilizing a custom sampler class something along the lines of this with a parameter to resume the indices from a given location but that too seems quite the hack for the given problem.
What could be an alternative that I could try to save on this wasted compute cycles?


title: Equivalent of np.multiply.at in Pytorch

body:
Are there equivalent of np.multiply.at in Pytorch? I have two 4d arrays and one 2d index array:
base = torch.ones((2, 3, 5, 5))
to_multiply = torch.arange(120).view(2, 3, 4, 5)
index = torch.tensor([[0, 2, 4, 2], [0, 3, 3, 2]])

As shown in this question I asked earlier (in Numpy), the row index of the index array corresponds to the 1st dimension of base and to_multiply, and the value of the index array corresponds to the 3rd dimension of base. I want to take the slice from base according to the index and multiply with to_multiply, it can be achieved in Numpy as follows:
np.multiply.at(base1, (np.arange(2)[:,None,None],np.arange(3)[:,None],index[:,None,:]), to_multiply)

However, now when I want to translate this to PyTorch, I cannot find an equivalent of np.multiply.at in Pytorch, I can only find the "index_add_" method but there is no "index_multiply". And I want to avoid doing explicit for loop.
So how can I achieve above in PyTorch? Thanks!


title: Low Validation Score on Pretrained Alexnet from Pytorch models for ImageNet 2012 dataset

body:
I am using pre-trained AlexNet network to validate some prior work.
The code is as follows:
import os
import torch

import torchvision
import torchvision.datasets as datasets
import torchvision.models as models
import torchvision.transforms as transforms

model = torch.hub.load('pytorch/vision:v0.6.0', 'alexnet', pretrained=True)
model.eval()

batchsize = 50000
workers = 1
dataset_path = 'data/imagenet_2012/'
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
val_data = datasets.ImageFolder(root=os.path.join(dataset_path, 'val'), transform=transforms.Compose( [transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), normalize,]))
val_loader = torch.utils.data.DataLoader(val_data, batch_size=batchsize, num_workers=workers)

batch = next(iter(val_loader))
images, labels = batch

with torch.no_grad():
    output = model(images)

for i in output:
    out_soft = torch.nn.functional.softmax(i, dim=0)
    print(int(torch.argmax(out_soft)))

When I execute this and compare with ILSVRC2012_validation_ground_truth.txt, I get top-1 accuracy of 5% only.
What am I doing wrong here?
Thank you.


title: Saving from colab to google drive with pytorch makes black image

body:
I mounted a google drive with Colab:
from google.colab import drive
drive.mount('/content/drive')

this works fine. Using pytorch to save a tensor image to the drive as a png:
robin = image.type(torch.uint8)
torchvision.io.write_png(robin, '/content/drive/My Drive/ImgFolder/robin.png')

saves a file called robin.png in the drive, which is just a black square.
viewing the tensor image with plot.imshow, i can see that it is an image of a robin, but for some reason saving it to drive makes it black. The image is from the STL10 dataset.
Here is a link to a notebook which reproduces the problem: https://colab.research.google.com/drive/1ayAs7HmzuA-eRBnat9HoziMM4yDdlyTA?usp=sharing