# Inplace operation in pytorch

## Basic introduction

An inplace operation is something which modifies the data directly. For example

```
x += 1
x[0] = 1
```

But

```
x = x + 1
```

is not inplace operation. Since it puts the results in a new variable and takes the objects pointed to by x.

## Some Bugs when use pytorch

- The first bug we usually meet is
`RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.`

This is because some function requires the variables to not change for computing the gradient. But you use some inplace operations and change the value.

The following table show some common functions.

Input needs to not change in computing gradient multiplication, relu Output needs to not change in computing gradient sigmoid A example is

`def forward(self,a): b = Variable(torch.Tensor(10,1)) y = 0 b = torch.mm(self.param,a) for i in range(10): b[i] = b[i] + 1 #inpalce operation y = y + torch.mm(self.param,b) return y`

Assinment to

`b[i]`

is a inplace operation. It will change the data in`b`

. The computation of the gradient of`torch.mm`

need the input to not change. Hence, it will generate the error above.To avoid this error, we should prevent using inplace operation in building the network. For example, we can use tensor operation. The code above can be modified into the following one.

`def forward(self,a): b = Variable(torch.Tensor(10,1)) y = 0 b = torch.mm(self.param,a) for i in range(10): bione = zeros(10,1) bione[i] = 1 b = b + bione #out-of-palce operation y = y + torch.mm(self.param,b) return y`

Although

`bione[i] = 1`

is also an inplace operation, it is not used to compute the gradient.`b=b+bione`

is a out-of-place operation, it does not change the`b`

in last iteration. So this code performs well.Another solution is to use

`clone()`

, which will generate a new variable that copys the origin one, i.e., we can use the following code`def forward(self,a): b = Variable(torch.Tensor(10,1)) y = 0 b = torch.mm(self.param,a) for i in range(10): b[i] = b[i] + 1 #inpalce operation y = y + torch.mm(self.param,b.clone()) return y`

In the computation of the gradient of

`torch.mm`

, the new varaible generated by`b.clone()`

instead of`b`

is used. Hence, although`b`

is changed by an inplace operation, it does not affect the calculus of gradients. - Another similar error is
`RuntimeError: a leaf Variable that requires grad has been used in an in-place operation`

A leaf variable is a variable you create directly, not as the results of some other operation. This error is generated since you use an inplace operation in a leaf variable.

For example,

`x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4])) # leaf variable x += 1 y = x + 1 # not a leaf variable`

`x`

is leaf variable and`+=`

is an inplace operation, so it generates the error above.The solution is to avoid using inplace operation

`x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4])) # leaf variable x = x + 1 #out-of-place operation y = x + 1 # not a leaf variable`

or using

`clone()`

`x = torch.autograd.Variable(torch.Tensor([1, 2, 3, 4])) # leaf variable x2 = x.clone() x2 += 1 y = x2 + 1 # not a leaf variable`