First 5 things to start learning PyTorch Tensors in Sagemaker Notebooks
Let’s begin our TorchAdventure in AWS! with this 11 basic functions distributed in the following sections:
Creating tensors: Empty() and Zeros()
Measuring tensors: Size() and Shape(), Sum() and Dimensions
Changing and copying tensors: Reshape(), view() and randn()
Modifying tensors: Unsqueeze()
Comparing tensors: Element-Wise Equality: Eq
First lets spin up a new sagemaker instance:
Now go to Jupyter Lab and import or create a new Notebook with this kernel for Pytorch:
Select the appropiate kernel, for this case it could be: conda_pytorch_p36
Here is where I will start from scratch.
Before we begin, let’s install and import PyTorch
*# Uncomment and run the appropriate command for your operating system, if required*
# Linux / Binder
# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
# Windows
# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
# MacOS
# !pip install numpy torch torchvision torchaudio
Import Pytorch from the Notebook instance
**import** **torch**
Function 1 — Empty and Zeros — how to initialize tensors
We need to start working with the basics pytorch functions, and the first thing is create our matrix
*# Creates a 3 x 2 matrix which is empty*
a = torch.empty(3, 2)
print(a)
tensor([[1.5842e-35, 0.0000e+00],
[4.4842e-44, 0.0000e+00],
[ nan, 0.0000e+00]])
Here is how PyTorch is allocating memory for this tensor. Whatever, it will not erase anything previous content in the memory.
by default, when you initializes a tensor is used the float32 dtype. you can review it here: https://pytorch.org/docs/stable/generated/torch.set_default_tensor_type.html#torch.set_default_tensor_type
But you can also start working with torch.zeros
*# Applying the zeros function and *
*# storing the resulting tensor*
a = torch.zeros([3, 5])
print("a = ", a)
b = torch.zeros([2, 4])
print("b = ", b)
c = torch.zeros([4, 1])
print("c = ", c)
d = torch.zeros([4, 4, 2])
print("d = ", d)
the result will be:
a = tensor([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])
b = tensor([[0., 0., 0., 0.],
[0., 0., 0., 0.]])
c = tensor([[0.],
[0.],
[0.],
[0.]])
d = tensor([[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]],
[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]],
[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]],
[[0., 0.],
[0., 0.],
[0., 0.],
[0., 0.]]])
this tensor is filled with zeros, so PyTorch allocates memory and zero-initializes the tensor elements inside
You cannot change the way a tensor is created, if you create a zeros tensor, make sure is not referenced to any other.
*# correctly initialized*
a = torch.empty(3,3)
print(a)
#also correct
b = torch.empty(1,2,3)
print(b)
the results will be:
tensor([[2.4258e-35, 0.0000e+00, 1.5975e-43],
[1.3873e-43, 1.4574e-43, 6.4460e-44],
[1.4153e-43, 1.5274e-43, 1.5695e-43]])
tensor([[[2.3564e-35, 0.0000e+00, 1.4013e-45],
[1.4574e-43, 6.4460e-44, 1.4153e-43]]])
Lets see now, how you cannot use the zeros function:
*# incorrect reference, you must create a new one*
c = torch.zeros(b,1)
print("c = ", c)
output:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-35-c044e2995879> in <module>()
** 1** # incorrect reference, you must create a new one
----> 2 c = torch.zeros(b,1)
** 3** print("c = ", c)
TypeError: zeros() received an invalid combination of arguments - got (Tensor, int), but expected one of:
* (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
* (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
lets review the zero method closely:
Our documentation says:
Syntax: torch.zeros(size, out=None)
Parameters:
size: a sequence of integers defining the shape of the output tensor
out (Tensor, optional): the output tensor
Return type: A tensor filled with scalar value 0, of same shape as size.
torch.zeros and torch.empty are the first functions to start working with pytorch tensors and learning a little bit of matrix and vectors
Function 2 — Tensor Size, Shape and Dimension Operations
Lets understand dimensions in Pytorch.
Now lets create some tensors and determine the size of every one
**import** **torch**
*# Create a tensor from data*
c = torch.tensor([[3.2 , 1.6, 2], [1.3, 2.5 , 6.9]])
print(c)
print(c.size())
output
tensor([[3.2000, 1.6000, 2.0000],
[1.3000, 2.5000, 6.9000]])
torch.Size([2, 3])
Lets see torch.shape and take a closer look at how size is given here
Now in the next example lets use shape functions to get the size of a tensor
In [ ]:
x = torch.tensor([
[1, 2, 3],
[4, 5, 6]
])
x.shape
torch.Size([2, 3])
Out[ ]:
torch.Size([2, 3])
Shape and Size give us the same correct dimensions of the tensor.
in this case we have a 3D-tensor (with 3 dimensions)
Dimension 0 Dimension 1 and Dimension 2
lets create a new tensor:
In [ ]:
y = torch.tensor([
[
[1, 2, 3],
[4, 5, 6]
],
[
[1, 2, 3],
[4, 5, 6]
],
[
[1, 2, 3],
[4, 5, 6]
]
])
y.shape
Out[ ]:
torch.Size([3, 2, 3])
Lets se how we can make operations using a 3d tensor now for every dimension layer and see how it behaves
In [ ]:
sum1 = torch.sum(y, dim=0)
print(sum1)
sum2 = torch.sum(y, dim=1)
print(sum2)
sum3 = torch.sum(y, dim=2)
print(sum3)
tensor([[ 3, 6, 9],
[12, 15, 18]])
tensor([[5, 7, 9],
[5, 7, 9],
[5, 7, 9]])
tensor([[ 6, 15],
[ 6, 15],
[ 6, 15]])
we can see now a 3d tensor is more complicated as we advance, and we can perform custom operations within every dimension
its limited right now to 3 dims so we cannot perform this:
In []
sum2 = torch.sum(y, dim=3)
print(sum1)
Out [ ]:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-56-7caf723c6102> in <module>()
----> 1 sum2 = torch.sum(y, dim=3)
** 2** print(sum1)
IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)
to accomplish adding more layers of dimensions we can review the unsqueeze functions → .unsqueeze() but for now let’s go to the next basic function
Function 3 — Reshape, View and Random
There is a function in NunPy called ndarray.reshape() for reshaping an array.
Now in pytorch, there is torch.view(tensor) for the same purpose, but at the same time, there is also a torch.reshape(tensor).
let’s figure out the differences between them and when you should use either of them.
First of all we are going to use a new functions for randomize our tensor.
In [ ]:
**import** **torch**
x = torch.randn(5, 3)
print(x)
Out []
tensor([[-0.5793, 0.6999, 1.7417],
[-0.9810, 0.0626, 0.4100],
[-0.6519, -0.0595, -1.2156],
[-0.3973, -0.3103, 1.6253],
[ 0.2775, -0.0045, -0.2985]])
this is basic usage of torch.randm, so now lets use View from another variable “y” and describe all elements
In [ ]:
*# Return a view of the x, but only having *
*# one dimension and max number of elements*
y = x.view(5 * 3)
*#lets see the size of every tensor*
print("lets see the size of every tensor")
print('Size of x:', x.size())
print('Size of y:', y.size())
*#and the elements of very tensor to compare*
print("and the elements of very tensor to compare")
print("X:", x)
print("Y:", y)
Out []
lets see the size of every tensor
Size of x: torch.Size([5, 3])
Size of y: torch.Size([15])
and the elements of very tensor to compare
X: tensor([[-0.5793, 0.6999, 1.7417],
[-0.9810, 0.0626, 0.4100],
[-0.6519, -0.0595, -1.2156],
[-0.3973, -0.3103, 1.6253],
[ 0.2775, -0.0045, -0.2985]])
Y: tensor([-0.5793, 0.6999, 1.7417, -0.9810, 0.0626, 0.4100, -0.6519, -0.0595,
-1.2156, -0.3973, -0.3103, 1.6253, 0.2775, -0.0045, -0.2985])
take a look at Y tensor, it only has 1 dimension. so Viewing another tensor may be difficult for some operations
Now lets use Reshape to replicate the exact dimensions of X
In [ ]:
*# Get back the original tensor with reshape()*
z = y.reshape(5, 3)
print(z)
tensor([[-0.2927, 0.0329, 0.8485],
[ 1.9581, 0.8313, -0.1529],
[-0.2330, -0.1887, 1.8206],
[ 1.5252, 1.0909, 0.0547],
[-0.1231, -0.4238, -0.6724]])
we cannot only reshape the original, we can also change the dimensions with some limited actions related to the maximum elements:
first lets reshape to 1 more dimension
In [ ]:
*# Get back the original tensor with reshape()*
z = y.reshape(15)
print(z)
#reshaping 15 elements, 1 dim
z = y.reshape(3*5)
print(z)
#reshaping in different order, 3 dimensions
z = y.reshape(3,5)
print(z)
#Reshaping with more dimensions but its 15 elements always
z = y.reshape(3,5,1)
print(z)
tensor([-0.5793, 0.6999, 1.7417, -0.9810, 0.0626, 0.4100, -0.6519, -0.0595,
-1.2156, -0.3973, -0.3103, 1.6253, 0.2775, -0.0045, -0.2985])
tensor([-0.5793, 0.6999, 1.7417, -0.9810, 0.0626, 0.4100, -0.6519, -0.0595,
-1.2156, -0.3973, -0.3103, 1.6253, 0.2775, -0.0045, -0.2985])
tensor([[-0.5793, 0.6999, 1.7417, -0.9810, 0.0626],
[ 0.4100, -0.6519, -0.0595, -1.2156, -0.3973],
[-0.3103, 1.6253, 0.2775, -0.0045, -0.2985]])
tensor([[[-0.5793],
[ 0.6999],
[ 1.7417],
[-0.9810],
[ 0.0626]],
[[ 0.4100],
[-0.6519],
[-0.0595],
[-1.2156],
[-0.3973]],
[[-0.3103],
[ 1.6253],
[ 0.2775],
[-0.0045],
[-0.2985]]])
Now lets reshape exceeding the number of elements in the tensor:
In [ ]:
z = y.reshape(16)
print(z)
Out []
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-78-c7ae174fce73> in <module>()
----> 1 z = y.reshape(16)
** 2** print(z)
RuntimeError: shape '[16]' is invalid for input of size 15
it will fail also for z = y.reshape(3*6) or putting more elements that does not exist in the tensor.
Now let’s keep going to the next section.
Function 4 — Unsqueeze()
Mainly, it allows us to add more dimensions at specific index you define.
lets take a look:
In [ ]:
**import** **torch**
#dim=1, that is (3)
x = torch.tensor([1, 2, 3])
print('x: ', x)
print('x.size: ', x.size())
#x1 becomes a matrix of (3,1)
x1 = torch.unsqueeze(x, 1)
print('x1: ', x1)
print('x1.size: ', x1.size())
Out []
x: tensor([1, 2, 3])
x.size: torch.Size([3])
x1: tensor([[1],
[2],
[3]])
x1.size: torch.Size([3, 1])
x2: tensor([[1, 2, 3]])
x2.size: torch.Size([1, 3])
Our initial tensor is Tensor([1,2,3]), and the output size is [3].
And then we proceed with adding 1 dimenson with unsqueeze operation, namely torch.unsqueeze(x, 1), the size of x1 is [3,1].
In [ ]:
*#x2 becomes a matrix of (1,3)*
x2 = torch.unsqueeze(x, 0)
print('x2: ', x2)
print('x2.size: ', x2.size())
Out []
x2: tensor([[1, 2, 3]])
x2.size: torch.Size([1, 3])
When we perform torch.unsqueeze(x, 0), the size of x2 is [1,3].
In [ ]:
*# Example 3 - breaking (to illustrate when it breaks)*
print(x.unsqueeze())
Out []
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-89-5a320a828907> in <module>()
** 2**
** 3**
----> 4 print(x.unsqueeze())
TypeError: unsqueeze() missing 1 required positional arguments: "dim"
We must specified the dimension correctly, although we are just adding 1 dim, it is necessary to put like this: x.unqueeze(0)
Function 5 — Torch Eq (Element Wise equality)
This function is under comparison category and it computes equality in element-wise and returns a boolean tensor. True if equal, False otherwise.
Lets review how we can operate with different sizes of tensors:
In [ ]:
*# Example 1 - working *
x1 = torch.tensor([[1, 2], [3, 4.]])
x2 = torch.tensor([[2, 2], [2, 5]])
x3 = torch.randn(3,5)
#size x1 and z2
print(x1.size())
print(x2.size())
# tensors 1 and 2
print(x1)
print(x2)
#size x3
print(x3.size())
#tensors 3
print(x3)
torch.eq(x1,x2)
torch.Size([2, 2])
torch.Size([2, 2])
tensor([[1., 2.],
[3., 4.]])
tensor([[2, 2],
[2, 5]])
torch.Size([3, 5])
tensor([[-1.3040, -0.4658, -0.5269, 0.7409, 0.9135],
[ 1.0780, 2.0584, -0.9629, -1.1412, -0.3105],
[ 0.3613, -1.4196, 2.1145, 0.3649, 0.2037]])
Out[ ]:
tensor([[False, True],
[False, False]])
x1 and x3 have the same size, but x3 is [3,5], has bigger size.
comparing x1 and x2 is Ok.
In [ ]:
*# Example 2 - working (with broadcasting)*
x4 = torch.tensor([[1, 2], [3, 4]])
print(x4.size())
x5 = torch.tensor([2, 5])
print(x5.size())
torch.eq(x4, x5)
torch.Size([2, 2])
torch.Size([2])
Out[ ]:
tensor([[False, False],
[False, False]])
we can also compare, different sizes only if the seconf value, in this case x5, that has size of [2] is broadcastable with the frist one thst is [2,2]
In [ ]:
*# Example 3 - breaking (to illustrate when it breaks)*
x6 = torch.tensor([[0, 2, 4], [3, 4, 5]])
print(x6.size())
x7 = torch.tensor([[2, 3], [2, 4]])
print(x7.size())
torch.eq(x6, x3)
torch.Size([2, 3])
torch.Size([2, 2])
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-98-ac2ad1ecd5b0> in <module>()
5 print(x7.size())
6
----> 7 torch.eq(x6, x3)
RuntimeError: The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 1
finally, we cant compare different sizes if the second arguments shape is not broadcastable with the first argument.
Conclusion
we review 5 basic topics covering more than 10 PyTorch functions.
in the next post i’ll talk about Linear Regression.
Reference Links
Provide links to your references and other interesting articles about tensors
Official documentation for tensor operations: https://pytorch.org/docs/stable/torch.html
Squeeze and unsqueeze functions here: https://www.programmersought.com/article/65705814717/
Understanding dimensions in Pytorch: https://towardsdatascience.com/understanding-dimensions-in-pytorch-6edf9972d3be
#aws #reinvent2020 #awsperu #awsugperu #awscloud
Carlos Cortez — AWS UG Perú Leader / AWS ML Community Builder ccortez@aws.pe @ccortezb Podcast: imperiocloud.com @imperiocloud twitch.tv/awsugperu cennticloud.thinkific.com Carlos Cortez ML Engineer — Senior Cloud Architect — Pro Services Cluster — BGH Tech Partner |… Hello Everyone, I’m the Founder of AWS User Group Perú 🇵🇪 (2014) and also an AWS Solutions Architect Certified…linkedin.com