In case you are interested in hearing about Hugging Face and SageMaker, I spoke in depth a few days ago about this topic and its recent alliance with AWS here:

The learning curve

The learning curve is long and can get complex if we want to. For that there are flows and frameworks that will help us know where we are right now.

• Natural Language Processing (NLP) flow *

Many times we skip key steps before we really understand each of these procedures. For example, data ingestion, web scraping, text analytics, EDA (exploratory analysis, and much more)

Remember CRISP-DM?

It is a framework for data mining, which is now known more as Machine Learning, Deep Learning, Data Science, etc.

Essentials to get started on AWS

Always to each place we go, we keep this pair in our pocket:

• Python 3.x + boto3 *

Boto3 , is the AWS python library that will give us access to run and manage any resource within the cloud, as simple as import boto3, and have your private keys configured or your roles correctly created .

Pre-trained models

Pre-trained models help us to shorten research, training and analysis time, and not start inventing what already exists. and for this we can use cloud services.

Amazon Comprehend (Applied NLP) , It is a SaaS type service, which means that it is ready to consume an API with the AWS boto3 library and begin to obtain insights from the data. It contains models already trained by Amazon Web Services, which we can reuse to start our own, and we will start knowing this great service and start building interesting applications using other AWS services.

Hugging Face (advanced) , later we will play with transformers, tokenizers, and more with these Deep Learning models of Hugging Face.

Build your own models:

Once we have a good understanding of NLP's self-managed cloud services, we can move to the next level with Amazon Sagemaker.

Or, with your own containers, which we will also learn, or locally

NLP Open Source Libraries

I will mention these for now and we will surely know more than 2 of this list in the coming months:

• NLTK

• Spark NLP

• CoreNLP

• SpaCy

• Pytorch

• Tensorflow

• etc..

Tools and IDEs

From now on we will begin to know and learn step by step, what it is like to work in the cloud with an IDE for Data Science, and learn about AI services on AWS.

I recommend taking a look at the following, some paid and some free.

1. Sagemaker Studio / Sagemaker Notebooks

2. Jupyter Notebooks / Jupyter Lab

3. Databricks

4. Google Colab

5. among others ...

To start for free, I suggest Databricks and Colab , the first one, has a community platform that allows you to create free clusters and they are eliminated if you do not use it for limited times, and Google's Colab, you can run it directly on your Google Drive, at no cost.

Deployment of my models in the cloud

It is not yet time to think about deploying, and yes, we will find many, many ways to do it, from the oldest to the most modern.

• Traditional servers (VMs, Bare Metal)

• Containers (such as AWS ECS, EKS, Fargate)

• Without Servers (Functions as a Service)

This is a very good way to start, putting together serverless and Sagemaker.

Think Serverless First

Now we can start working without having to run servers using AWS Lambda, AWS Step Functions, Sagemaker Pipelines, AWS CodePipeline, which are services already deployed in the cloud that we can use to speed up our development and start-up.

In the following posts we learn how to deploy with AWS Lambda and Amazon Comprehend.

Amazon Comprehend and its superpowers

Can you imagine something as simple as writing a few lines of code to create a word detector and phrase extraction in just a few minutes instead of many hours of research? It is possible if you start to take advantage of Amazon Comprehend, a little Python and a couple of drops of Serverless

Phrase extraction:

We are using Python 3.6 here with boto3, the library

response **=** client**.**detect_key_phrases(

Text=’string’,

LanguageCode=’en’|’es’

)

Extraction of Entities, sentiment, language, syntax, topics, and document classification, are some of the additional things that we are going to do with Comprehend, as well as a specific version for Medicine, called Amazon Comprehend Medical

What do we continue with?

Data ingestion and text Analytics is part of the important bases to understand NLP, so in the next post, we will start to build our first data ingestion of a web, to be able to obtain data, texts, titles, in a data frame to play.

Next post: (on air on April 02)

(Post Original: Medium and Dev.cortez.cloud )

If you liked this post, give it a like, share and comment.

Learn by breaking things.

Subscribe to my channel, Breaking the Cloud and be Up to date with AWS at https://cortez.cloud

⭐Subscribe to my channel : http://bit.ly/aldiaconaws

videos, AWS news, analytics, reviews, demos, workshops

🔥🔥 Sígueme en mis redes 🔥🔥

follow me:

🦜 My Twitter: https://twitter.com/ccortezb

📺 Youtube Channel: http://bit.ly/aldiaconaws

📺 AWSUGPerú: https://www.youtube.com/awsusergroupperuoficial

📟 My Facebook: https://www.facebook.com/ccortezb/

🤳 My Instagram: ccortezbazan

📜My AWS Courses: https://cennticloud.thinkific.com

🕮 My blog — cortez.cloud

Thank you very much, I hope we see each other again

🔥🔥 About me 🔥🔥

Cortez.Cloud/whoami

Let me present to you, my little personal website https://www.Cortez.Cloud called “Breaking the Cloud”.

I will continue to create content each week from AWS on Al / ML, Serverless, Security and how to break the rules!

Also my next initiatives, workshops, courses, free videos, awsugperu and more.

📢 Leveleando nuestro guerrero 3 niveles hoy!

Soy Carlos Cortez y bienvenidos a la tercera entrega de estas series, pero, ¿qué me estás contando? Vamos a por 3 niveles más!, hoy aprenderemos las bases de Comprehend y de cómo analizar sentimientos, así como extraer las entidades de los textos, usando un modelo pre entrenado de Amazon Comprehend.

Al mismo tiempo, hago Referencia a unas series en paralelo sobre Serverless, llamadas: Away from Servers, porque hoy también repasamos API rest en AWS y codificaremos una plantilla de AWS SAM, el marco de trabajo de amazon web services para gestionar la configuración de nuestras aplicaciones sin servidor.

Además de estos 4 servicios, aprenderemos más sobre boto3, y Python.

Fuente: Imagen creada por Carlos Cortez, NLP Series parte 3

🧩 Pasar a level 2 es tarea sencilla

Puedes elegir entre, desarrollar el nivel 2, nivel 3 o nivel 4 que veremos más abajo en este episodio, en los 3 lograrás el mismo resultado, que es aprender sobre Amazon Comprehend.

OJO: Siempre es necesario un rol, un permiso, una política cuando creamos un Lambda, en este caso estaremos usando Full Access para Comprehend en IAM.

El nivel 2, trata principalmente de crear una función lambda en la consola de AWS directamente y conectarla con Amazon Comprehend, luego definir un evento de prueba con un JSON y empezar a jugar.

Fuente: Imagen creada por Carlos Cortez, AWS Warrior Level 2

Usemos esta función para darle vida a nuestro Lambda,

import json
import boto3

#Using boto3 to call the Comprehend API
client = boto3.client('comprehend')

#Lambda function to work with Comprehend
def lambda_handler(event, context):

    print("event puro:", event)
    #Accessing data
    puro = json.dumps(event) #intentando cambiar aqui sino vuelvo a la linea 13
    text = puro[0]
    json_object = json.loads(puro)
    print("texto a analizar: ", json_object["body"])
    Body = json.dumps(json_object["body"])
    print("Body:", Body)


    #Sentiment Analysis
    print("iniciando analisis de sentimiento....")
    sentiment = client.detect_sentiment(Text = Body, LanguageCode = 'en') #API call for sentiment analysis
    sentRes = sentiment['Sentiment'] #Positive, Neutral, or Negative
    sentScore = sentiment['SentimentScore'] #Percentage of Positive, Neutral, and Negative
    print(sentRes)
    print(sentScore)

    print("iniciando extraccion de entidades....")
    #Entity Extraction
    entities = client.detect_entities(Text = Body, LanguageCode = 'en') #API call for entity extraction
    entities = entities['Entities'] #all entities
    print(entities)
    textEntities = [dict_item['Text'] for dict_item in entities] #the text that has been identified as entities
    typeEntities = [dict_item['Type'] for dict_item in entities] #the type of entity the text is
    print(textEntities)
    print(typeEntities)


    return {
        'statusCode': 200,
        'body': str(sentiment) + str(entities) #body returned from our function
    }

Explicación:

1. LÍNEA 12 extraemos el JSON del evento

2. LÍNEA 20, detectamos el sentimiento sea positivo negativo o neutral

3. LÍNEA 28: extraemos las entidades detectadas definidas por AWS

Las pruebas aquí son más sencillas, usando la IDE de Cloud9 en Lambda directo en la consola:

Y agregar el evento, es muy rápido, en el level 3 y 4 verán maneras más óptimas de realizar esta prueba. Pueden pasar directo allá.

🧩 Si pasaste el nivel 2, ahora en nivel 3 completamos el Backend

Al adicionar la capa de API rest vamos a darle más cuerpo a nuestra solución. Además como arquitectos de soluciones que somos o queremos ser, tenemos que construir aplicaciones resilientes, escalables y altamente disponibles. Para ello implementamos una capa de API que nos permita escalar. Más adelante nos servirá para agregar seguridad como Cognito, CORS, Caché, custom headers.

Fuente: Imagen creada por Carlos Cortez, AWS Warrior Level 3

No vamos a detallar esta etapa, porque será mucho más genial en nivel 4, pero sí mencionar que probar con API Gateway es mucho más eficiente desde un terminal o una aplicación como Postman. (https://www.postman.com/)

🧩 Nivel 4: ¡Evolucionando a SAM!

Llegó la hora, dejen de ver y pongámonos a trabajar aquí, que esta es la parte más interesante.

Los marcos de trabajo para aplicaciones sin servidor, están tomando un lugar más frecuente en el desarrollo de servicios, AWS SAM llegó algo tarde, y es la opción nativa dentro de los servicios de Amazon Web Services que compite con grandes como Serverless Framework, Chalice, Amplify, pulumi, etc.

No se preocupen, en las series dedicadas a Serverless, nos pelearemos con otros frameworks y veremos qué tan divertido es usarlos! Ahora, me enfocaré en subir a mi nivel 4!

Fuente: Imagen creada por Carlos Cortez, AWS Warrior Level 4

Cada vez que hago un cambio, se construye toda mi aplicación desde cero. Invierto menos tiempo configurando la consola, que no debería tocarla ni verla más, y me dedico a mejorar el código e integrarlo con más servicios.

🧩 Actualizar sam-cli

Puedes usar una versión más actual si lo prefieres, en este caso:

➜  comprehend-nlp01 brew upgrade aws-sam-cli

Warning: aws/tap/aws-sam-cli 1.13.2 already installed

crear ambiente virtual de python 3.6, (yo tengo múltiples versiones)

➜  comprehend-nlp01 virtualenv -p `which python3.6` nlp01


created virtual environment CPython3.6.5.final.0-64 in 486ms
  creator CPython3Posix(dest=/Users/carlos/repos/cc/nlp/comprehend-nlp01/nlp01, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/carlos/Library/Application Support/virtualenv)
    added seed packages: pip==20.1.1, setuptools==47.3.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Iniciamos un template vacío de SAM para Python 3.6

**➜ comprehend-nlp01 source nlp01/bin/activate**

(nlp01) ➜ comprehend-nlp01 python — version

Python 3.6.5

🧩 Inicializamos SAM

(nlp01) ➜ comprehend-nlp01 sam init

Elegimos un template por default:

Which template source would you like to use?

1 — AWS Quick Start Templates

2 — Custom Template Location

Choice: 1

Which runtime would you like to use?

1 — nodejs12.x

2 — python3.8

3 — ruby2.7

4 — go1.x

5 — java11

6 — dotnetcore3.1

7 — nodejs10.x

8 — python3.7

9 — python3.6

10 — python2.7

11 — ruby2.5

12 — java8

13 — dotnetcore2.1

14 — dotnetcore2.0

15 — dotnetcore1.0

Runtime: 9

Project name [sam-app]: comprehend-01-nlpseries-ep-3

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:

1 — Hello World Example

2 — EventBridge Hello World

3 — EventBridge App from scratch (100+ Event Schemas)

4 — Step Functions Sample App (Stock Trader)

Template selection: 1

 — — — — — — — — — — — -

Generating application:

 — — — — — — — — — — — -

Name: comprehend-01-nlpseries-ep-3

Runtime: python3.6

Dependency Manager: pip

Application Template: hello-world

Output Directory: .

Next steps can be found in the README file at ./comprehend-01-nlpseries-ep-3/README.md

Errores de compatibilidad a la hora de crear los templates

Error: Permission denied @ apply2files — /usr/local/lib/node_modules/@aws-amplify/cli/node_modules/extglob/lib/.DS_Store

No debemos darle permisos a todo, salvo lo necesario

Solución:

sudo chown -R ${LOGNAME}:staff /usr/local/lib/node_modules

Aquí vamos a modificar varios archivos,

App.py es el archivo principal, que será el Lambda que usamos, el mismo del nivel 2 y 3!

import json
import boto3

#Using boto3 to call the Comprehend API
client = boto3.client('comprehend')

#Lambda function to work with Comprehend
def lambda_handler(event, context):

    print("event puro:", event)
    #Accessing data
    puro = json.dumps(event) #intentando cambiar aqui sino vuelvo a la linea 13
    text = puro[0]
    json_object = json.loads(puro)
    print("texto a analizar: ", json_object["body"])
    Body = json.dumps(json_object["body"])
    print("Body:", Body)


    #Sentiment Analysis
    print("iniciando analisis de sentimiento....")
    sentiment = client.detect_sentiment(Text = Body, LanguageCode = 'en') #API call for sentiment analysis
    sentRes = sentiment['Sentiment'] #Positive, Neutral, or Negative
    sentScore = sentiment['SentimentScore'] #Percentage of Positive, Neutral, and Negative
    print(sentRes)
    print(sentScore)

    print("iniciando extraccion de entidades....")
    #Entity Extraction
    entities = client.detect_entities(Text = Body, LanguageCode = 'en') #API call for entity extraction
    entities = entities['Entities'] #all entities
    print(entities)
    textEntities = [dict_item['Text'] for dict_item in entities] #the text that has been identified as entities
    typeEntities = [dict_item['Type'] for dict_item in entities] #the type of entity the text is
    print(textEntities)
    print(typeEntities)


    return {
        'statusCode': 200,
        'body': str(sentiment) + str(entities) #body returned from our function
    }

Luego, el template.yaml que es dónde se definirá la estructura de nuestra aplicación serverless con API Gateway y Lambda.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  comprehend-01-nlpseries-ep-3
  Sample SAM Template for comprehend-01-nlpseries-ep-3
Globals:
  Function:
    Timeout: 3

Resources:
  Comprehend01nlpseriesep3Function:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: comprehend_01_nlpseries_ep_3/
      Handler: app.lambda_handler
      Runtime: python3.6
      Policies: ComprehendFullAccess
      Events:
        HttpPost:
          Type: Api
          Properties:
            Path: '/sentiment'
            Method: post

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  ComprehendApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/sentiment/"
  Comprehend01nlpseriesep3Function:
    Description: "comprehend_01_nlpseries_ep_3 Function ARN"
    Value: !GetAtt Comprehend01nlpseriesep3Function.Arn
  Comprehend01nlpseriesep3FunctionIamRole:
    Description: "Implicit IAM Role created for comprehend_01_nlpseries_ep_3 function"
    Value: !GetAtt Comprehend01nlpseriesep3FunctionRole.Arn

Puedes ir al archivo del repo aquí: ccortezb/comprehend_apirest_sentiment_nlpseries You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com

En este template, creamos el API, los recursos y los métodos, en este caso necesitamos hacer un POST a /sentiment que hará un proxy directo a nuestro Lambda para luego comunicarse con Comprehend

Events:

HttpPost:

   Type: Api

      Properties:

         Path: ‘/sentiment’

         Method: post

🧩 SAM Build:

Ejecutar lo siguiente:

(nlp01) ➜ comprehend-01-nlpseries-ep-3 **sudo sam build**

Resultado:

Building resource ‘Comprehend01nlpseriesep3Function’

Running PythonPipBuilder:ResolveDependencies

Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts : .aws-sam/build

Built Template : .aws-sam/build/template.yaml

Commands you can use next

=========================

[*] Invoke Function: sam local invoke

[*] Deploy: sam deploy — guided

🧩 SAM Deploy:

Haremos el Deploy directo desde la consola, para ejecutar nuestro Stack de CloudFormation de manera programática

(nlp01) ➜  comprehend-01-nlpseries-ep-3 sudo sam deploy --guided

Configuring SAM deploy
======================

  Looking for samconfig.toml :  Found
  Reading default arguments  :  Success

  Setting default arguments for 'sam deploy'
  =========================================
  Stack Name [comprehend-01-nlp-03]:
  AWS Region [ca-central-1]:
  #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
  Confirm changes before deploy [Y/n]: Y
  #SAM needs permission to be able to create roles to connect to the resources in your template
  Allow SAM CLI IAM role creation [Y/n]: Y
  Save arguments to samconfig.toml [Y/n]: Y

  Looking for resources needed for deployment: Found!

    Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-18d1u3hw3hac2
    A different default S3 bucket can be set in samconfig.toml

  Saved arguments to config file
  Running 'sam deploy' for future deployments will use the parameters saved above.
  The above parameters can be changed by modifying samconfig.toml
  Learn more about samconfig.toml syntax at
  https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

  Deploying with following values
  ===============================
  Stack name                 : comprehend-01-nlp-03
  Region                     : ca-central-1
  Confirm changeset          : True
  Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-18d1u3hw3hac2
  Capabilities               : ["CAPABILITY_IAM"]
  Parameter overrides        : {}

Initiating deployment
=====================

Waiting for changeset to be created..
Error: Failed to create changeset for the stack: comprehend-01-nlp-03, ex: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Template error: instance of Fn::GetAtt references undefined resource HelloWorldFunctionRole
(nlp01) ➜  comprehend-01-nlpseries-ep-3 sudo sam deploy --guided

Configuring SAM deploy
======================

  Looking for samconfig.toml :  Found
  Reading default arguments  :  Success

  Setting default arguments for 'sam deploy'
  =========================================
  Stack Name [comprehend-01-nlp-03]:
  AWS Region [ca-central-1]:
  #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
  Confirm changes before deploy [Y/n]: Y
  #SAM needs permission to be able to create roles to connect to the resources in your template
  Allow SAM CLI IAM role creation [Y/n]: Y
  Save arguments to samconfig.toml [Y/n]: Y

  Looking for resources needed for deployment: Found!

    Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-18d1u3hw3hac2
    A different default S3 bucket can be set in samconfig.toml

  Saved arguments to config file
  Running 'sam deploy' for future deployments will use the parameters saved above.
  The above parameters can be changed by modifying samconfig.toml
  Learn more about samconfig.toml syntax at
  https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

  Deploying with following values
  ===============================
  Stack name                 : comprehend-01-nlp-03
  Region                     : ca-central-1
  Confirm changeset          : True
  Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-18d1u3hw3hac2
  Capabilities               : ["CAPABILITY_IAM"]
  Parameter overrides        : {}

Initiating deployment
=====================
Uploading to comprehend-01-nlp-03/eff52ec0ae2140c2154eb8486ef244aa.template  1286 / 1286.0  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                        LogicalResourceId                                ResourceType
------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                            Comprehend01nlpseriesep3FunctionHttpPostPermis   AWS::Lambda::Permission
                                                 sionProd
+ Add                                            Comprehend01nlpseriesep3FunctionRole             AWS::IAM::Role
+ Add                                            Comprehend01nlpseriesep3Function                 AWS::Lambda::Function
+ Add                                            ServerlessRestApiDeployment1449b782f7            AWS::ApiGateway::Deployment
+ Add                                            ServerlessRestApiProdStage                       AWS::ApiGateway::Stage
+ Add                                            ServerlessRestApi                                AWS::ApiGateway::RestApi
------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:ca-central-1:[aws_account_number]:changeSet/samcli-deploy1617735990/54bf479f-01a3-49ac-a71f-30219a23f2ba


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2021-04-06 14:06:44 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                       ResourceType                         LogicalResourceId                    ResourceStatusReason
-------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                   AWS::IAM::Role                       Comprehend01nlpseriesep3FunctionRo   -
                                                                          le
CREATE_IN_PROGRESS                   AWS::IAM::Role                       Comprehend01nlpseriesep3FunctionRo   Resource creation Initiated
                                                                          le
CREATE_COMPLETE                      AWS::IAM::Role                       Comprehend01nlpseriesep3FunctionRo   -
                                                                          le
CREATE_IN_PROGRESS                   AWS::Lambda::Function                Comprehend01nlpseriesep3Function     Resource creation Initiated
CREATE_IN_PROGRESS                   AWS::Lambda::Function                Comprehend01nlpseriesep3Function     -
CREATE_COMPLETE                      AWS::Lambda::Function                Comprehend01nlpseriesep3Function     -
CREATE_IN_PROGRESS                   AWS::ApiGateway::RestApi             ServerlessRestApi                    -
CREATE_COMPLETE                      AWS::ApiGateway::RestApi             ServerlessRestApi                    -
CREATE_IN_PROGRESS                   AWS::ApiGateway::RestApi             ServerlessRestApi                    Resource creation Initiated
CREATE_IN_PROGRESS                   AWS::Lambda::Permission              Comprehend01nlpseriesep3FunctionHt   -
                                                                          tpPostPermissionProd
CREATE_IN_PROGRESS                   AWS::ApiGateway::Deployment          ServerlessRestApiDeployment1449b78   -
                                                                          2f7
CREATE_COMPLETE                      AWS::ApiGateway::Deployment          ServerlessRestApiDeployment1449b78   -
                                                                          2f7
CREATE_IN_PROGRESS                   AWS::ApiGateway::Deployment          ServerlessRestApiDeployment1449b78   Resource creation Initiated
                                                                          2f7
CREATE_IN_PROGRESS                   AWS::Lambda::Permission              Comprehend01nlpseriesep3FunctionHt   Resource creation Initiated
                                                                          tpPostPermissionProd
CREATE_IN_PROGRESS                   AWS::ApiGateway::Stage               ServerlessRestApiProdStage           -
CREATE_COMPLETE                      AWS::ApiGateway::Stage               ServerlessRestApiProdStage           -
CREATE_IN_PROGRESS                   AWS::ApiGateway::Stage               ServerlessRestApiProdStage           Resource creation Initiated
CREATE_COMPLETE                      AWS::Lambda::Permission              Comprehend01nlpseriesep3FunctionHt   -
                                                                          tpPostPermissionProd
CREATE_COMPLETE                      AWS::CloudFormation::Stack           comprehend-01-nlp-03                 -
-------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
-------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
-------------------------------------------------------------------------------------------------------------------------------------------------
Key                 ComprehendApi
Description         API Gateway endpoint URL for Prod stage for Hello World function
Value               https://[api_id].execute-api.ca-central-1.amazonaws.com/Prod/sentiment/

Key                 Comprehend01nlpseriesep3FunctionIamRole
Description         Implicit IAM Role created for comprehend_01_nlpseries_ep_3 function
Value               arn:aws:iam::[aws_account_number]:role/comprehend-01-nlp-03-Comprehend01nlpseriesep3Funct-1363JG49IXNQG

Key                 Comprehend01nlpseriesep3Function
Description         comprehend_01_nlpseries_ep_3 Function ARN
Value               arn:aws:lambda:ca-central-1:[aws_account_number]:function:comprehend-01-nlp-03-Comprehend01nlpseriesep3Funct-BFO9E0QBGG47
-------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - comprehend-01-nlp-03 in ca-central-1

Una vez esté 100% desplegado, si nos fijamos en los Outputs, tendremos todos los valores necesarios para empezar a testar.

CloudFormation outputs from deployed stack

 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

Outputs

 — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 

Key ComprehendApi

Description API Gateway endpoint URL for Prod stage for your function

Value https://{api}.execute-api.ca-central-1.amazonaws.com/Prod/sentiment/

Key Comprehend01nlpseriesep3FunctionIamRole

Description Implicit IAM Role created for comprehend_01_nlpseries_ep_3 function

Value arn:aws:iam::cuenta_aws:role/comprehend-01-nlp-03-Comprehend01nlpseriesep3Funct-1363JG49IXNQG

Key Comprehend01nlpseriesep3Function

Description comprehend_01_nlpseries_ep_3 Function ARN

Value arn:aws:lambda:ca-central-1:cuenta_aws:function:comprehend-01-nlp-03-Comprehend01nlpseriesep3Funct-BFO9E0QBGG47

🧩 Pruebas con API rest:

Vamos a probar con un simple request nuestra nueva API vía CuRL:

{“body”: “Hola a todos, estamos aprendiendo cómo usar servicios de inteligencia artificial en AWS para el entendimiento del lenguaje natural así que pongamos mucha atención”}

curl -X POST \
  https://d5601zkukf.execute-api.ca-central-1.amazonaws.com/Prod/sentiment \
  -H 'cache-control: no-cache' \
  -H 'content-type: application/json' \
  -H 'postman-token: ea059b31-e081-24e6-8911-c179362ee99c' \
  -d '{"body": "{\"Body\":\"estoy de humor bueno\"}"}'

y por qué no también por Postman, más interactivo:

Imagen: request desde Postman

{
  'Sentiment': 'NEUTRAL',
  'SentimentScore': {
    'Positive': 0.019190235063433647,
    'Negative': 0.027914147824048996,
    'Neutral': 0.937386691570282,
    'Mixed': 0.015508887358009815
  },
  'ResponseMetadata': {
    'RequestId': 'c45999f3-3021-4e4d-95e8-bd8763d54337',
    'HTTPStatusCode': 200,
    'HTTPHeaders': {
      'x-amzn-requestid': 'c45999f3-3021-4e4d-95e8-bd8763d54337',
      'content-type': 'application/x-amz-json-1.1',
      'content-length': '163',
      'date': 'Wed, 07 Apr 2021 00:12:21 GMT'
    },
    'RetryAttempts': 0
  }
}

Otro ejemplo:

{
 “body”: “Me encanta aprender cosas nuevas, más aún si es sobre Ai o ML, la verdad es un tema muy interesante y lo explicas de manera entretenida”
}

Resultado a prueba de análisis de sentimiento

‘Sentiment’: ‘POSITIVE’, ‘SentimentScore’: {‘Positive’: 0.9948458671569824, ‘Negative’: 9.346976730739698e-05, ‘Neutral’

Mientras más se acerca a 1, la probabilidad de que sea positivo es mayor. así como mientras más se acerca a 0, la probabilidad de que NO lo sea, es casi nula.

Probemos de nuevo cambiando algunos términos, porque no obtuvimos ningún ENTITY reconocido por el API de Amazon Comprehend usando el lenguaje ‘es’, así que veamos:

{
 “body”: “Hola, Soy Carlos y me encanta aprender lenguajes nuevos en internet, más aún si es sobre AI o ML, la verdad es un tema muy interesante y lo explicas de manera entretenida en tu web. visitalo en cortez.cloud a las 8pm de lunes a domingo y en youtube también. ¡ya tiene más de 30 suscriptores! que bueno”
}

Resultado a prueba para extracción de entidades:

[‘Carlos’, ‘AI’, ‘ML’, ‘8pm’, ‘lunes’, ‘domingo’, ‘youtube’, ’30 suscriptores’]
[‘PERSON’, ‘OTHER’, ‘OTHER’, ‘DATE’, ‘DATE’, ‘DATE’, ‘TITLE’, ‘QUANTITY’]

🧩 Conclusión

Estamos en la punta del iceberg, recién entendiendo NLP usando servicios auto administrados, lo cual nos da resultados aceptables y discutibles. Pero espera!Recién estamos empezando con Comprehend, y además también vamos a aprender librerías propias open source para poder compararlo y llevarlo a AWS Sagemaker luego. Eso ya suena como level 10 o 15 no? Vamos con calma y a paso seguro.

¡Estamos en la nube podemos romperlo todo y no pasa nada. (en desarrollo…)

Puedes revisar todo el código aquí en mi github: ccortezb/comprehend_apirest_sentiment_nlpseries estos son los 3 archvos que necesitas editar si empieazas desde cero un repo de SAM de Hello World.github.com

Siguiente post: (Al Aire el 12 de Abril)

Parte 2: https://ccortezb.medium.com/ingestando-noticias-de-mi-propia-web-text-analytics-basics-nlp-series-parte-2-por-carlos-7899bedd0995

Parte 1: https://ccortezb.medium.com/empezando-con-nlp-en-aws-desde-cero-nlp-series-parte-1-por-carlos-cortez-breaking-the-cloud-13da2adc0151

Si te gustó este post, dale un like, comparte y comenta.

Estas series son parte de una nueva sección llamada AI for Mortals, donde me incluyo con ustedes para enseñarles lo que voy aprendiendo.

Visitar AI for Mortals: (https://aiformortals.cortez.cloud)

☢ Rompiendo se aprende

Suscríbete a mi canal, Breaking the Cloud y Al día con AWS en https://cortez.cloud

⭐Suscríbete a mi canal : http://bit.ly/aldiaconaws

videos, noticas de AWS, análisis, demos, workshops

🔥🔥 Sígueme en mis redes 🔥🔥

follow <- me()

🦜 Mi Twitter: https://twitter.com/ccortezb

📺 Youtube Channel: http://bit.ly/aldiaconaws

📺 AWSUGPerú: https://www.youtube.com/awsusergroupperuoficial

📟 Mi Facebook: https://www.facebook.com/ccortezb/

🤳 Mi Instagram: ccortezbazan

📜 Mis cursos de AWS: https://cennticloud.thinkific.com

🕮 Mi blog — cortez.cloud

Muchas gracias, espero nos volvamos a ver

🔥🔥 Acerca de mí 🔥🔥

Cortez.Cloud/whoami

Les presento mi pequeña web personal https://www.Cortez.Cloud llamado “Breaking the Cloud”.

Seguiré creando contenido cada semana de AWS sobre Al/ML, Serverless, Security y como romper las reglas!

También mis próximas iniciativas, talleres, cursos, videos gratuitos, awsugperu y más.

De quiénes se ensucian las manos, ruedan en el barro, y se llenan de esa adrenalina de hacer las cosas tú mismo y no por un managed service de algún cloud provider. De quiénes pueden inventar sus propios métodos, sus propios principios y su propio camino. Yo te pregunto, ¿aún puedes seguir inventando si mezclas lo mejor de ambos mundos no?

Estas series son parte de una nueva sección llamada AI for Mortals, donde me incluyo con ustedes para enseñarles lo que voy aprendiendo.

Visitar AI for Mortals: (https://aiformortals.cortez.cloud)

🧩 Primero lo primero: A ingestar con nuestras propias manos

Generaremos una ingesta de datos con información propia, tomada de mi blog https://cortez.cloud, para obtener categorías, títulos, y descripciones de los posts publicados.

Si estuviésemos en un MMORPG , nuestro guerrero aún estaría en Level 1:

Hagámoslo lo más simple posible pero no Simplón — Albert Einstein

y recordando dónde nos encontramos en nuestro viaje en este mundo:

🧩 ¿Que usaremos?

Beautiful Soap, una librería de Python que nos ayudará a extraer data de la web, lo que llamamos “Web Scraping”

Veamos paso a paso que es lo que estamos haciendo aquí,

1. Importamos nuestras librerías que usaremos durante todos los ejemplos desde hoy en adelante:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os

1. Definimos cuáles URLs vamos a usar:

cortezcloud_urls = ['https://www.cortez.cloud/categories/fundamentals',
             'https://www.cortez.cloud/categories/serverless',
             'https://www.cortez.cloud/categories/awsugperu']

1. Inspeccionamos y entendamos las etiquetas que existen en la web:

En una de las URLS, https://www.cortez.cloud/categories/awsugperu

Los títulos están en un <div class=”blog-title”>, y esto es lo que tenemos que ir buscando

1. Obtenemos palabras clave de las URLS como las categorías

cloud_category = url.split('/')[4]

1. Ahora bajamos toda la info y parseamos todo el HTML con Beautiful Soap. Parsing es un método de programación que nos permite analizar el HTML, pero también diferentes tipos de archivos como json.

data = requests.get(url)
soup = BeautifulSoup(data.content, 'html.parser')

1. Buscamos todo los divs que hemos ido encontrando para los títulos “headlines”, los blog posts “news_blog” y las categorías ya las tenemos .

news_blogs = [{'news_headline': headline.find('div', 
                                                         attrs={"class": "blog-title"}).string,
                          'news_blog': blog.find('div', 
                                                       attrs={"class": "preview-text"}).string,
                          'cloud_category': cloud_category}

1. Creamos nuestro Data Frame y colocamos toda la data encontrada y segmentada en 3 columnas:

df =  pd.DataFrame(cortezcloud_data)
    df = df[['news_headline', 'news_blog', 'cloud_category']]

1. Revisamos nuestra data recién salida del horno:

ultimos_posts = ingestar_noticias(cortezcloud_urls)
ultimos_posts.head(10)

🧩 Ejercicio completo de este tutorial:

El código completo es el siguiente y lo puedes encontrar en mi github en Colab para que puedan editarlo y usarlo:

jupyter notebook { "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "name": "Ingestando noticias de mi propia web.ipynb", "provenance": [], "authorship_tag": "ABX9TyPpymjo5tVL1p5qAm0AbRN5", "include_colab_link": true }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "<a href=\"https://colab.research.google.com/gist/ccortezb/315c16d6a61208229e0db3ffd095affe/ingestando-noticias-de-mi-propia-web.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>" ] }, { "cell_type": "code", "metadata": { "id": "BBWvh-jVtn8J" }, "source": [ "import requests\n", "from bs4 import BeautifulSoup\n", "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import os\n", "%matplotlib inline\n", "\n", "cortezcloud_urls = ['https://www.cortez.cloud/categories/fundamentals',\n", " 'https://www.cortez.cloud/categories/serverless',\n", " 'https://www.cortez.cloud/categories/awsugperu']\n", "\n", "def ingestar_noticias(cortezcloud_urls):\n", " cortezcloud_data = []\n", " for url in cortezcloud_urls:\n", " cloud_category = url.split('/')[4]\n", " data = requests.get(url)\n", " soup = BeautifulSoup(data.content, 'html.parser')\n", " \n", " news_blogs = [{'news_headline': headline.find('div', \n", " attrs={\"class\": \"blog-title\"}).string,\n", " 'news_blog': blog.find('div', \n", " attrs={\"class\": \"preview-text\"}).string,\n", " 'cloud_category': cloud_category}\n", " \n", " for headline, blog in \n", " zip(soup.find_all('div', \n", " class_=[\"thumbnail-text\"]),\n", " soup.find_all('div', \n", " class_=[\"thumbnail-text\"]))\n", " ]\n", " cortezcloud_data.extend(news_blogs)\n", " \n", " df = pd.DataFrame(cortezcloud_data)\n", " df = df[['news_headline', 'news_blog', 'cloud_category']]\n", " return df\n" ], "execution_count": 1, "outputs": [] }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 162 }, "id": "ERIXIl2ewiwn", "outputId": "eb1495a0-7d4c-4671-b00f-c10b67558ebf" }, "source": [ "ultimos_posts = ingestar_noticias(cortezcloud_urls)\n", "ultimos_posts.head(10)" ], "execution_count": 2, "outputs": [ { "output_type": "execute_result", "data": { "text/html": [ "<div>\n", "<style scoped>\n", " .dataframe tbody tr th:only-of-type {\n", " vertical-align: middle;\n", " }\n", "\n", " .dataframe tbody tr th {\n", " vertical-align: top;\n", " }\n", "\n", " .dataframe thead th {\n", " text-align: right;\n", " }\n", "</style>\n", "<table border=\"1\" class=\"dataframe\">\n", " <thead>\n", " <tr style=\"text-align: right;\">\n", " <th></th>\n", " <th>news_headline</th>\n", " <th>news_blog</th>\n", " <th>cloud_category</th>\n", " </tr>\n", " </thead>\n", " <tbody>\n", " <tr>\n", " <th>0</th>\n", " <td>2 Trabajando con Amazon DocumentDB desde mi local</td>\n", " <td>Seguimos en las series de Docdb, esta vez, 2da...</td>\n", " <td>fundamentals</td>\n", " </tr>\n", " <tr>\n", " <th>1</th>\n", " <td>5 cosas que necesitas empezar a aprender para ...</td>\n", " <td>Aún no entiendes bien cómo empezar a migrar tu...</td>\n", " <td>fundamentals</td>\n", " </tr>\n", " <tr>\n", " <th>2</th>\n", " <td>Primeros pasos con Amazon DocumentDB y AWS CLI</td>\n", " <td>Estas serán unas series donde hablaremos de un...</td>\n", " <td>fundamentals</td>\n", " </tr>\n", " <tr>\n", " <th>3</th>\n", " <td>Cómo me convierto en un AWS Community Builder ...</td>\n", " <td>Empieza tu travesía para educar AWS en el Perú</td>\n", " <td>awsugperu</td>\n", " </tr>\n", " </tbody>\n", "</table>\n", "</div>" ], "text/plain": [ " news_headline ... cloud_category\n", "0 2 Trabajando con Amazon DocumentDB desde mi local ... fundamentals\n", "1 5 cosas que necesitas empezar a aprender para ... ... fundamentals\n", "2 Primeros pasos con Amazon DocumentDB y AWS CLI ... fundamentals\n", "3 Cómo me convierto en un AWS Community Builder ... ... awsugperu\n", "\n", "[4 rows x 3 columns]" ] }, "metadata": { "tags": [] }, "execution_count": 2 } ] } ] }

🧩 Casos de uso distintos y automatización

Pasémoslo a Funciones como Servicio usando AWS Lambda y empecemos a automatizar y crear nuestros propios pipelines de ingesta de datos.

Lo puedes replicar para hacer ingesta en Twitter o de cualquier otra red social y empezar a generar tus propios análisis.

Nosotros nos iremos por la rama de Procesamiento de lenguaje natural tanto con librerías open source como con Amazon Web Services

🧩 ¿Con qué continuamos?

Si en el próximo episodio, cambiamos de aires y nos vamos a Amazon Comprehend?

No se preocupen vamos paso a paso y alternando entre ambos caminos.

📢 📢 Spoiler alert:

Doble click a nuestro MMORPG , nuestro guerrero tiene la meta de llegar a level 5:

Siguiente post: (Al Aire el 05 de Abril)

Post Anterior: Empezando con nlp en AWS desde cero

Si te gustó este post, dale un like, comparte y comenta.

Estas series son parte de una nueva sección llamada AI for Mortals, donde me incluyo con ustedes para enseñarles lo que voy aprendiendo.

Visitar AI for Mortals: (https://aiformortals.cortez.cloud)

☢ Rompiendo se aprende

Suscríbete a mi canal, Breaking the Cloud y Al día con AWS en https://cortez.cloud

⭐Suscríbete a mi canal : http://bit.ly/aldiaconaws

videos, noticas de AWS, análisis, demos, workshops

🔥🔥 Sígueme en mis redes 🔥🔥

follow <- me()

🦜 Mi Twitter: https://twitter.com/ccortezb

📺 Youtube Channel: http://bit.ly/aldiaconaws

📺 AWSUGPerú: https://www.youtube.com/awsusergroupperuoficial

📟 Mi Facebook: https://www.facebook.com/ccortezb/

🤳 Mi Instagram: ccortezbazan

📜 Mis cursos de AWS: https://cennticloud.thinkific.com

🕮 Mi blog — cortez.cloud

Muchas gracias, espero nos volvamos a ver

🔥🔥 Acerca de mí 🔥🔥

Cortez.Cloud/whoami

Les presento mi pequeña web personal https://www.Cortez.Cloud llamado “Breaking the Cloud”.

Seguiré creando contenido cada semana de AWS sobre Al/ML, Serverless, Security y como romper las reglas!

También mis próximas iniciativas, talleres, cursos, videos gratuitos, awsugperu y más.

Qué es? Y quiénes somos?

Somos 3! A la izquierda, Carlos Cortez (AWS UG Perú Leader & Founder), al centro Damian Olguín (AWS Hero), y a la derecha Matías Kreder (re:invent 2020 qualifier finalist)

Unas series educativas para acercarnos al Aprendizaje por refuerzos (reinforcement Learning) y en AWS

Aquí los episodios que hemos lanzado hasta el momento:

Episodio 1: https://www.youtube.com/watch?v=TtOznh3G6z0

Episodio 2: https://www.youtube.com/watch?v=eUWSdrxQOrw

Episodio 3: https://www.youtube.com/watch?v=Ww1HMEzEJ-8

Episode 4: the final cup at awsugperuconf

Nos acercamos a una tecnología emergente dentro de la inteligencia artificial, llamada aprendizaje reforzado, y AWS DeepRacer es un servicio autogestionado donde podemos aplicar este tipo de modelos basado en este tipo de aprendizaje por recompensas.

‍ Creamos ligas de AWS DeepRacer en Latinoamérica

Las primeras hechas en español en latam.

Las 3 ligas oficiales que creamos:

League #1:

League #2:

League #3:

Hacemos que el entrenamiento reforzado sea más sencillo de entender

En los siguientes posts, les enseñaré cómo entrar en el mundo de RL, teórico y práctico y así entender cómo es que funciona detrás de tanta genialidad que es AWS DeepRacer.

• Creating tensors: Empty() and Zeros()

• Measuring tensors: Size() and Shape(), Sum() and Dimensions

• Changing and copying tensors: Reshape(), view() and randn()

• Modifying tensors: Unsqueeze()

• Comparing tensors: Element-Wise Equality: Eq

First lets spin up a new sagemaker instance:

Now go to Jupyter Lab and import or create a new Notebook with this kernel for Pytorch:

Select the appropiate kernel, for this case it could be: conda_pytorch_p36

Here is where I will start from scratch.

Before we begin, let’s install and import PyTorch

*# Uncomment and run the appropriate command for your operating system, if required*

# Linux / Binder
# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

# Windows
# !pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

# MacOS
# !pip install numpy torch torchvision torchaudio

Import Pytorch from the Notebook instance

**import** **torch**

Function 1 — Empty and Zeros — how to initialize tensors

We need to start working with the basics pytorch functions, and the first thing is create our matrix

*# Creates a 3 x 2 matrix which is empty*
a = torch.empty(3, 2)
print(a)

tensor([[1.5842e-35, 0.0000e+00],
        [4.4842e-44, 0.0000e+00],
        [       nan, 0.0000e+00]])

Here is how PyTorch is allocating memory for this tensor. Whatever, it will not erase anything previous content in the memory.

by default, when you initializes a tensor is used the float32 dtype. you can review it here: https://pytorch.org/docs/stable/generated/torch.set_default_tensor_type.html#torch.set_default_tensor_type

But you can also start working with torch.zeros

*# Applying the zeros function and *
*# storing the resulting tensor*

a = torch.zeros([3, 5]) 
print("a = ", a) 

b = torch.zeros([2, 4]) 
print("b = ", b) 

c = torch.zeros([4, 1]) 
print("c = ", c) 

d = torch.zeros([4, 4, 2]) 
print("d = ", d)

the result will be:

a =  tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])
b =  tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.]])
c =  tensor([[0.],
        [0.],
        [0.],
        [0.]])
d =  tensor([[[0., 0.],
         [0., 0.],
         [0., 0.],
         [0., 0.]],

[[0., 0.],
         [0., 0.],
         [0., 0.],
         [0., 0.]],

[[0., 0.],
         [0., 0.],
         [0., 0.],
         [0., 0.]],

[[0., 0.],
         [0., 0.],
         [0., 0.],
         [0., 0.]]])

this tensor is filled with zeros, so PyTorch allocates memory and zero-initializes the tensor elements inside

You cannot change the way a tensor is created, if you create a zeros tensor, make sure is not referenced to any other.

*# correctly initialized*
a = torch.empty(3,3)
print(a)

#also correct
b = torch.empty(1,2,3)

print(b)

the results will be:

tensor([[2.4258e-35, 0.0000e+00, 1.5975e-43],
        [1.3873e-43, 1.4574e-43, 6.4460e-44],
        [1.4153e-43, 1.5274e-43, 1.5695e-43]])
tensor([[[2.3564e-35, 0.0000e+00, 1.4013e-45],
         [1.4574e-43, 6.4460e-44, 1.4153e-43]]])

Lets see now, how you cannot use the zeros function:

*# incorrect reference, you must create a new one*
c = torch.zeros(b,1) 
print("c = ", c)

output:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-35-c044e2995879> in <module>()
**      1** # incorrect reference, you must create a new one
----> 2 c = torch.zeros(b,1)
**      3** print("c = ", c)

TypeError: zeros() received an invalid combination of arguments - got (Tensor, int), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)

lets review the zero method closely:

Our documentation says:

Syntax: torch.zeros(size, out=None)

Parameters:
size: a sequence of integers defining the shape of the output tensor
out (Tensor, optional): the output tensor

Return type: A tensor filled with scalar value 0, of same shape as size.

torch.zeros and torch.empty are the first functions to start working with pytorch tensors and learning a little bit of matrix and vectors

Function 2 — Tensor Size, Shape and Dimension Operations

Lets understand dimensions in Pytorch.

Now lets create some tensors and determine the size of every one

**import** **torch** 
*# Create a tensor from data*
c = torch.tensor([[3.2 , 1.6, 2], [1.3, 2.5 , 6.9]])
print(c)

print(c.size())

output

tensor([[3.2000, 1.6000, 2.0000],
        [1.3000, 2.5000, 6.9000]])
torch.Size([2, 3])

Lets see torch.shape and take a closer look at how size is given here

Now in the next example lets use shape functions to get the size of a tensor

In [ ]:

x = torch.tensor([
     [1, 2, 3],
     [4, 5, 6]
   ])
x.shape
torch.Size([2, 3])

Out[ ]:

torch.Size([2, 3])

Shape and Size give us the same correct dimensions of the tensor.

in this case we have a 3D-tensor (with 3 dimensions)

Dimension 0 Dimension 1 and Dimension 2

lets create a new tensor:

In [ ]:

y = torch.tensor([
     [
       [1, 2, 3],
       [4, 5, 6]
     ],
     [
       [1, 2, 3],
       [4, 5, 6]
     ],
     [
       [1, 2, 3],
       [4, 5, 6]
     ]
   ])

y.shape

Out[ ]:

torch.Size([3, 2, 3])

Lets se how we can make operations using a 3d tensor now for every dimension layer and see how it behaves

In [ ]:

sum1 = torch.sum(y, dim=0)
print(sum1)

sum2 = torch.sum(y, dim=1)
print(sum2)

sum3 = torch.sum(y, dim=2)
print(sum3)

tensor([[ 3,  6,  9],
        [12, 15, 18]])
tensor([[5, 7, 9],
        [5, 7, 9],
        [5, 7, 9]])
tensor([[ 6, 15],
        [ 6, 15],
        [ 6, 15]])

we can see now a 3d tensor is more complicated as we advance, and we can perform custom operations within every dimension

its limited right now to 3 dims so we cannot perform this:

In []

sum2 = torch.sum(y, dim=3)
print(sum1)

Out [ ]:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-56-7caf723c6102> in <module>()
----> 1 sum2 = torch.sum(y, dim=3)
**      2** print(sum1)

IndexError: Dimension out of range (expected to be in range of [-3, 2], but got 3)

to accomplish adding more layers of dimensions we can review the unsqueeze functions → .unsqueeze() but for now let’s go to the next basic function

Function 3 — Reshape, View and Random

There is a function in NunPy called ndarray.reshape() for reshaping an array.

Now in pytorch, there is torch.view(tensor) for the same purpose, but at the same time, there is also a torch.reshape(tensor).

let’s figure out the differences between them and when you should use either of them.

First of all we are going to use a new functions for randomize our tensor.

In [ ]:

**import** **torch** 
x = torch.randn(5, 3)

print(x)

Out []

tensor([[-0.5793,  0.6999,  1.7417],
        [-0.9810,  0.0626,  0.4100],
        [-0.6519, -0.0595, -1.2156],
        [-0.3973, -0.3103,  1.6253],
        [ 0.2775, -0.0045, -0.2985]])

this is basic usage of torch.randm, so now lets use View from another variable “y” and describe all elements

In [ ]:

*# Return a view of the x, but only having *
*# one dimension and max number of elements*
y = x.view(5 * 3)

*#lets see the size of every tensor*
print("lets see the size of every tensor")
print('Size of x:', x.size())
print('Size of y:', y.size())

*#and the elements of very tensor to compare*
print("and the elements of very tensor to compare")
print("X:", x)
print("Y:", y)

Out []

lets see the size of every tensor
Size of x: torch.Size([5, 3])
Size of y: torch.Size([15])
and the elements of very tensor to compare
X: tensor([[-0.5793,  0.6999,  1.7417],
        [-0.9810,  0.0626,  0.4100],
        [-0.6519, -0.0595, -1.2156],
        [-0.3973, -0.3103,  1.6253],
        [ 0.2775, -0.0045, -0.2985]])
Y: tensor([-0.5793,  0.6999,  1.7417, -0.9810,  0.0626,  0.4100, -0.6519, -0.0595,
        -1.2156, -0.3973, -0.3103,  1.6253,  0.2775, -0.0045, -0.2985])

take a look at Y tensor, it only has 1 dimension. so Viewing another tensor may be difficult for some operations

Now lets use Reshape to replicate the exact dimensions of X

In [ ]:

*# Get back the original tensor with reshape()*
z = y.reshape(5, 3)
print(z)

tensor([[-0.2927,  0.0329,  0.8485],
        [ 1.9581,  0.8313, -0.1529],
        [-0.2330, -0.1887,  1.8206],
        [ 1.5252,  1.0909,  0.0547],
        [-0.1231, -0.4238, -0.6724]])

we cannot only reshape the original, we can also change the dimensions with some limited actions related to the maximum elements:

first lets reshape to 1 more dimension

In [ ]:

*# Get back the original tensor with reshape()*
z = y.reshape(15)
print(z)

#reshaping 15 elements, 1 dim
z = y.reshape(3*5)
print(z)

#reshaping in different order, 3 dimensions
z = y.reshape(3,5)
print(z)

#Reshaping with more dimensions but its 15 elements always
z = y.reshape(3,5,1)
print(z)

tensor([-0.5793,  0.6999,  1.7417, -0.9810,  0.0626,  0.4100, -0.6519, -0.0595,
        -1.2156, -0.3973, -0.3103,  1.6253,  0.2775, -0.0045, -0.2985])
tensor([-0.5793,  0.6999,  1.7417, -0.9810,  0.0626,  0.4100, -0.6519, -0.0595,
        -1.2156, -0.3973, -0.3103,  1.6253,  0.2775, -0.0045, -0.2985])
tensor([[-0.5793,  0.6999,  1.7417, -0.9810,  0.0626],
        [ 0.4100, -0.6519, -0.0595, -1.2156, -0.3973],
        [-0.3103,  1.6253,  0.2775, -0.0045, -0.2985]])
tensor([[[-0.5793],
         [ 0.6999],
         [ 1.7417],
         [-0.9810],
         [ 0.0626]],

        [[ 0.4100],
         [-0.6519],
         [-0.0595],
         [-1.2156],
         [-0.3973]],

        [[-0.3103],
         [ 1.6253],
         [ 0.2775],
         [-0.0045],
         [-0.2985]]])

Now lets reshape exceeding the number of elements in the tensor:

In [ ]:

z = y.reshape(16)
print(z)

Out []

---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-78-c7ae174fce73> in <module>()
----> 1 z = y.reshape(16)
**      2** print(z)

RuntimeError: shape '[16]' is invalid for input of size 15

it will fail also for z = y.reshape(3*6) or putting more elements that does not exist in the tensor.

Now let’s keep going to the next section.

Function 4 — Unsqueeze()

Mainly, it allows us to add more dimensions at specific index you define.

lets take a look:

In [ ]:

**import** **torch**

#dim=1, that is (3)
x = torch.tensor([1, 2, 3])
print('x: ', x)
print('x.size: ', x.size())

#x1 becomes a matrix of (3,1)
x1 = torch.unsqueeze(x, 1)
print('x1: ', x1)
print('x1.size: ', x1.size())

Out []

x:  tensor([1, 2, 3])
x.size:  torch.Size([3])
x1:  tensor([[1],
        [2],
        [3]])
x1.size:  torch.Size([3, 1])
x2:  tensor([[1, 2, 3]])
x2.size:  torch.Size([1, 3])

Our initial tensor is Tensor([1,2,3]), and the output size is [3].

And then we proceed with adding 1 dimenson with unsqueeze operation, namely torch.unsqueeze(x, 1), the size of x1 is [3,1].

In [ ]:

*#x2 becomes a matrix of (1,3)*
x2 = torch.unsqueeze(x, 0)
print('x2: ', x2)
print('x2.size: ', x2.size())

Out []

x2:  tensor([[1, 2, 3]])
x2.size:  torch.Size([1, 3])

When we perform torch.unsqueeze(x, 0), the size of x2 is [1,3].

In [ ]:

*# Example 3 - breaking (to illustrate when it breaks)*


print(x.unsqueeze())

Out []

---------------------------------------------------------------------------
TypeError  Traceback (most recent call last)
<ipython-input-89-5a320a828907> in <module>()
**      2** 
**      3** 
----> 4 print(x.unsqueeze())

TypeError: unsqueeze() missing 1 required positional arguments: "dim"

We must specified the dimension correctly, although we are just adding 1 dim, it is necessary to put like this: x.unqueeze(0)

Function 5 — Torch Eq (Element Wise equality)

This function is under comparison category and it computes equality in element-wise and returns a boolean tensor. True if equal, False otherwise.

Lets review how we can operate with different sizes of tensors:

In [ ]:

*# Example 1 - working *
x1 = torch.tensor([[1, 2], [3, 4.]])
x2 = torch.tensor([[2, 2], [2, 5]])
x3 = torch.randn(3,5)

#size x1 and z2
print(x1.size())
print(x2.size())

# tensors 1 and 2
print(x1)
print(x2)


#size x3
print(x3.size())

#tensors 3
print(x3)

torch.eq(x1,x2)

torch.Size([2, 2])
torch.Size([2, 2])
tensor([[1., 2.],
        [3., 4.]])
tensor([[2, 2],
        [2, 5]])
torch.Size([3, 5])
tensor([[-1.3040, -0.4658, -0.5269,  0.7409,  0.9135],
        [ 1.0780,  2.0584, -0.9629, -1.1412, -0.3105],
        [ 0.3613, -1.4196,  2.1145,  0.3649,  0.2037]])

Out[ ]:

tensor([[False,  True],
        [False, False]])

x1 and x3 have the same size, but x3 is [3,5], has bigger size.

comparing x1 and x2 is Ok.

In [ ]:

*# Example 2 - working (with broadcasting)*
x4 = torch.tensor([[1, 2], [3, 4]])
print(x4.size())
x5 = torch.tensor([2, 5])
print(x5.size())
torch.eq(x4, x5)

torch.Size([2, 2])
torch.Size([2])

Out[ ]:

tensor([[False, False],
        [False, False]])

we can also compare, different sizes only if the seconf value, in this case x5, that has size of [2] is broadcastable with the frist one thst is [2,2]

In [ ]:

*# Example 3 - breaking (to illustrate when it breaks)*
x6 = torch.tensor([[0, 2, 4], [3, 4, 5]])
print(x6.size())
x7 = torch.tensor([[2, 3], [2, 4]])
print(x7.size())

torch.eq(x6, x3)

torch.Size([2, 3])
torch.Size([2, 2])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-98-ac2ad1ecd5b0> in <module>()
      5 print(x7.size())
      6 
----> 7 torch.eq(x6, x3)

RuntimeError: The size of tensor a (3) must match the size of tensor b (5) at non-singleton dimension 1

finally, we cant compare different sizes if the second arguments shape is not broadcastable with the first argument.

Conclusion

we review 5 basic topics covering more than 10 PyTorch functions.

in the next post i’ll talk about Linear Regression.

Reference Links

Provide links to your references and other interesting articles about tensors

• Official documentation for tensor operations: https://pytorch.org/docs/stable/torch.html

• Squeeze and unsqueeze functions here: https://www.programmersought.com/article/65705814717/

• Understanding dimensions in Pytorch: https://towardsdatascience.com/understanding-dimensions-in-pytorch-6edf9972d3be

#aws #reinvent2020 #awsperu #awsugperu #awscloud

Carlos Cortez — AWS UG Perú Leader / AWS ML Community Builder ccortez@aws.pe @ccortezb Podcast: imperiocloud.com @imperiocloud twitch.tv/awsugperu cennticloud.thinkific.com Carlos Cortez ML Engineer — Senior Cloud Architect — Pro Services Cluster — BGH Tech Partner |… Hello Everyone, I’m the Founder of AWS User Group Perú 🇵🇪 (2014) and also an AWS Solutions Architect Certified…www.linkedin.com

Seremos los primeros en generar conocimiento y crear una liga en Perú de AWS DeepRacer. Es un gran reto y para eso nos reuniremos.

Para empezar, agradecemos a todos los inscritos a este programa realizado y creado por la comunidad AWS User Group Perú y queremos darle la bienvenida en este evento. Por este medio les enviaremos información relevante respecto a este programa.

Nuestro objetivo

Es hacer que AWS realice una competencia oficial de DeepRacer durante nuestro próximo AWS Community Day 2020. para ello debemos de crear un grupo inicial de líderes y entusiastas en aprender y difundir.

De qué trata este programa gratuito?

El programa de DeepRacer League Perú, se creó para incentivar el aprendizaje reforzado, y los servicios de AWS involucrados en el funcionamiento y entrenamiento de éste vehículo autónomo.

El programa incluye lo siguiente:

• 1ra Convocatoria de voluntariado e interesados a unirse al clan
• Asistir a CETAM para revisar DeepRacer en funcionamiento con la PUCP
• Introducción a AWS DeepRacer y AWS Robomaker, series de 3 partes
• Introducción a RL series de 3 partes
• 2da Convocatoria de voluntariado e interesados a unirse al clan
• Taller de AWS DeepRacer básico, series de 3 partes
• Gestionar la impresión de 1 pista básica 3ra Convocatoria de voluntariado e interesados a unirse al clan Taller de AWS DeepRacer a profundidad Conseguir créditos para participantes.

Las reglas del programa son las siguientes:

• No somos expertos en AWS DeepRacer, aprenderemos juntos cada vez investigando.
• Cada uno tendrá oportunidad de exponer y enseñar lo investigado y logrado ante todos.
• Haremos sesiones online y presenciales
• Divertirnos y aprender muchísimo de cómo funciona Reinforcement Learning en AWS

Gracias, El equipo de AWS User Group Perú

awsperu

awsperu

AWSPeru

AWS User Group Perú, 931886680