A Gentle Introduction to AWS Lambda
You've probably heard of AWS Lambda and serverless by now. But what is Lambda all about? The short definition of AWS Lambda is a "Functions as a Service" (FaaS) technology. The longer and more complicated answer is that Lambda is a lightweight runtime that requires no infrastructure to be defined by the developer.
FaaS allows developers to build software features quickly with less emphasis on the question of where and how they run. This allows for more focus on the business logic that users interact with. Functions as a Service is the most recent innovation in the long line of virtualization technologies. Compute virtualization started with physical servers, then moved to virtual machines (VMs), which grew into Infrastructure as a Service (IaaS). Platforms as a Service came next with companies like Heroku allowing developers to deploy without caring about infrastructure specifics. Now we're arriving at FaaS, which promises to abstract all these details away. Here are a couple good descriptions of these origins and where we are as an industry today.
Lambda vs. Serverless Terminology
The terms serverless and AWS Lambda are often used interchangeably, but they aren't exactly the same thing. Serverless refers to a collection of tools that require no infrastructure configurations from the developer, including databases, queues, functions and gateways as a service. AWS Lambda is only the FaaS component in this list. That means there are many more products such as API gateways, message queueing systems, notification tools, and serverless databases that fill out the serverless landscape. In this blog post we'll only work with a couple of these, namely AWS Lambda and AWS API Gateway for our demonstration.
FaaS Platforms
There are many FaaS providers in the industry. From the big players like Google, IBM, and AWS to the smaller ones like the open source OpenWhisk and OpenFaaS, they all provide similar functionality (generally) with different developer experiences.
There are also Edge computing FaaS providers, which instead of executing the code in one central data center, run the code in a data center nearest the user, known as the "Edge". The players in this area of serverless technology are companies like Cloudflare, Lambda@Edge (AWS), and Fastly.
AWS Lambda in Plan English
After reading the above definition of serverless, Functions as a Service, and AWS Lambda, your understanding might still be cloudy (no pun intended). I'll attempt to explain it in more simple terms.
The description on the AWS documentation website says "Lambda is a compute service that lets you run code without provisioning or managing servers." The essence of Lambda is that AWS built all the scaffolding to allow you and I to send them a single piece of code, which they will run for us.
We don't need to build servers or container images in order to run this single piece of code. All we have to think about is writing code, and then a small SAM template file which defines the requirements of the deployment, and finally shipping it.
Is it Really a Single Function?
In order to wrap your brain around this idea of functions as a service, think of a single method or function in a small program. The function is called by another function, and may return a result.
Below you see a simple Python program in which there is a parent function (main()) calling a child function (childfunction()).
def childfunction(inputstring):
if inputstring == "hello":
return "world"
return "hi"
def main():
value = childfunction("hello")
print(f"value: {value}")
if __name__ == "__main__":
main()
The output of this program looks like this:
$ python hello.py
value: world
At it's simplest form, AWS Lambda can be thought of as a single child function being called by a parent function. AWS owns the parent function, which executes your function (the child function) in it's own virtual environment.
Your objective during the rest of this blog post is to write, deploy and execute a single Lambda function.
Let's Write One Bite Sized Function
For this example, we'll write a small function that does nothing particularly important, but demonstrates the mechanics of running a function in AWS Lambda. You are probably familiar with shell environment variables. In your terminal you can print out the environment variables defined in your session, or even create new environment variables.
The below terminal command prints out an environment variable showing the user name that you are logged in as:
$ echo $USER
mbacchi
What Will the Function Do?
The function we'll deploy will print out the default runtime environment variables in the AWS Lambda execution environment.
Similar to the terminal session above, our Lambda function can access the default environment variables such as AWS_LAMBDA_FUNCTION_NAME, AWS_LAMBDA_LOG_GROUP_NAME, or TZ. But how do we see the output of print statements in a Lambda? Anything that gets printed to STDOUT will be logged to CloudWatch Logs using the log group name of the function.
How Does Lambda Work?
The concepts that you'll need to understand are described in the AWS Lambda documentation.
Here are some of the most important details you'll need to write your Lambda functions:
- Invocation: The function is invoked by one of the triggers that AWS provides. It can also be triggered by cron like scheduling events.
- Entrypoint: Your Lambda function needs to implement a handler which is the method (or function) that gets called by AWS when triggered.
- Environment: The handler is passed an event object and context object from AWS that provides your function both data to be processed and context (information) about how it was called and other clues that you can use in your code.
- Response: In certain cases you will want to syncronously return a value from your function, but that is optional. If you are writing an API backend you will return a value through API Gateway. If instead you are performing some scheduled data processing you may not return a value.
Background on Associated Serverless Components
We mentioned earlier how serverless isn't just AWS Lambda, and in this project we'll use API Gateway and CloudWatch Logs, but only to enable us to show how our Lambda function executes.
AWS API Gateway is, as the name suggests, an API gateway product. But it also allows for more than just API calls. For example it enables generic HTTP/HTTPS traffic, REST APIs and Websocket APIs. You can host a standard web server with API Gateway and Lambda, you aren't limited to just APIs with API Gateway.
CloudWatch Logs is a logging product from AWS that allows all other AWS products to log their activity to a central location. In our example we'll be looking at the output of our Lambda function to see the environment variables available in the Lambda itself during runtime.
We're not going to cover these in any more detail because they're not the main focus of this post.
Sample Lambda Function
For this example deployment, we'll use a GitHub repository that has our AWS Lambda function defined. The Lambda function in this repo was written with the intention of being used for demonstration purposes, and as a simple function that can be deployed when testing out CI/CD or the many serverless deployment tools available today (such as Serverless Framework, Cloud Development Kit (CDK) etc.)
Deploying a Lambda Function with AWS SAM
AWS SAM is the Serverless Application Model, which allows you to write a template describing the Lambda function, and any related resources you need to run your Lambda. The SAM template uses the YAML markup language and is based on AWS Cloudformation. After reading the template, SAM performs the deployment for you, and you can watch Cloudformation events to identify what the outcome was.
We are going to use the template in the above repository to deploy the function. I won't go into detail breaking down all of the items in the template for now. That information can be found in the documentation above or in other blogs. But I do want to highlight the two resources used here, the LambdaEnvVarsFunction which is of the type AWS::Serverless::Function and the LambdaLogGroup, which is the definition of our AWS::Logs::LogGroup CloudWatch log group, where our Lambda output will be collected.
Install SAM CLI
You'll need to install a few components before running SAM. I would recommend running this in a Python virtual environment (venv), and I will provide basic steps here to set that up.
From your terminal run these commands (this is on a Linux system):
- Step 1: Make sure you have Python 3.8 (you may have to install this first, on Fedora Linux the command would be
sudo dnf install python3.8)
$ which python3.8
/usr/bin/python3.8
- Step 2: Create a Python 3.8 virtual environment
$ python3.8 -m venv venv38
$ ls venv38
bin include lib lib64 pyvenv.cfg
- Step 3: Activate the virtual environment
$ . venv38/bin/activate
$ which python
~/FAKE_PATH/lambda-env-vars7/venv38/bin/python
$ which pip
~/FAKE_PATH/lambda-env-vars7/venv38/bin/pip
- Step 4: Install SAM CLI NOTE: This installs Docker in the Python virtual environment
$ pip install aws-sam-cli
Collecting aws-sam-cli
Using cached aws_sam_cli-1.37.0-py3-none-any.whl (5.0 MB)
Collecting boto3==1.*,>=1.18.32
Using cached boto3-1.20.46-py3-none-any.whl (131 kB)
...
Now you can move on to the next step, where we run the Lambda function locally.
Running Your Lambda Function with SAM Local
Before we deploy to AWS itself, we want to test our code by invoking the Lambda function locally. What is this magic you ask? SAM provides a utility command called sam local invoke which allows your function to run locally in a Docker container to test that it runs as expected before deploying to AWS.
NOTE: We assume you haven't left the terminal session in step 4 above. If you have, activate your Python virtual environment as in step 3 above.
Run sam local invoke as described in the Git repository (this uses test events supplied with Lambda function in the repository)
$ sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-python3.8:rapid-1.37.0-x86_64.
Mounting /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6 Version: $LATEST
Hello there this is the body: {'_HANDLER': 'app.lambda_handler', 'AWS_REGION': 'us-east-1', 'AWS_EXECUTION_ENV': 'AWS_Lambda_python3.8', 'AWS_LAMBDA_FUNCTION_NAME': 'LambdaEnvVarsFunction', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': '128', 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_LOG_GROUP_NAME': 'aws/lambda/LambdaEnvVarsFunction', 'AWS_LAMBDA_LOG_STREAM_NAME': '$LATEST', 'LANG': 'en_US.UTF-8', 'TZ': ':/etc/localtime', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'PATH': '/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin', 'LD_LIBRARY_PATH': '/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib', 'PYTHONPATH': '/var/runtime', 'AWS_LAMBDA_RUNTIME_API': '127.0.0.1:9001'}
END RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6
REPORT RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6 Init Duration: 0.11 ms Duration: 100.72 ms Billed Duration: 101 ms Memory Size: 128 MB Max Memory Used: 128 MB
{"statusCode": 200, "body": "{\"_HANDLER\": \"app.lambda_handler\", \"AWS_REGION\": \"us-east-1\", \"AWS_EXECUTION_ENV\": \"AWS_Lambda_python3.8\", \"AWS_LAMBDA_FUNCTION_NAME\": \"LambdaEnvVarsFunction\", \"AWS_LAMBDA_FUNCTION_MEMORY_SIZE\": \"128\", \"AWS_LAMBDA_FUNCTION_VERSION\": \"$LATEST\", \"AWS_LAMBDA_LOG_GROUP_NAME\": \"aws/lambda/LambdaEnvVarsFunction\", \"AWS_LAMBDA_LOG_STREAM_NAME\": \"$LATEST\", \"LANG\": \"en_US.UTF-8\", \"TZ\": \":/etc/localtime\", \"LAMBDA_TASK_ROOT\": \"/var/task\", \"LAMBDA_RUNTIME_DIR\": \"/var/runtime\", \"PATH\": \"/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin\", \"LD_LIBRARY_PATH\": \"/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib\", \"PYTHONPATH\": \"/var/runtime\", \"AWS_LAMBDA_RUNTIME_API\": \"127.0.0.1:9001\"}"}
Deploy to AWS
NOTE: Before deploying you need to setup your AWS credentials properly.
In order to deploy the Lambda function, we'll follow the steps in the lambda-env-vars repository:
- Step 1: Run
sam build
$ sam build
Building codeuri: /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/env_vars runtime: python3.8 metadata: {} architecture: x86_64 functions: ['LambdaEnvVarsFunction']
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided
- Step 2: Run
sam package(this creates an S3 bucket, which must be removed later)
$ sam package --output-template-file packaged.yaml --resolve-s3
Creating the required resources...
Successfully created!
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
A different default S3 bucket can be set in samconfig.toml
Or by specifying --s3-bucket explicitly.
Uploading to eb6c7534d67a5b044653e48f2802a548 452695 / 452695 (100.00%)
Successfully packaged artifacts and wrote output template to file packaged.yaml.
Execute the following command to deploy the packaged template
sam deploy --template-file /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/packaged.yaml --stack-name <YOUR STACK NAME>
- Step 3: Run
sam deploy
$ sam deploy --template-file packaged.yaml --region us-east-2 --capabilities CAPABILITY_IAM --stack-name lambda-env-vars --resolve-s3
Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
A different default S3 bucket can be set in samconfig.toml
Or by specifying --s3-bucket explicitly.
Deploying with following values
===============================
Stack name : lambda-env-vars
Region : us-east-2
Confirm changeset : False
Disable rollback : False
Deployment s3 bucket : aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
Capabilities : ["CAPABILITY_IAM"]
Parameter overrides : {}
Signing Profiles : {}
Initiating deployment
=====================
Uploading to 77524ed205c6d82d5487433831c24c0f.template 1635 / 1635 (100.00%)
Waiting for changeset to be created..
CloudFormation stack changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation LogicalResourceId ResourceType Replacement
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd AWS::Lambda::Permission N/A
+ Add LambdaEnvVarsFunctionRole AWS::IAM::Role N/A
+ Add LambdaEnvVarsFunction AWS::Lambda::Function N/A
+ Add LambdaLogGroup AWS::Logs::LogGroup N/A
+ Add ServerlessRestApiDeploymentcaa6ada684 AWS::ApiGateway::Deployment N/A
+ Add ServerlessRestApiProdStage AWS::ApiGateway::Stage N/A
+ Add ServerlessRestApi AWS::ApiGateway::RestApi N/A
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Changeset created successfully. arn:aws:cloudformation:us-east-2:592431548397:changeSet/samcli-deploy1643607514/85a31faf-1358-47ff-9eeb-4428020bb482
2022-01-30 22:38:46 - Waiting for stack create/update to complete
CloudFormation events from stack operations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus ResourceType LogicalResourceId ResourceStatusReason
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS AWS::IAM::Role LambdaEnvVarsFunctionRole -
CREATE_IN_PROGRESS AWS::IAM::Role LambdaEnvVarsFunctionRole Resource creation Initiated
CREATE_COMPLETE AWS::IAM::Role LambdaEnvVarsFunctionRole -
CREATE_IN_PROGRESS AWS::Lambda::Function LambdaEnvVarsFunction -
CREATE_IN_PROGRESS AWS::Lambda::Function LambdaEnvVarsFunction Resource creation Initiated
CREATE_COMPLETE AWS::Lambda::Function LambdaEnvVarsFunction -
CREATE_IN_PROGRESS AWS::Logs::LogGroup LambdaLogGroup -
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_IN_PROGRESS AWS::ApiGateway::RestApi ServerlessRestApi Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::RestApi ServerlessRestApi -
CREATE_IN_PROGRESS AWS::Logs::LogGroup LambdaLogGroup Resource creation Initiated
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeploymentcaa6ada684 -
CREATE_IN_PROGRESS AWS::Lambda::Permission LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd Resource creation Initiated
CREATE_IN_PROGRESS AWS::Lambda::Permission LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd -
CREATE_COMPLETE AWS::Logs::LogGroup LambdaLogGroup -
CREATE_IN_PROGRESS AWS::ApiGateway::Deployment ServerlessRestApiDeploymentcaa6ada684 Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::Deployment ServerlessRestApiDeploymentcaa6ada684 -
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage -
CREATE_IN_PROGRESS AWS::ApiGateway::Stage ServerlessRestApiProdStage Resource creation Initiated
CREATE_COMPLETE AWS::ApiGateway::Stage ServerlessRestApiProdStage -
CREATE_COMPLETE AWS::Lambda::Permission LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd -
CREATE_COMPLETE AWS::CloudFormation::Stack lambda-env-vars -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CloudFormation outputs from deployed stack
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key LambdaEnvVarsApi
Description API Gateway endpoint URL for Prod stage for Lambda Env Vars function
Value https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/
Key LambdaLogGroup
Description Cloudwatch Log Group ARN
Value /aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp
Key LambdaEnvVarsFunctionIamRole
Description Implicit IAM Role created for Lambda Env Vars function
Value arn:aws:iam::592431548397:role/lambda-env-vars-LambdaEnvVarsFunctionRole-L05B77IT7WK0
Key LambdaEnvVarsFunction
Description Lambda Env Vars Function ARN
Value arn:aws:lambda:us-east-2:592431548397:function:lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Successfully created/updated stack - lambda-env-vars in us-east-2
The sam deploy command has deployed the Lambda function to AWS and returned the API Gateway endpoint which we can use to invoke the Lambda with an API call. We need the API Gateway URL from the above output, look for the LambdaEnvVarsApi key. In our example above, this is:
https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/
GET Request to Invoke the Lambda Function
Now that we have the API Gateway URL (also known as the endpoint,) we can use a tool like curl to perform an HTTP GET request against the endpoint. Because we defined the Events stanza with a Type of Api, and Method get, SAM has deployed an API Gateway resource to trigger our Function. (More detailed discussion on this implicit API Gateway resource here).
We will use the command curl to perform a GET request as below:
$ curl -X GET https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/
{"_HANDLER": "app.lambda_handler", "AWS_REGION": "us-east-2", "AWS_EXECUTION_ENV": "AWS_Lambda_python3.8", "AWS_LAMBDA_FUNCTION_NAME": "lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp", "AWS_LAMBDA_FUNCTION_MEMORY_SIZE": "128", "AWS_LAMBDA_FUNCTION_VERSION": "$LATEST", "AWS_LAMBDA_LOG_GROUP_NAME": "/aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp", "AWS_LAMBDA_LOG_STREAM_NAME": "2022/01/31/[$LATEST]c2b9b9c953154c8daa7c9b2c0637b70e", "LANG": "en_US.UTF-8", "TZ": ":UTC", "LAMBDA_TASK_ROOT": "/var/task", "LAMBDA_RUNTIME_DIR": "/var/runtime", "PATH": "/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin", "LD_LIBRARY_PATH": "/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib", "PYTHONPATH": "/var/runtime", "AWS_LAMBDA_RUNTIME_API": "127.0.0.1:9001"}
On line 2, we can see the response JSON, which is what the
Lambda function returned to us via the API Gateway. And we see all the environment variables that we had expected, just as in the sam local invoke command earlier.
But how can we observe the output of the print statement at line 50?
To do that we look at the CloudWatch log output. Browse to your AWS console, and search for the Cloudwatch log group name that was in the Outputs section of the sam deploy command output.
In our case the LambdaLogGroup name was: /aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp
When I browse there in CloudWatch, I see the "Hello there" line in the output, which was sent to CloudWatch because we performed a print to standard output within the Lambda function code:
What Just Occurred? What's Next?
Congratulations, you just deployed and invoked your first Lambda function.
Take some time to examine the output of the function in CloudWatch Logs, as well as what was returned during your curl command.
Challenge 1: If you've heard of the Postman tool, you can also call the API endpoint from there. Give it a shot.
Challenge 2: You can display the HTTP headers when calling the API endpoint. How would you see the headers with the curl command? How would you see them with Postman?
Challenge 3: Read more about Lambda functions, SAM, Serverless Framework, CDK. Build your own Lambda and deploy it. Let me know how it goes on Twitter @fshwsprr.
Clean Up AWS Resources
When you are done playing around with this Lambda function, run these commands to remove your resources:
aws cloudformation delete-stack --stack-name lambda-env-vars --region us-east-2-
aws s3 rb s3://aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0 --forceNOTE: This requires you to look at the output of yoursam packageorsam deploycommand above to get the name of the S3 bucket to remove! - Finally, you can deactivate your python virtual environment with the command:
deactivate
Summary
If you've read this far, the important takeaways about AWS Lambda in my opinion, are:
- The concept that Lambda functions are only one part of the cloud serverless landscape.
- Understanding invocation options and how your single function gets called by AWS.
- The fact that you can locally invoke the function in SAM Local before you deploy to AWS.
- Anything you print inside your function will be written to CloudWatch Logs (assuming you created a Log Group for your Lambda.)
Thanks for reading! Enjoy your serverless and Lambda journey!
Cover photo by Avel Chuklanov on Unsplash