How to Deploy a Serverless WSGI App using Zappa, CloudFront, RDS, and VPC
This post is more of a brain dump than a step by step guide. Every application is different some of the services, libraries, and techniques may not apply to you. Feel free to skip over them. This is how I deploy my apps with Zappa. I hope this post helps.
These steps are AWS focued and I only listed them here so you know what happens. I recommend you write a CloudFormation template (use Troposphere) to automate these tasks for future deployments.
Create Two S3 BucketsYou should only have to complete this stage once even if you create a CloudFormation template in the future. We are creating one bucket for Zappa to store our code when deploying (by default Zappa will only store it there briefly while deploying or updating the Lambda function) and another to store static files (css, images, and js).
Configure your VPCMaybe your app doesn't need VPC. I personally believe if you are running a database that uses RDS and ElastiCache you need a VPC. Below is a list of things I recommend to get a Zappa powered app working with VPC, RDS, and Elasticache.
What you'll typically need
Public and Private SubnetsCreate at least two public and private subnets in different availability zones.
NAT GatewayEnables Lambda functions and instances (RDS and Elasticache) in a private subnet to connect to the Internet or other AWS services, but prevent the Internet from initiating a connection with those instances. Associate your NAT Gateway with one of your private subnets. This is needed if you are planning on using any API (MailChimp, Twilio, etc.) other than S3's.
Route TableControls the routing for your subnets. You could setup a different route table for each subnet but for the purposes of this guide we're setting one for the private subnets and one for the public ones. After we associate our subnets we will route 0.0.0.0/0 traffic to our NAT Gateway for our private subnets and 0.0.0.0/0 to our Internet Gateway for our public subnets.
VPC Endpoint for S3This is needed so your Lambda function can communicate with S3's API. This comes in handy later when we run zappa manage "collectstatic --noinput".
Create a MySQL, Aurora, or MariaDB instance(s)Create a MySQL instance(s) and associate it with the VPC and subnets you created in the previous step.
Create Redis Cluster with ElastiCacheCreate a Redis instance(s) and associate it with the VPC and subnets you created in step 2.
Code & Local Environment Steps
You got it!! These are the steps that will require a change in your code or your local environment. Mainly the stuff you normally do when you start a new Django project.
Install virtualenv, virtualenvwrapper, and eva globally (Optional)This step is only useful if you plan on working on this project locally without something like Docker or Vagrant. If you just install eva it will install the other two.
sudo pip install eva
Configure virtualenvwrapperAdd three lines to your shell startup file (.bashrc,.profile, etc.) to set the location where the virtual environments should live, the location of your development project directories, and the location of the script installed with this package:
Create virtualenv and .env file
mkvirtualenv -a ~/workspace/yourblog ybNext, create an .env (example)file and then run the workon command again to load the environment variables.
Install zappa, envs, django, mysql-python, django-storages, boto, and django-redis (Partially Optional)
pip install zappa envs django mysql-python
pip install django-storages boto django-redis
Run django-admin startproject to well...start the projectOnly needed if you are deploying a Django app.
django-admin startproject yourblog
Replace the default settings.py values with environment variables, create an .env file, and create dev stage json settings file (yourblog-dev.json).Only needed if you are deploying a Django app. Click here for the example files
Create a custom storage backendOnly needed if you are deploying a Django app. Create a new package named util and in that package create a file named custom_storages.py (click here for the example files)
List of the various Zappa steps that you need to run.
Run zappa initThis command just creates a zappa_settings.json configuration file. You will answer questions pertaining to the S3 buckets you created earlier and the default stage for your app. After the initial setup is complete open the settings file to reference the environment variable JSON file you created earlier and the private subnets and security groups you created. Here is an example file.
Run zappa deploy devThis command will deploy the app to Lambda and setup API Gateway. You will only run this command one time. After the first time you will only need to run the update command.
Run zappa manage dev migrateThis command is only needed for Django projects. The zappa manage migrate command is the equivalent of python manage.py migrate generally used for Django apps.
Run zappa certify devRun this command to request a SSL cert from the Let's Encrypt project and apply it to your API Gateway stage. It uses the domain you entered above in the zappa_settings.json file for the stage you are deploying.
Run zappa manage "collectstatic --noinput"This command runs another one of Django's management commands, collectstatic. It will gather all of the static files (css, js, images, and videos) and upload them to S3 from inside the Lambda function. Since the files were already uploaded inside of the zip file Zappa deploys the upload to S3 is really fast.
CloudFront Steps (Optional)
Here is where my approach differs from most. I like to use CloudFront for whole site delivery for dynamic websites and APIs. This allows your site to load much faster across the globe. Depending on the TTLs you require and what needs to be cached on an per-user basis you can also bear much heavier loads because the site is served from cache at the edge. There are a lot of large scale applications that use this model (ex. Slack) it is not that different from using Varnish or Fastly when you really think about it.
Create a SSL Cert with AWS Certificate ManagerCreate a SSL cert for the URL your end users will see. I generally create a request that has the apex domain (yourblog.com) but also supports wildcard (*.yourblog.com). These are not the same thing but can be achieved on the same cert request.
Create a Multi-Origin, Multi-Behavior CloudFront DistroBelow I'm going to list the some of your CloudFront ditribution's attributes you should put some thought into.
- Your distribution should be a web (not RTMP) distribution
- Use the domain returned after you completed the zappa deploy command earlier as the origin domain name (should be something like to dev-yourblog-apigw.yourdomain.com)
- Leave origin path blank
- Type whatever you want in Origin Id
- Origin Custom Headers can be left blank as well
- Viewer Protocol Policy should be always be set to Redirect HTTP to HTTPS
- Allowed HTTP Methods should be set to GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE unless it is the behavior for the static files on S3 then you can just set it to GET, HEAD
- Forward Headers should be set to None (improves caching) if most of your site doesn't require the user to login and you don't have any forms that require CSRF protection. If require headers then whitelist the following Accept, Accept-Language, Cache-Control, HTTP_X_CSRFTOKEN, x-csrfmiddlewaretoken, Upgrade-Insecure-Requests, and Referer.
- Object Caching should be set to whatever you're comfortable with. Here is what I did for this blog, for the default behavior I set the TTL to 3600 seconds, for the homepage I set it for 300 seconds, for the django admin I set it for 0 seconds, for the static files and site media I set it for 3600 seconds.
- Forward Cookies this falls into the same category of what does your application require. If you need cookies for authentication or some other purpose then you can select All or Whitelist (at this time API Gateway only forwards one cookie so the Zappa project decided to create only one, zappa).
- Query String Forwarding and Caching should almost always be Forward All
- Smooth Streaming should be No
- Restrict Viewer Access should be No
- Compress Objects Automatically should be Hell Yes
- How many origins should I have? Two, one for your application and one for the S3 bucket that holds your static files.
- Why do I have an origin for my S3 bucket too? I like doing it this way because I cut down the DNS latency by converting my references to images, css, and js from something like this https://cdn.yourdomain.com/yourblog-prod-static/cs... to someting like this /yourblog-prod-static/css/base.css. Plus, its cool as hell.
Hopefully this post helps you get started or come up with your own strategy. Above all, I hope it helps your organization save money and time.