Tag Archives: Heroku

How Flask, Heroku & Alembic Play Together

I just spent a couple days getting up to speed on database migrations in general, how to make it work with Flask and Postgres and how to make them work on Heroku. There is some information out there but it took a little time hunting down what I needed; thus, I’ve summarized some of the main steps to help get others up and running with Flask, Heroku and Alembic.

Why migrate? Best practice is to avoid recreating your database(s). Usually you just want to make changes to the existing database(s) and track those changes. If at any time you need to go back to a previous version, the migration docs will help you easily revert to an old version / schema and then upgrade back to the most recent depending on your needs.

Migration Types: They are usually discussed as either schema (structure of the database) or data (the stored stuff – aka creamy filling). Sometimes they take place at the same time and sometimes not.

A Flask Migration Package Option: Alembic

Previous posts included how much I leveraged Flask Mega Tutorial for building a web application (app). In regards to migrations, Flask Mega primarily focuses on SLQLite which is not as helpful because Postgres is needed for Heroku deployment.

Alembic is a migration tool that is better maintained than the sqlalchemy-migrate package, and it is from SQLAlchemy’s author.  There is documentation on how to setup and run database migrations. I’ve listed out some of the main, basic steps needed to setup alembic, migrate revisions and run it all on Heroku below. This is assuming you’ve already created a database locally as well as added it on Heroku and promoted it as the default.

Where there is a $ or => the words following should be run in the command line and yes, these directions are based on Mac. 

How to Start

  1. Install alembic $ pip install alembic
  2. Add to requirements $ pip freeze > requirements.txt
  3. Initialize it inside your project root folder $ alembic init alembic
  4. Ignore the .ini file for this basic installation
  5. Change env.py with directions at this link.  If the app is in the Flask Mega structure then just replace with app but make sure your Config file is setup for postgres:

import os
if os.environ.get(‘DATABASE_URL’) is None:
SQLALCHEMY_DATABASE_URI = ‘postgresql://localhost/<db_name>’

  1. Create first revision $ alembic revision -m “First revision.”
  2. Find and add change scripts for upgrade and downgrade to the new revision file
  3. Migrate $ alembic upgrade head
  4. Repeat 6 – 8 for further local revisions and migrations
  5. Revise Procfile:

migrate: alembic upgrade head
upgrade: alembic upgrade +1
downgrade: alembic downgrade -1

  1. Git add all changes $ git add .
  2. Git commit $ git commit -m “Procfile and running revisions>”
  3. Push to Heroku $ git push heroku master
  4. Run alembic migrate on Heroku $ heroku run alembic upgrade head

If you get something like the following then it went well:

Running `alembic upgrade head` attached to terminal… up, run.****
INFO [alembic.migration] Context impl PostgresqlImpl.
INFO [alembic.migration] Will assume transactional DDL.
INFO [alembic.migration] Running upgrade None -> *********, Create account table
INFO [alembic.migration] Running upgrade ******** -> ********* Add zoomlevel to locations.
INFO [alembic.migration] Running upgrade ******** -> *********, Test add favorite.
INFO [alembic.migration] Running upgrade ******** -> *******, Test add favorite.

To double check changes went through on Heroku, here are a couple commands:

  1. Launch Postgres interactive environment on Heroku $ heroku pg:psql
  2. Look at tables => \dt
  3. Look at table schema => \d <table name>

That should cover it to get started. Creating a revision file and migrating locally as well as on Heroku are steps that should be repeated for each new migration.

As always there is more info out there for nuances and complexities to migration. There is also the autogenerate functionality that can automatically define change scripts for things like a schema change, but it is limited in what it can do. Check that out in the reference documents. No matter what, I recommend always taking a look at the revision file just to make sure it will do what you need. And have fun migrating.



Deployment is not the Devil (Flask & Heroku Tips)

On a previous post I claimed deployment is the devil because after several successful (not always easy) deployments, pushing up my Sun Finder app proved elusive. I seriously wanted to scratch my eyes out at times with all the errors and issues. Still it was a good learning experience (one that I fought against but a good one all the same), and I did finally deploy as of last week! Check it out at sunfinder.io.

So deployment isn’t all bad but it sure can be frustrating if there is a ton more work that is needed to deploy after the long haul of web app development.

To help others new to deployment and especially working with Flask, here are some things I learned along the way.

First off, if you are working with Flask then use this help page as a starting point on how to develop and setup apps on Heroku.  On the left side of the page, there are links that provide similar support for other frameworks. Just make sure to setup a repository to store your project and configure remote access.

1. Local Database

The biggest deployment challenge I had was importing a pre-populated database from my personal computer (local).

Heroku provides a Postgres add-on from Heroku Postgres which adds a remote database onto the application. When a remote database is provisioned, a config variable is assigned to the repository which usually includes a color in the name. This variable is a reference to the URL where the empty database is accessible.

DB Remote Storage Option
In order to load a pre-populated database, it has to be stored remotely in a service provider like AWS S3 (Amazon Web Services Simple Storage Service) or Dropbox and then imported to Heroku. S3 is a popular storage solution because its been around for a while and is known for its optimization. I haven’t looked into Dropbox as a solution, but I suspect its a good option as well.

High-level directions on how to setup S3 for Heroku are provided at this help page. AWS also provides pretty extensive usage directions.

IAM: User & Permission Setup
After signing onto AWS, go to the IAM section under Deployment & Management section. In this area, setup a user and obtain security credentials. Make sure to capture the credentials  (Access Key ID and Secret Access Key) for later reference. Then setup a group and assign access permissions. Go for the admin permissions setup if its just one person. Once the group is created, click on it and look below the tab section for the option to add a user. Make sure to add the user to the group.

S3: Bucket Setup
From IAM, return to the main section by just clicking on the box in the top left corner and choose S3 under Storage & Content Delivery section. In this area, create a bucket (aka root folder) to store the database. Make sure to create the bucket in the region where the app is stored which for Heroku is the US Standard region. This is important because S3 is free for in region data transfer rates. Also, make sure the group is granted access permission to this bucket. At this point the database can be uploaded.

DB: Compression,  Upload & Configuration
In order to upload a database, compress the local copy first which is also referred to as dumping. Heroku’s directions on how to create a compressed version of the file can be found at this page.  Create the dump file and open the AWS bucket in order to access the upload option. Once the file is uploaded, add the AWS security credentials into Heroku configuration following the directions on the Heroku S3 help page.

DB: Import
I followed the import directions on Heroku’s help page that explained compression but there were a number of errors (like “invalid dump format: /tmp/…/sunfinder.dump: XML  document text”). In order to resolve these problems, I logged into the Heroku Postgres site.  It showed all of my Postgres provisioned databases and when I clicked on one of the databases, I was able to see connection settings and statistics. There is an icon of two arrows pointing in opposite directions in the top, right corner that provides a list of additional options. There I clicked on PG Restore and found more explicit command directions for import. The only part of the command that needed to be changed is to swap “your data file” with the dump file name that is inside the bucket. This resolved my errors and enabled database setup.

Just remember that any changes made to the local database need to be compressed, uploaded onto AWS again and imported in order for it to be seen in the remote application.

2. Static Content

When I first read the Heroku S3 help page, I mistakenly thought I had to store all of my static content on S3 (e.g. img, js, css). Granted previous deployments seemed to work and not require this, but I couldn’t get the css and js files to load correctly on my application.  I was getting a 403 error with a link to an XML page that said “Access Denied”. 

So I loaded all the static content on AWS S3 and made it public. This actually made the application work once I changed the static file references to the new AWS location and links. Then I finally figured out that the Heroku application key configuration was incorrect and thus, the problem. So I rolled back my changes to keep the static reference links internal vs. pointing at AWS.

Using AWS to store and reference static files is more useful in situations where there is a significant amount of content and/or users are loading content onto the application. There is a lot of literature out there that provides more details.

3. Heroku Configuration & Updates

In the process of setting up the Heroku repository, don’t forget configuration. It can make things go wonky like 403 errors if its wrong. In the errors around static page loads, my configuration of the Flask app secret key was incorrect on Heroku. Definitely read this link and make sure to load all the secret keys that are needed.

Additionally, make sure to git add and commit changes in order to push updates to Heroku. If a change doesn’t show on the remote site, it’s possible that it wasn’t committed before the Heroku push.

4. Not All Browsers are the Same

Another error/warning I found was in the he Chrome browser’s Inspect EIement console: “The page at … displayed insecure content for ….”. The dots represent a link that the warning referenced. My current hypothesis is this is because I loaded HTTPS Everywhere on my computer and some of the links in my site point to sites that do not use secure socket layer protection. Its just not an option at some sites. This is just a warning and does not prevent my application from functioning. If I learn more, I will update this post.

One thing that I was reminded of while troubleshooting my app is that not all browsers function the same way. So I opened my app in a different browser to check if some errors and warnings would go away. Its just one more way to test the application and help narrow down the problems. Granted there are many browsers and versions of browsers that can impact functionality and plenty of materials on how to develop for all those variations.

5. When All Else Fails – Reboot

Initially I made the first Sun Finder deployment attempt in June. When I returned to deployment in Aug., I tried working with the already established Heroku repository and configuration. At some point in my error tackling, I realized its better to just reset and restart. So I deleted the Heroku repository and created a new one. This didn’t resolve all my errors, but it did help clear out some of the more mysterious ones (e.g. the ones unknowingly created during the learning process).

For those out there working on Heroku deployment, I still stick by a previous post comment that Rails is an easy experience, but it is very doable to achieve deployment with other frameworks. If anything it can be a little more educational at times. Just don’t let the challenges keep you from that final hill to launch.