Category Archives: Heroku

Makahiki Testing Part 2: Heroku Installation

Overview

Continuing the Makahiki series started by my earlier post, I set up a Makahiki installation on Heroku which used an Amazon AWS S3 bucket to host static files (images, CSS scripts, Javascript, etc), according to the official guide. The Heroku documentation recommends using AWS to host static files as a complement to its “ephemeral file system,” which only stores the latest running version of an application’s code. Files in a bucket can be managed more similarly to a traditional file system, as opposed to files which are stored in a Heroku slug.

Unlike the previous localhost installation, I ran into several problems. Though trying to coordinate Amazon and Heroku deployment did add some additional complexity, the biggest obstacle came from mistakes in my local configuration that resulted in my having to erase Heroku apps and start over several times.
I do not have any experience in working with cloud-hosted applications apart from my other Heroku-based assignments, but the installation instructions that we were given were simple enough that they did not pose problems for me.

AWS S3 Buckets: The Importance Of US Standard

The Heroku and Makahiki developers both suggest locating your bucket in the US Standard region. As the Heroku documentation points out, Heroku uses the US East region, and transfers between US East and US Standard are free. Normally, however, Amazon’s default settings for the bucket are set by auto-detection of where your traffic originated from. As of this article, I am in Hawaii, which is auto-detected as US West (Oregon). It is important to set US Standard as the region when the bucket is created, because bucket regions cannot be changed after the bucket has been created. In addition, buckets which are not declared as US Standard use a base URL that is different from the s3.amazonaws.com URL that Makahiki expects.

Virtualenv Setup for a Heroku Upload

I configured a new virtualenv, makahiki-heroku, to avoid conflicts with the makahiki virtualenv created in the last article. Following the guide, I set the environment variables manually, as specified in the guide:

$ export MAKAHIKI_ADMIN_INFO=admin:Dog4Days56
$ export MAKAHIKI_AWS_ACCESS_KEY_ID=<AWS access key id>
$ export MAKAHIKI_AWS_SECRET_ACCESS_KEY=<AWS secret access key>
$ export MAKAHIKI_AWS_STORAGE_BUCKET_NAME=<AWS S3 bucket name>

The user can also put these variables in the virtualenv permanently by editing the $WORKON_HOME//bin/postactivate script. However, storing credentials in a static file runs the risk of exposing your S3 credentials if someone gets unauthorized access to your computer. Note also that the
heroku config:add command will not work yet because the Heroku app has not been created. It is created later, by the initialize_instance.py script.

The initialize_instance.py Script

Though running the initialization script is not necessarily the hardest step, I think it is the step that has the most room for error. It is important not to run this script with sudo. Switching to a root user context erases your user’s environment variables, so the script runs with its Amazon S3 variables not set and will fail to upload files to S3.

Running the script with sudo overrides the environment variables.

Above: Do not use the initialize_instance.py script with sudo. If you do, you will get a public key error and the S3 upload operations will fail.


When I resolved the environment variables problem, I went to my application page and saw that the stylesheets and images had failed to load:
The page without stylesheets or Javascript.

Above: The initialize_instance.py script failed to upload the static assets, including scripts and images.


Inspecting the Amazon S3 bucket revealed that the site_media/static directory, which contained images, scripts, and page templates, was missing. The script was detecting the site_media/static directory, but not using s3put to push the files to S3:
The site_media/static directory is not uploaded.

The script failed to push the static files to Amazon S3.

There was one other error in the upload involving a missing Django.core.management module. As T.A. Yongwen Xu and my classmate Zach Tomaszewski have pointed out, this happened because I did not run pip install -r requirements.txt, which installs Makahiki’s dependencies for a local project. After this step, the uploads were successful. Though it is probably something a user could be expected to remember, the instructions should still specify that running pip install -r requirements.txt is needed for the Heroku upload to work.

The Makahiki home page on Heroku.

The Makahiki home page on Heroku.

Suggested Improvements

The instructions are easy enough to understand. However, they do not specify US Standard as the desired bucket region and do not specify that the user needs to run pip install -r requirements.txt before running the initialize_instance.py script. The Heroku S3 tutorial that is linked to by the official guide recommends US Standard as the bucket region, but this is not specified in the manual, and users who already know how to install S3 may think that they do not need to see the instructions and end up with a bucket that is in the wrong region. Likewise, pip install -r requirements.txt may seem like an obvious requirement for a project that uses pip to handle dependencies, but the instructions should explicitly require it to be run.

Conclusion

Installing Makahiki on Heroku took much longer than installing it locally. Most of the extra time was the result of needing to set up an S3 account or having to erase and re-upload the Makahiki application on Heroku several times due to problems with missing S3 credentials or not having dependencies installed. Most of my problems with S3 uploads resulted from not using pip to install dependencies, and I think this is a step which should be included in the documentation. Despite all this, it is still easier than I expected it to be to get a cloud-hosted application up and running for a free or cheap monthly price, which demonstrates the lengths the Makahiki developers and companies such as Amazon and Heroku have gone to in order to make the setup process understandable for each of their applications or web platforms.

Advertisements

Corrections to Heroku Posts

Corrections

Some corrections have been made to my previous posts as a result of errors pointed out by instructors and classmates. The articles listed below have been updated to reflect the corrections.

Corrections to Lessons Learned from Deploying Django on Heroku:

1. Step 2: Corrections to the section on pip freeze > requirements.txt:

NEW VERSION

(CORRECTION): Contrary to both previous versions of this post, you can run pip freeze at any time. This will change requirements.txt by adding the new dependencies that have been installed.

If you edit requirements.txt manually, run pip install -r requirements.txt to install any new dependencies:

(venv)mainuser@webdevbox:~/hellodjango$ pip install -r requirements.txt

PREVIOUS VERSION

(CORRECTION): An earlier version of this post stated that the pip freeze command had to be run while the server was running. Actually, as long as the pip install django psycopg2 dj-database-url command succeeded, the new dependencies will be added to the file when you run pip freeze. DO NOT RUN pip freeze WHILE THE SERVER IS RUNNING.

2. Step 2: Corrections to the section on DATABASES in settings.py:

NEW VERSION

Next, the tutorial states “Next, configure the application for the Heroku environment, including Heroku’s Postgres database. The dj-database-url module will parse the values of the DATABASE_URL environment variable and convert them to something Django can understand.” Add these lines to the end of helloheroku/settings.py as specified in the tutorial:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

(CORRECTION) An earlier version of this post stated that values in settings.py’s DATABASES section needed to be changed. This was unnecessary. The DATABASE_URL environment variable must be set with Linux’s export command. Thanks to Zach Tomaszewski for pointing this out.

(venv)mainuser@webdevbox:~/hellodjango$ export DATABASE_URL=postgres://testadmin:testing@localhost/django_db

Here, testadmin is a postgresql user, testing is the password for that user, and django_db is the name of the local database. On the local machine, dj_database_url.config() reads the value of DATABASE_URL specified by export and returns its value, which becomes the value of DATABASES. On Heroku, dj_database_url.config() will read the value of DATABASES from Heroku’s environment variables instead.

PREVIOUS VERSION

Next, the tutorial states “Next, configure the application for the Heroku environment, including Heroku’s Postgres database. The dj-database-url module will parse the values of the DATABASE_URL environment variable and convert them to something Django can understand.” Before this will work, the DATABASES section should be changed to match the database you created earlier:

(venv)mainuser@webdevbox:~/hellodjango$ nano hellodjango/settings.py
# Some of the file omitted
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'django_testdb',
        'USER': 'testadmin',
        'PASSWORD': 'testing',
        'HOST': '',
        'PORT': '',
    }
}
# Now go to the very end of the file and add these lines, as in the tutorial:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

3. Corrections to Step 3:

Note: This section was accidentally titled Part 3.

NEW VERSION

If, like me, you had at first forgotten to configure PostgreSQL and do an export, you will see errors like this if you try to run the server locally:

[…]

If this happens, set up the database as described in Step 1 and edit settings.py as described in Step 2, then try again.

PREVIOUS VERSION

If, like me, you had at first forgotten to configure postgreSQL, you will see errors like this:

[…]

If this happens, set up the database as described in Step 1 and edit settings.py to configure the DATABASES values as described in Step 2, then try again.

4. Corrections to Step 4:

NEW VERSION

  1. In settings.py, change DEBUG to False. You should always do this before deploying a Django application with gunicorn, because gunicorn does not respect the DEBUG flag. In general, always change DEBUG to False before deploying any production Django application.
    DEBUG = False
    
  2. Add the line gunicorn==0.16.1 to your requirements.txt
  3. Change your Procfile’s web: line to web: gunicorn hellodjango.wsgi
  4. Uncorrected sections omitted…

PREVIOUS VERSION

  1. Add the line gunicorn==0.16.1 to your requirements.txt
  2. Change your Procfile’s web: line to web: gunicorn hellodjango.wsgi
  3. Uncorrected sections omitted…

Corrections to Lessons Learned from Deploying Django on Heroku, Part 2:

Corrections to Step 2:

NEW VERSION

Now open helloheroku/settings.py in your text editor of choice and edit the following lines at the end of the file:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

The file for the example project used here should already have the dj_database_url lines. The SECURE_PROXY_SSL_HEADER setting is used by Django to confirm that a request served from behind a proxy is being served as HTTPS.

(CORRECTION) Ignore the DATABASES section; dj_database_url.config() will import the values for DATABASES based on what is specified in the export command.

OLD VERSION

Now open helloheroku/settings.py in your text editor of choice and edit the following lines:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle',
        'NAME': 'helloheroku_db',        # Or path to database file if using sqlite3.
        'USER': 'testadmin',             # Not used with sqlite3.
        'PASSWORD': 'testing',           # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
    }
}

# Now go to the end of the file:
# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

The file for the example project used here should already have the dj_database_url lines. The SECURE_PROXY_SSL_HEADER setting is used by Django to confirm that a request served from behind a proxy is being served as HTTPS.

Problems with Steps 3, 4, and 4A:

As it turns out, it’s supposed to be possible to get the site working with gunicorn as the server. As this would require substantial changes to Parts 3, 4, and 4A, I might just write a new article as an addition to this one, instead of trying to further revise the existing one.

In large part the problems with gunicorn and collectstatic stemmed from needing to run python manage.py collectstatic locally, and then to set DEBUG = False in settings.py.

One of our instructors, Yongwen Xu, gave this explanation:

6. “collectstatic” and “gunicorn” issues:
The static file handling is confusing both in Django and Heroku. One culprit is the flag “DEBUG” whose default value is set to True in settings.py. When the DEBUG flag is true, “manage.py runserver” will serve the static files automatically from the STATIC_ROOT and STATICFILES_DIRS settings, as defined in settings.py (“PROJECT_ROOT/site_media/static” and “PROJECT_ROOT/static”).
That is why students did not encounter this problem when using the “runserver” command locally and on Heroku.

The DEBUG setting is not honored by gunicorn, which is a server intended for production deployment (in which case the DEBUG flag should be set to false). So, when using gunicorn, the static files are NOT automatically served from STATIC_ROOT, thus, we don’t see any style in the app page, as in Jordan’s case.

One way we could improve the system is to add the following line to the urls.py, to explicitly tell the server to serve the static files from STATIC_ROOT:

urlpatterns += patterns('',
    (r'^' + settings.STATIC_URL[1:] + '(?P.*)$', 'django.views.static.serve',
         {'document_root': settings.STATIC_ROOT}), )

(Some text omitted…)

For local development, it’s best to always run “collectstatic”, so that the static files from both the app and the django admin are copied to the /site_media/static directory. Otherwise, even [if] the DEBUG flag is set to True, the style for the django admin interface will not be available because the admin static files are not in /site_media/static or /static dir.

On Heroku, the “collectstatic” is normally run by heroku automatically. see below on how to disable this:
https://devcenter.heroku.com/articles/django-assets

As a result of all this, for local development, “manage.py runserver” is easiest, while for production deployment, using “gunicorn” is recommended.

I will work on further revisions to the Part 2 post later, as time allows.

Lessons Learned from Deploying Django on Heroku, Part 2

NOTE: This article will set up a working Heroku application. However, there are issues with the way in which it handles collectstatic and gunicorn issues. For more information, see Corrections to Heroku Posts.

In a continuation of last week’s post, this post deploys a Github project on Heroku for a course assignment.

Heroku logo.

This blog is neither endorsed nor sponsored by Heroku.


SYSTEM DETAILS
Ubuntu 12.04.1 LTS (VirtualBox)

Prerequisites:

The prerequisites are the same as in my last article. Linux is recommended, since this guide uses its package manager. Additional steps may be required to install the equivalent software in Windows.

  • Heroku Toolbelt: $ sudo wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
  • Git: $ sudo apt-get install git
    As far as I know, Git should be installed as part of the Heroku toolbelt. If not, install it manually.
  • PostgreSQL: $ sudo apt-get install postgresql
  • Python-Dev: $ sudo apt-get install python-dev
  • Virtualenv: $ sudo apt-get install virtualenv
  • A Heroku account. All services provided by Heroku will be free so long as the 750-hour monthly limit is not exceeded. Examine Heroku pricing here.
  • Other required packages (psycopg2, dj-database-url) are installed during the tutorial.

This article covers the process of deploying a Github project with Twitter Bootstrap features to Heroku.

Step 1: Import The Github Project

Create a new folder, cd to it, and import the Github project:

mainuser@webdevbox:~/workspace/$ mkdir HelloHeroku && cd HelloHeroku
mainuser@webdevbox:~/workspace/HelloHeroku$ git init
Initialized empty Git repository in /home/mainuser/workspace/HelloHeroku/.git/
mainuser@webdevbox:~/workspace/HelloHeroku$ git remote add origin https://github.com/cammoore/responsive-heroku.git
mainuser@webdevbox:~/workspace/HelloHeroku$ git pull origin master
remote: Counting objects: 1975, done.
remote: Compressing objects: 100% (1899/1899), done.
remote: Total 1975 (delta 69), reused 1975 (delta 69)
Receiving objects: 100% (1975/1975), 22.76 MiB | 695 KiB/s, done.
Resolving deltas: 100% (69/69), done.
From https://github.com/cammoore/responsive-heroku
 * branch            master     -> FETCH_HEAD
mainuser@webdevbox:~/workspace/HelloHeroku$

Next, initialize and source the virtual environment. Use pip to install dependencies:

mainuser@webdevbox:~/workspace/HelloHeroku$ virtualenv venv --distribute
mainuser@webdevbox:~/workspace/HelloHeroku$ source venv/bin/activate
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ sudo pip install -r requirements.txt
// Output omitted

Step 2: Create a New Postgresql Database

If you did not configure the pg_hba.conf file as described in Step 1 of the tutorial in the previous article, do so now.

You will need to create a database for the new application:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ sudo su postgres
(venv)postgres@webdevbox:/home/mainuser/workspace/HelloHeroku$ psql
psql (9.1.8)
Type "help" for help.

postgres=# CREATE DATABASE helloheroku2_db OWNER testadmin;
CREATE DATABASE
postgres=# \q
postgres@webdevbox:/home/mainuser/workspace/HelloHeroku$ exit
exit
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$

Now open helloheroku/settings.py in your text editor of choice and edit the following lines at the end of the file:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] = dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

The file for the example project used here should already have the dj_database_url lines. The SECURE_PROXY_SSL_HEADER setting is used by Django to confirm that a request served from behind a proxy is being served as HTTPS.

(CORRECTION) Ignore the DATABASES section; dj_database_url.config() will import the values for DATABASES based on what is specified in the export command.

Now, export the database URL to your environment variables, and use set to check that it has been added. Note that you will need to re-export the environment variable every time you exit the terminal in which you started the virtual environment.

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ export DATABASE_URL=postgres://testadmin:testing@localhost/helloheroku_db
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ set | grep postgres
DATABASE_URL=postgres://testadmin:testing@localhost/helloheroku_db

Now add the changed file to Git:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ git add helloheroku/settings.py
// Output omitted
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ git commit -m "Configured settings.py to use postgresql database."
// output omitted

Step 3: Check or Configure Server Settings

The Github project used for this post is set to use Django’s own default web server by default, which runs with manage.py runserver:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ cat Procfile
web: python manage.py runserver 0.0.0.0:$PORT --noreload

NOTE: Though gunicorn was used in the last tutorial, it should not be used here.
Use pip to install the dependencies:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ pip install -r requirements.txt
// Output omitted

Step 4: Create Heroku Application

DISCLAIMER: If you plan to run the app locally, skip to Step 4A. For whatever reason, doing the python manage.py collectstatic manually in the local project before pushing it to Heroku may have caused my Heroku app to be unable to load the .less style files. I recommend that local and Heroku versions of this project be managed in separate directories to avoid this problem.

At this point, I was ready to create the Heroku application:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku create
Creating still-dawn-7589... done, stack is cedar
http://still-dawn-7589.herokuapp.com/ | git@heroku.com:still-dawn-7589.git
Git remote heroku added

Now, push the source code to Heroku:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ git push heroku master
// Output omitted

The Twitter Bootstrap static files are automatically detected by Heroku, which runs a python manage.py collectstatic on them:

// Output omitted
       Successfully installed Django argparse distribute dj-database-url psycopg2
       Cleaning up...
-----> Collecting static files
// Output omitted

It is important not to run python manage.py collectstatic locally before creating the Heroku project, or the .less files used by Twitter Bootstrap may not work correctly. (I am not sure that this was the cause of the problem. For a longer explanation see the “Conclusions” section.)

The page without CSS styles.

Above: If you run a “python manage.py collectstatic” command before pushing to Heroku, styles might not be applied to the page. (The hidden-journey-6676 app has been deleted.)


Thanks to Justin Lee for pointing this out.

Now, sync the Heroku database and set up a superuser for Django:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.2022
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log
Creating table polls_poll
Creating table polls_choice

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'u15523'): helloherokuadmin
E-mail address: admin@example.com
Password: hhadmin
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$

Now, start the app and open it in your browser:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku ps:scale web=1
Scaling web processes... done, now running 1
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ heroku open
Opening still-dawn-7589... done
(venv)mainuser@webdevbox:~/workspace/HelloHeroku$

If you see this error, continue:

Error message.

Above: Errors related to unresponsive .less scripts can usually be ignored. The page will eventually load.


The final page will look like this:
The Twitter Bootstrap-powered page.

Above: The final page.

Step 4A: (Optional) Run The App Locally

To run the example app locally, follow this post up until the end of Step 3. Then synchronize the database:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ python manage.py syncdb
// Output omitted. It is generally the same as the output from 
// the "heroku run python manage.py syncdb" command in Step 3.

Now run collectstatic:

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ python manage.py collectstatic

You have requested to collect static files at the destination
location as specified in your settings.

This will overwrite existing files!
Are you sure you want to do this?

Type 'yes' to continue, or 'no' to cancel: yes
// Output omitted...
Copying '/home/mainuser/workspace/HelloHeroku/static/js/prefixfree.min.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/Makefile'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/CONTRIBUTING.md'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/component.json'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/composer.json'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/package.json'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/README.md'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/LICENSE'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-popover.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-affix.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-transition.js'
Copying '/home/mainuser/workspace/HelloHeroku/static/twitter-bootstrap/js/bootstrap-scrollspy.js'
// Output omitted...
258 static files copied.

This will copy all the .less, .js, and other Twitter Bootstrap files to the site_media directory in the HelloHeroku directory.

Afterwards, start the local Django server with this command.

(venv)mainuser@webdevbox:~/workspace/HelloHeroku$ python manage.py runserver

Next, go to http://127.0.0.1:8000/ to view your page.

Conclusions

Deploying Github projects on Heroku seems to work fairly well as long as the default Django server is used. I am not sure whether my deployment problems were caused by the collectstatic command or by using gunicorn as the server; however, even when the collectstatic command was handled by Heroku, using gunicorn as the server caused the same problem. This suggests that the problem stems from using gunicorn as the web server. Otherwise, Heroku seems to be well-integrated with Git and allows you to manage your projects much as you would manage GitHub or a regular remote repository.

This post was last revised on February 27, 2013. See Corrections to Heroku Posts for more details.

Lessons Learned From Deploying Django on Heroku

This article describes my experience with the Heroku/Django tutorial at https://devcenter.heroku.com/articles/django. It largely follows the original tutorial, but points out the areas where the current tutorial seemed to be unclear on what was required.

Heroku logo.

This blog is neither endorsed nor sponsored by Heroku.


SYSTEM DETAILS
Ubuntu 12.04.1 LTS (VirtualBox)

Prerequisites:

I recommend that anyone following the tutorial do so in Linux if possible. Versions of many of these packages exist for Windows, but the setup process will be more complicated.

*NOTE: This is not mentioned in the original tutorial, but an error occurs without it. See the rest of this post.

Heroku is a cloud application platform: the Heroku servers manage web applications which are isolated into Unix-like environments called dynos, which execute a single command on a packaged version of the application called a slug. The slug is compiled from a Git repository of the files that make up the web applications.

Step 1: Configure PostGreSQL

The tutorial says that you need to have PostgreSQL installed “to test locally.” They do not explain how to set it up properly. To set up a database in PostgreSQL, you need to first make sure that it accepts local connections. Open the file pg_hba.conf in the text editor (vi, nano, Emacs, etc.) of your choice.

mainuser@webdevbox:~/$ sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Look for a line similar to the below image:

pg_hba.conf

Above: The pg_hba.conf file open in nano.


The last parameter in the line will say something like “peer” or “md5.” Change it to “trust”: we want it to allow all local connections. Now, restart postgresql:

mainuser@webdevbox:~/$ sudo service postgresql restart
 * Restarting PostgreSQL 9.1 database server                             [ OK ] 

You should now be able to switch to the postgres user and access a postgresql prompt (postgres=#):

mainuser@webdevbox:~/$ sudo su postgres
postgres@webdevbox:~/$ psql
postgres=#

If you like, you can set a service password here. By default, the postgres account has no password on UNIX / Linux. For a brief explanation, see the section on “Service Password” in this post by Dave Page.

postgres=# \password postgres
Enter new password:

Note that this is not the superuser password; without further configuration, you will not be able to use su / sudo / other switching commands as the postgres user. Changing/setting the superuser password for postgres is beyond the scope of this post.

Now create a user and create a table with that user as an administrator, then exit postgres and the postgres user:

postgres=# CREATE USER testadmin WITH CREATEDB PASSWORD 'testing';
CREATE ROLE
postgres=# CREATE DATABASE django_testdb OWNER testadmin;
CREATE DATABASE
postgres=# \q
postgres@webdevbox:~/$ exit
mainuser@webdevbox:~/$

Remember the new user, password, and database name; you will need them later.

Step 2: Set Up a Local Django App

As the tutorial says, run these commands to install Django and some other dependencies:

mainuser@webdevbox:~/$ mkdir hellodjango && cd hellodjango
mainuser@webdevbox:~/hellodjango$ virtualenv venv --distribute
New python executable in venv/bin/python
Installing distribute...............done.
Installing pip...............done.
mainuser@webdevboxL~/hellodjango$ source venv/bin/activate
(venv)mainuser@webdevbox:~/hellodjango$ pip install Django psycopg2 dj-database-url

I encountered a bug while psycopg2 was being compiled. (If you installed python-dev, you probably will not see this.)

// Some output omitted
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.6 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090108 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.1/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.7/psycopg/psycopgmodule.o -Wdeclaration-after-statement

In file included from psycopg/psycopgmodule.c:27:0:

./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

According to a psycopg2 FAQ, this error is caused by not having the python-dev package installed, so I installed it:

(venv)mainuser@webdevbox:~/hellodjango$ sudo apt-get install python-dev

The command should now work correctly. Continuing to follow the tutorial, you should run these commands:

(venv)mainuser@webdevbox:~/hellodjango$ pip install Django psycopg2 dj-database-url
// Output omitted
(venv)mainuser@webdevbox:~/hellodjango$ django-admin.py startproject hellodjango .
(venv)mainuser@webdevbox:~/hellodjango$ python manage.py runserver
0 errors found
Django version 1.4, using settings 'hellodjango.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
^C
(venv)mainuser@webdevbox:~/hellodjango$ pip freeze > requirements.txt
(venv)mainuser@webdevbox:~/hellodjango$ cat requirements.txt
Django==1.4
psycopg2==2.4.5
dj-database-url==0.2.0

(CORRECTION): Contrary to both previous versions of this post, you can run pip freeze at any time. This will change requirements.txt by adding the new dependencies that have been installed.

If you edit requirements.txt manually, run pip install -r requirements.txt to install any new dependencies:

(venv)mainuser@webdevbox:~/hellodjango$ pip install -r requirements.txt

The contents of your requirements.txt file may look different from the example in the tutorial. As long as the lines in the tutorial are part of the requirements.txt file, everything should be fine.

Next, the tutorial states “Next, configure the application for the Heroku environment, including Heroku’s Postgres database. The dj-database-url module will parse the values of the DATABASE_URL environment variable and convert them to something Django can understand.” Add these lines to the end of helloheroku/settings.py as specified in the tutorial:

# Parse database configuration from $DATABASE_URL
import dj_database_url
DATABASES['default'] =  dj_database_url.config()

# Honor the 'X-Forwarded-Proto' header for request.is_secure()
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

(CORRECTION) An earlier version of this post stated that values in settings.py’s DATABASES section needed to be changed. This was unnecessary. The DATABASE_URL environment variable must be set with Linux’s export command. Thanks to Zach Tomaszewski for pointing this out.

(venv)mainuser@webdevbox:~/hellodjango$ export DATABASE_URL=postgres://testadmin:testing@localhost/django_db

Here, testadmin is a postgresql user, testing is the password for that user, and django_db is the name of the local database. On the local machine, dj_database_url.config() reads the value of DATABASE_URL specified by export and returns its value, which becomes the value of DATABASES. On Heroku, dj_database_url.config() will read the value of DATABASES from Heroku’s environment variables instead.

Next, the Procfile needs to be created. As far as I know, it goes in the top-level directory of the project (hellodjango, where manage.py is).

(venv)mainuser@webdevbox:~/hellodjango$ nano Procfile
web: python manage.py runserver 0.0.0.0:$PORT --noreload

As suggested by the tutorial, I used Github’s Python .gitignore file as the project .gitignore file. Though the line *.py[cod] probably covers *.pyc, I added lines for venv and *.pyc anyway, to be safe. Then I initialized the Git repository:

(venv)mainuser@webdevbox:~/hellodjango$ git init
Initialized empty Git repository in /home/mainuser/hellodjango/.git/
(venv)mainuser@webdevbox:~/hellodjango$ git add .
(venv)mainuser@webdevbox:~/hellodjango$ git commit -m "My Django app."
[master (root-commit) 6c2765c] My Django app.
 9 files changed, 262 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Procfile
 create mode 100644 hellodjango/settings.py
 create mode 100644 hellodjango/urls.py
 create mode 100644 hellodjango/wsgi.py
 create mode 100644 manage.py
 create mode 100644 requirements.txt
 create mode 120000 venv/include/python2.7
 create mode 120000 venv/local/include

Step 3: Deploying The App to Heroku

When you first installed the Heroku toolbelt, you should have been asked to generate SSH keys:

(venv)mainuser@webdevbox:~/$ heroku login
Enter your Heroku credentials.
Email: admin@example.com
Password: same-as-heroku-account
Could not find an existing public key.
Would you like to generate one? [Yn] Y
Generating new SSH public key.
Uploading ssh public key /Users/mainuser/.ssh/id_rsa.pub.

Otherwise, follow the developers’ example to generate new keys with ssh-keygen -t rsa. Once you have a public SSH key (check in /Users/<username>/.ssh), you should be able to create a Heroku app.

(venv)mainuser@webdevbox:~/$ heroku create
Creating still-badlands-2085... done, stack is cedar
http://still-badlands-2085.herokuapp.com/ | git@heroku.com:still-badlands-2085.git
Git remote heroku added

Now push your code to the Heroku repository:

(venv)mainuser@webdevbox:~/$ git push heroku master
Counting objects: 16, done.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (16/16), 4.34 KiB, done.
Total 16 (delta 0), reused 0 (delta 0)
-----> Python app detected
-----> No runtime.txt provided; assuming python-2.7.3.
-----> Preparing Python runtime (python-2.7.3)
-----> Installing Distribute (0.6.34)
-----> Installing Pip (1.2.1)
-----> Installing dependencies using Pip (1.2.1)
       Downloading/unpacking Django==1.4.4 (from -r requirements.txt (line 1))
         Running setup.py egg_info for package Django
           
       Downloading/unpacking argparse==1.2.1 (from -r requirements.txt (line 2))
         Running setup.py egg_info for package argparse
           
           no previously-included directories found matching 'doc/_build'
           no previously-included directories found matching 'env24'
           no previously-included directories found matching 'env25'
           no previously-included directories found matching 'env26'
           no previously-included directories found matching 'env27'
       Downloading/unpacking distribute==0.6.24 (from -r requirements.txt (line 3))
         Running setup.py egg_info for package distribute
           
           warning: no files found matching 'Makefile' under directory 'docs'
           warning: no files found matching 'indexsidebar.html' under directory 'docs'
       Downloading/unpacking dj-database-url==0.2.1 (from -r requirements.txt (line 4))
         Downloading dj-database-url-0.2.1.tar.gz
         Running setup.py egg_info for package dj-database-url
           
       Downloading/unpacking psycopg2==2.4.6 (from -r requirements.txt (line 5))
         Running setup.py egg_info for package psycopg2
           
           no previously-included directories found matching 'doc/src/_build'
       Installing collected packages: Django, argparse, distribute, dj-database-url, psycopg2
         Running setup.py install for Django
           changing mode of build/scripts-2.7/django-admin.py from 600 to 755
           
           changing mode of /app/.heroku/python/bin/django-admin.py to 755
         Running setup.py install for argparse
           
           no previously-included directories found matching 'doc/_build'
           no previously-included directories found matching 'env24'
           no previously-included directories found matching 'env25'
           no previously-included directories found matching 'env26'
           no previously-included directories found matching 'env27'
         Found existing installation: distribute 0.6.34
           Uninstalling distribute:
             Successfully uninstalled distribute
         Running setup.py install for distribute
           Before install bootstrap.
           Scanning installed packages
           Setuptools installation detected at /app/.heroku/python/lib/python2.7/site-packages
           Non-egg installation
           Removing elements out of the way...
           Already patched.
           /app/.heroku/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info already patched.
           
           warning: no files found matching 'Makefile' under directory 'docs'
           warning: no files found matching 'indexsidebar.html' under directory 'docs'
           Installing easy_install script to /app/.heroku/python/bin
           Installing easy_install-2.7 script to /app/.heroku/python/bin
           After install bootstrap.
           /app/.heroku/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info already exists
         Running setup.py install for dj-database-url
           
         Running setup.py install for psycopg2
           building 'psycopg2._psycopg' extension

// gcc output omitted           

           no previously-included directories found matching 'doc/src/_build'
       Successfully installed Django argparse distribute dj-database-url psycopg2
       Cleaning up...
-----> Collecting static files
       0 static files copied.

-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size: 29.2MB
-----> Launching... done, v6
       http://still-badlands-2085.herokuapp.com deployed to Heroku

To git@heroku.com:still-badlands-2085.git
 * [new branch]      master -> master
(venv)mainuser@webdevbox:~/hellodjango$ 

If you sign in to your Heroku account in a browser and navigate to My Apps, then to your-app-name-####, your app should be visible:

Heroku dashboard.

Above: The Heroku dashboard for app still-badlands-2085. If the “web” box is checked and the dyno count is 1, your app may be running already.


Now, you should be able to start the app with heroku ps:scale web=#, where # is the number of web dynos. Free accounts can have more than one app, but if more than one dyno is running for an extended period, you will exceed your 750 free hours (750 hours = 31.25 days) per month. Use heroku ps to view the app state, heroku open to view the app in your default browser, and heroku logs to view recent log information.

You can also start the app from your browser by checking the “web” checkbox, then clicking “Apply Changes.”

(venv)mainuser@webdevbox:~/hellodjango$ heroku ps:scale web=1
Scaling web processes... done, now running 1
(venv)mainuser@webdevbox:~/hellodjango$ heroku ps
=== web: `python manage.py runserver 0.0.0.0:$PORT --noreload`
web.1: up 2013/02/19 18:07:46 (~ 6m ago)

(venv)mainuser@webdevbox:~/hellodjango$ heroku open
Opening still-badlands-2085... done

The Django example app.

Above: The still-badlands-2085 app in Google Chrome.


Now you should be able to sync your Heroku database.

(venv)mainuser@webdevbox:~/hellodjango$ heroku run python manage.py syncdb
Running `python manage.py syncdb` attached to terminal... up, run.6600
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_groups
Creating table auth_user_user_permissions
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'u55828'): someadmin
E-mail address: admin@example.com
Password: somethingbetter
Password (again): somethingbetter
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
(venv)mainuser@webdevbox:~/hellodjango$

If, like me, you had at first forgotten to configure PostgreSQL and do an export, you will see errors like this if you try to run the server locally:

(venv)mainuser@webdevbox:~/hellodjango$ python manage.py syncdb
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 57, in handle_noargs
    cursor = connection.cursor()
  File "/home/mainuser/hellodjango/venv/local/lib/python2.7/site-packages/django/db/backends/dummy/base.py", line 15, in complain
    raise ImproperlyConfigured("settings.DATABASES is improperly configured. "
django.core.exceptions.ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.

If this happens, set up the database as described in Step 1 and edit settings.py as described in Step 2, then try again.

Step 4: Change The Web Server To Gunicorn

If you’ve been following the Heroku tutorial so far, there’s one last step: switching the webserver to gunicorn.

Gunicorn logo.

Gunicorn is a Python port of Ruby’s WSGI HTTP server, Unicorn.

  1. In settings.py, change DEBUG to False. You should always do this before deploying a Django application with gunicorn, because gunicorn does not respect the DEBUG flag. In general, always change DEBUG to False before deploying any production Django application.
    DEBUG = False
    
  2. Add the line gunicorn==0.16.1 to your requirements.txt
  3. Change your Procfile’s web: line to web: gunicorn hellodjango.wsgi
  4. Run pip to update your dependencies:
    (venv)mainuser@webdevbox:~/hellodjango$ pip install -r requirements.txt
    
  5. Run foreman start to run the application locally. Use Ctrl-C to quit the server when you’re done.
    (venv)mainuser@webdevbox:~/hellodjango$ foreman start
    18:36:03 web.1  | started with pid 3420
    18:36:04 web.1  | 2013-02-19 18:36:04 [3423] [INFO] Starting gunicorn 0.16.1
    18:36:04 web.1  | 2013-02-19 18:36:04 [3423] [INFO] Listening at: http://0.0.0.0:5000 (3423)
    18:36:04 web.1  | 2013-02-19 18:36:04 [3423] [INFO] Using worker: sync
    18:36:04 web.1  | 2013-02-19 18:36:04 [3426] [INFO] Booting worker with pid: 3426
    ^CSIGINT received
    18:41:06 web.1  | 2013-02-19 18:41:06 [3426] [INFO] Worker exiting (pid: 3426)
    18:41:06 system | sending SIGTERM to all processes
    18:41:06 web.1  | 2013-02-19 18:41:06 [3423] [INFO] Handling signal: int
    SIGTERM received
    18:41:06 web.1  | terminated by SIGTERM
    
  6. Push your changes to Heroku:
    (venv)mainuser@webdevbox:~/hellodjango$ git add Procfile requirements.txt hellodjango/settings.py
    (venv)mainuser@webdevbox:~/hellodjango$ git commit -m "Changed server to gunicorn."
    (venv)mainuser@webdevbox:~/hellodjango$ git push heroku master
    Counting objects: 7, done.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (4/4), 364 bytes, done.
    Total 4 (delta 2), reused 0 (delta 0)
    -----> Python app detected
    -----> No runtime.txt provided; assuming python-2.7.3.
    -----> Using Python runtime (python-2.7.3)
    -----> Installing dependencies using Pip (1.2.1)
           Downloading/unpacking gunicorn==0.16.1 (from -r requirements.txt (line 7))
             Running setup.py egg_info for package gunicorn
               
           Installing collected packages: gunicorn
             Running setup.py install for gunicorn
               
               Installing gunicorn_paster script to /app/.heroku/python/bin
               Installing gunicorn script to /app/.heroku/python/bin
               Installing gunicorn_django script to /app/.heroku/python/bin
           Successfully installed gunicorn
           Cleaning up...
    -----> Collecting static files
           0 static files copied.
    
    -----> Discovering process types
           Procfile declares types -> web
    -----> Compiled slug size: 29.5MB
    -----> Launching... done, v7
           http://still-badlands-2085.herokuapp.com deployed to Heroku
    
    To git@heroku.com:still-badlands-2085.git
       6c2765c..697eb0f  master -> master
    mainuser@webdevbox:~/hellodjango$ 
    
  7. Run heroku logs. You should see some reference to gunicorn in the recent log entries.

Step 5: Stopping Your App

If you’re doing this for a course project like I did, or doing some testing later, you’ll want to leave the app running. When you’re done, shut off the process by scaling back the number of dynos to 0:

(venv)mainuser@webdevbox:~/hellodjango$ heroku ps:scale web=0
Scaling web processes... done, now running 0
(venv)mainuser@webdevbox:~/hellodjango$ 

Though I may have missed something, I could not find any advice on the official site on how to stop an app completely outside of the web interface (where you can uncheck the web) checkbox to stop the dyno). This tip comes from Stack Overflow.

Overall Thoughts

In my opinion, the biggest weakness of the tutorial is that it does not explain how to set up the Postgresql database or link to a tutorial which does. Maybe the developers assumed that anyone who is experienced enough in web development to be using Heroku would have used Postgresql before. Maybe they just expected people to Read The Manual. Otherwise, I found Heroku to be fairly easy to deploy and use.

This post was last revised on February 27, 2013. See Corrections to Heroku Posts for more details.