We’ve been making big improvements to developer workflow at the day job, from going test-driven, to using a Jenkins continuous integration server, to ACTIVATING CHUCK NORRIS. I just went through the exercise of getting our virtualenv stuff working with Jenkins and thought I’d post my findings.

First, this post from the horse’s mouth got me most of the way there. There were a couple things that I did differently.

EDIT: I had to take one more step to accomodate our PATH requirements and have modified the instructions below appropriately.

First, I had to use the Environment Injector plugin instead of the SetEnv Plugin mentioned in the above jenkins-ci.org link. The SetEnv plugin has apparently been deprecated. Fortunately, the Environment Injector plugin takes care of everything SetEnv did, and then some. Because it is so much more flexible, it wasn’t quite clear where to put my env vars in the job definition. Use the “Properties Content” field with a basic KEY=VALUE statement. In my case, it was

PATH=.env/bin:$PATH

…except that this didn’t work in our final context, which requires updating and running code out of directories that aren’t in the jenkins workspace. Since that PATH addition line above makes the .env dir RELATIVE, this won’t work when our working directory is outside the workspace. Jenkins automatically sets up the $WORKSPACE env var to point at the directory containing the workspace, but if you try to inject a variable like

PATH=$WORKSPACE/.env/bin:$PATH

…prior to any build steps, it seems to fail, perhaps because $WORKSPACE is undefined before any build steps. I ended up removing the PATH setup from the main Properties Content field and adding an Inject Environment Variables build step BEFORE my Execute Shell step with all the ‘meat’ of the test in it. Said Inject Environment Variables build step now contains the aforementioned PATH setting using the $WORKSPACE variable, which seems to work nicely.

Second, I had a specific environment that I wanted, and I already had a requirements file written! All I had to do was to clone it from the bitbucket repository and feed it to pip. Thus, I enhanced the little if/fi block from the above article as follows (some paths and credentials changed for public consumption):

# This if/fi block creates the virtualenv if it doesn't exist already
# Blowing away the workspace will cause this to start over again
if [ -d ".env" ]; then
echo "**> virtualenv exists"
else
echo "**> creating virtualenv"
virtualenv .env
echo "**> cloning nftools"
hg clone https://****:****@bitbucket.org/foo/bar
echo "**> installing requirements for 2.1"
pip install -r bar/requirements-2.1.txt
fi

That block went at the top of my first Execute Shell Build Step. After that I just did a quick test of the virtualenv before handing it back to the developer:

# This is a test of the virtualenv
python -c 'import zmq; print dir(zmq)'

And in the output, I got exactly what I wanted to see. First run of the job, I saw it build out the virtualenv. Second run, I just saw the test output:

[snip]
+ '[' -d .env ']'
+ echo '**> virtualenv exists'
**> virtualenv exists
+ python -c 'import zmq; print dir(zmq)'
['AFFINITY', 'BACKLOG', 'Context', 'DEALER', 'DONTWAIT', 'DOWNSTREAM', 'EADDRINUSE', 'EADDRNOTAVAIL', 'EAGAIN', 'ECONNREFUSED', 'EFAULT', 'EFSM', 'EINPROGRESS', 'EINVAL', 'EMTHREAD', 'ENETDOWN', 'ENOBUFS', 'ENOCOMPATPROTO', 'ENODEV', 'ENOMEM', 'ENOTSOCK', 'ENOTSUP', 'EPROTONOSUPPORT', 'ETERM', 'EVENTS', 'FD', 'FORWARDER', 'HWM', 'IDENTITY', 'LINGER', 'MCAST_LOOP', 'Message', 'MessageTracker', 'NOBLOCK', 'NotDone', 'PAIR', 'POLLERR', 'POLLIN', 'POLLOUT', 'PUB', 'PULL', 'PUSH', 'Poller', 'QUEUE', 'RATE', 'RCVBUF', 'RCVMORE', 'RECONNECT_IVL', 'RECONNECT_IVL_MAX', 'RECOVERY_IVL', 'RECOVERY_IVL_MSEC', 'REP', 'REQ', 'ROUTER', 'SNDBUF', 'SNDMORE', 'STREAMER', 'SUB', 'SUBSCRIBE', 'SWAP', 'Socket', 'Stopwatch', 'TYPE', 'UNSUBSCRIBE', 'UPSTREAM', 'XPUB', 'XREP', 'XREQ', 'XSUB', 'ZMQBaseError', 'ZMQBindError', 'ZMQError', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__revision__', '__version__', 'bytes_sockopts', 'core', 'device', 'devices', 'get_includes', 'initthreads', 'int64_sockopts', 'int_sockopts', 'pyzmq_version', 'pyzmq_version_info', 'select', 'strerror', 'sys', 'utils', 'zmq_version', 'zmq_version_info']
[snip]

I hope that helps anyone else casting about for a solution here!