Building Common Lisp FastCGI Applications for Dreamhost Shared Hosting

These are the step-by-step instructions to build a Common Lisp application with FastCGI on a Dreamhost shared hosting environment. Updated August, 2016.

Since you're reading this, I assume you have at least some knowledge of Common Lisp and a little bit of shell interaction. I also assume you're running Linux on your development system, but if you're a BSD/Mac user, you'll just have to figure out what virtual machine tools you want to use to start up your Linux virtual machine, and I imagine you've had to do that before.

Get a Compiler[edit]

Dreamhost doesn't (yet?) provide a Common Lisp compiler you can use, so you're going to have to build one yourself. In order to do that nicely, you should put together a virtual machine of some kind (or possibly a DreamCompute instance — more on that later) to do your compiling.

1. . Put together your compilation box.

Note DHUSER matches the Dreamhost user account under which I deploy it, just in case any paths end up hard-coded into the corefile or something:

Code: FROM ubuntu:12.04 MAINTAINER Bruce-Robert Fenn Pocock <brfennpocock@star-hope.org> RUN sh -c 'dhclient eth0 &' RUN apt-get update RUN apt-get install -y make sbcl curl gnupg unzip RUN cat /etc/passwd RUN echo DHUSER:x:1000:0::/home/DHUSER:/bin/bash >> /etc/passwd RUN cp -a /etc/skel/ /home/DHUSER RUN chown -R DHUSER /home/DHUSER RUN echo 'DHUSER:$6$norealpassword:16615:0:9999:7:::' >> /etc/shadow RUN su - DHUSER -c 'curl -OL https://github.com/sbcl/sbcl/archive/master.zip' RUN su - DHUSER -c 'unzip master.zip' COPY map-patch.patch /home/DHUSER/sbcl/map-patch.patch RUN apt-get install -y gcc patch RUN su - DHUSER -c 'cd sbcl-master; ls src/runtime/linux-os.c ; patch -p1 < ~/sbcl/map-patch.patch' RUN su - DHUSER -c 'cd sbcl-master; echo "\"1.3.0x\"" > version.lisp-expr' RUN apt-get install -y time RUN su - DHUSER -c 'cd sbcl-master; ./make.sh --prefix=/home/DHUSER/sbcl'

1. RUN apt-get install -y ed
2. RUN su - DHUSER -c 'cd sbcl-master/tests; ./run-tests.sh'

RUN chown DHUSER -R /home/DHUSER/sbcl RUN su - DHUSER -c 'cd sbcl-master; ./install.sh --prefix=/home/DHUSER/sbcl' RUN apt-get remove -y sbcl RUN su - DHUSER -c 'curl -LO https://beta.quicklisp.org/quicklisp.lisp' COPY quicklisp.lisp.asc /home/DHUSER/quicklisp.lisp.asc

1. RUN su - DHUSER -c 'gpg --verify quicklisp.lisp.asc quicklisp.lisp'

RUN su - DHUSER -c "./sbcl/bin/sbcl --load ./quicklisp.lisp --eval '(quicklisp-quickstart:install)' \

   --eval '(ql:add-to-init-file)' --eval '(quit)'"

RUN su - DHUSER -c "./sbcl/bin/sbcl --load ~/quicklisp/setup.lisp --eval '(quicklisp:quickload :caveman2)' --eval '(quit)'"

Clozure CL[edit]

1. . Visit http://ccl.clozure.com/download.html and figure out what the current stable Release version is. It is usually the first line of the first paragraph.

1. . Connect to your compilation box.

1. . Run:

       svn checkout http://svn.clozure.com/publicsvn/openmcl/release/1.11/linuxx86/ccl ~/ccl

 — replace `1.11` with a newer version if this document has gotten out-of-date.

1. . Follow the instructions on Clozure.com to rebuild the Lisp kernel. At the bottom section of http://ccl.clozure.com/install.html — At the moment, this is:

        cd lisp-kernel/linuxx64 && make clean && make

1. . Put this in ~/bin/ccl:

    #!/bin/sh
    exec $HOME/ccl/lx86cl64 "$@"

To compile, I use Xach's BuildApp and Quicklisp, so I went through the three or so steps to set up Quicklisp (quicklisp.org) and installed BuildApp through it, putting the binary in ~/bin

To deploy, I do a git checkout of my app code and run (“nice”) BuildApp over SSH, then move the file into the server root.

Steel Bank Common Lisp[edit]

Personally: I just happen to like SBCL a little more, so I built a Docker to compile SBCL in, with the following lame Dockerfile.

Yes, that's not super-efficient, running apt-get multiple times, … it grew on me. The run-tests.sh fails, BTW.

The associated patch for address-randomization corrections is: Code:

1. Patch mmap for Dreamhost's address-randomization code

--- master/src/runtime/linux-os.c +++ patched/src/runtime/linux-os.c @@ -337,6 +337,9 @@ is_validate(os_vm_address_t addr, os_vm_size_t len)

        addr=under_2gb_free_pointer;
    }

1. endif

+ + if (addr) flags |= MAP_FIXED; +

    actual = mmap(addr, len, OS_VM_PROT_ALL, flags, -1, 0);
    if (actual == MAP_FAILED) {
        perror("mmap");

That's against the master branch of SBCL as of summer 2016; I haven't probed the #sbcl cabal about getting it checked in (or why it isn't already) yet.

So, the above yields inside the Docker a working SBCL at /home/DHUSER/sbcl that can be used to compile. SFTP those up to Dreamhost as well.

Next bit: CLACK main function and .htaccess. This took some facepalming trial-and-error, sadly, and is probably not the most efficient either, but:

Code: Options ExecCGI SymLinksIfOwnerMatch AddHandler fastcgi-script .cgi

AcceptPathInfo On RewriteEngine On

RewriteCond %{HTTPS} !=on RewriteRule (.*) https://%7Cyour-domain-name%7C%{REQUEST_URI} [R=301,L]

RewriteRule ^|prefix|/(.*)$ |executable|.cgi/%1 [QSA,L] RewriteRule ^[Pp][Ll][Aa][Yy].* |executable|.cgi/login [QSA,L]

Header always set X-Frame-Options SAMEORIGIN

Here |your-domain-name| is obvious and that bit enforces TLS-only. Drop those two lines if you don't care.

The next two lines map |prefix| to |executable|.cgi and pass the path-info so (DEFROUTE) will work. (Or whatever Ningle routing you use.) And remap a “pretty” URI pattern to a specific point.

That gets you into the FCGI executable.

Here's my top-level Makefile:

Code: all: |your-app-name|.cgi |your-app-name|.cgi: |your-app-name|.asd $(shell find . -name \*.lisp -and -not -path '**/lib/**')

   PATH=~/bin:${PATH} ~/bin/buildapp --output |your-app-name|.cgi.new \
       --load ~/quicklisp/setup.lisp \
       --asdf-path . \
       --eval '(ql:quickload :|your-app-name|)' \
       --load-system |your-app-name| \
       --entry |your-app-name|:fastcgi-entry
   mv --backup=t |your-app-name|.cgi.new |your-app-name|.cgi

src/lib/jscl.js: $(shell find src/lib/jscl -name \*.lisp -or -name \*.lisp)

   cd src/lib/jscl; ./make.sh

Pretty typical stuff, I suppose. Note particularly that BuildApp cannot overwrite an executable while it's running, so building a new copy then moving it into place is required if you ever need to do an in-production patch job; BuildApp won't do that bit for you. (The --backup=t means you'll end up with myapp.cgi.~1~ for ever-increasing values of ~1~ over time, which is helpful if you mess up but you'll want to clean up behind yourself periodically.)

Then: Clack entry point #'FASTCGI-ENTRY: (I have this in src/main.lisp right after Caveman's generated #'START for Hunchentoot)

Code: (defun fastcgi-entry (argv)

 (format t "Content-Type: text/plain~c~c~c~c
 
 Starting FastCGI is half-working if you see this. But, only half.~%"
 #\Return #\Linefeed #\Return #\Linefeed)
 (format *error-output* "~%FastCGI started with ARGV=~s~%" argv)
 (let ((ql:*quickload-explain* nil) (ql:*quickload-verbose* nil))
   (clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0)))

The #'FORMAT calls are hopefully irrelevant now, but I had needed them for debugging this.

Code: (defun fastcgi-entry (argv)

 (declare (ignore argv))
 (clackup *appfile-path* :server :fcgi :port nil :use-thread nil :fd 0))

… that should really do it. NB I “had to” pass the :fd 0 explicitly, even though it seems like it would be the default. (FD 0 = *STANDARD-INPUT*)

Hope that helps!

   (08-03-2016 08:59 AM)brpocock Wrote:  
   Code:
   RewriteRule (.*) https://%7Cyour-domain-name%7C%{REQUEST_URI} [R=301,L]

PS. The pipe chars in the examples are metasyntactic. So eg Code: RewriteRule (.*) https://example.com%{REQUEST_URI} [R=301,L]