Proxy golang web server with relayd on OpenBSD 5.7

If you are new to OpenBSD, you may wonder why the httpd daemon that in the base install does not provide the ability to proxy web requests. The reason is that this capability is already provided in base by another daemon called relayd.

This post gives you a quick recipe for how to setup relayd to proxy http requests to a golang web server running as a non-priviledged user. Note that this barely scratches the surface of what relayd can do; for more info, you can read

• Basic relayd Config (cont)
• Recent work in OpenBSD relayd
• Relayd: a load balancer for OpenBSD - SlideShare
• relayd/etc/relayd.conf
• relayd(8)
• relay.conf(5)

Steps:

1. Add a new user, accepting defaults.
2. As that user, start up golang web server, listening on port 8080.
   

   $ cat > srv.go
   package main
   
   import (
           "fmt"
           "log"
           "net/http"
           "time"
   )
   
   func withalog(h http.Handler) http.Handler {
           f := func(w http.ResponseWriter, r *http.Request) {
                   fmt.Printf(
                           "%s - - [%s] \"%s %s %s\"\n",
                           r.Header["X-Forwarded-For"],
                           //r.RemoteAddr,
                           time.Now().Format("02/Jan/2006 03:04:05 EST"),
                           r.Method,
                           r.URL,
                           r.Proto)
                   h.ServeHTTP(w, r)
           }
           return http.HandlerFunc(f)
   }
   
   func main() {
           handler := http.FileServer(http.Dir("/usr/share/doc"))
           log.Fatal(http.ListenAndServe(":8080", withalog(handler)))
   }
   ^D
   $ go build srv.go
   $ ./srv &
   $ curl http://127.0.0.1:8080/../..
   [] - - [04/Jul/2015 10:20:21 EST] "GET / HTTP/1.1"
   <pre>
   <a href="mg/">mg/
   </pre>
   $
   

3. Configure relayd to forward 80 to 8080
   

   # cat > /etc/relayd.conf
   http protocol "littleproto" {
           return error
           match request header append "X-Forwarded-For" \
               value "$REMOTE_ADDR"
   }
   
   relay myproxy {
           listen on 192.168.30.6 port 80
           protocol "littleproto"
           forward to 127.0.0.1 port 8080
   }
   ^D
   # relayd -n
   configuration OK
   #
   

4. Set it to start on reboot.
   

   # echo relayd_flags="" >> /etc/rc.conf.local
   #
   

5. Start it and test.

   # relayd
   # curl http://192.168.30.6
   <pre>
   <a href="mg/">mg/
   </pre>
   

Questions

What does return error do in http protocol?

Tells relayd to return an error pages. If you don't specify the style option (see man page), this is the default styling:

If I just want a http reverse proxy, do I need to configure PF?

No.

What's the differences between relay httpproxy {... and redirect www {... , both of which are in /etc/examples/relayd.conf.

Not 100% sure. From reading relay.conf(5), it says that a redirection does stateful forwarding, and a relay is for general purpose TCP proxy.

Your own private git remote

I have some documents (writing, financial data, some code) that I want to keep private. This describes how to setup a git remote repository on your home network, and to back it up to the cloud (daily) with tarsnap.

Definitions:

office

The host that holds your remote git repo.

lap

The computer where you do your programming.

Here are the steps:

1. Create a git-private user on office. On OpenBSD, that is
   adduser git-private
2. Create ssh key-pair on lap
   

   mark@lap:~$ ssh-keygen
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/mark/.ssh/id_rsa): /home/mark/.ssh/id_rsa_git-private
   Enter passphrase (empty for no passphrase): 
   Enter same passphrase again: 
   Your identification has been saved in /home/mark/.ssh/id_rsa_git-private.
   Your public key has been saved in /home/mark/.ssh/id_rsa_git-private.pub.
   The key fingerprint is:
   cc:c7:c8:ab:36:9b:1a:d1:61:d1:2f:9a:06:4b:97:16 mark@lap
   The key's randomart image is:
   +--[ RSA 2048]----+
   |      ..         |
   |      E..        |
   |      oo .       |
   |    oo+=.o.      |
   |   ..=.oS.o      |
   |    ..+  o       |
   |    ..  .        |
   |     .oo         |
   |    .o+o         |
   +-----------------+
   mark@lap:~$
   

3. Authorize key.
   scp ~/.ssh/id_rsa_git-private.pub git-private@office:~/.ssh/authorized_keys
4. On lap, associate private key with office
   

   cat >> ~/.ssh/config
   Host            office
    Hostname office
    IdentityFile    ~/.ssh/id_rsa_git-private
    User            git-private
   ^D
   

5. Set up bare remote git repo on office.
   

   mark@lap:~$ ssh git-private@office
   git-private@office:~$ git init --bare myrepo.git
   

6. On lap, add remove repo as remote.
   

   mark@lap:~$ git remove -v
   mark@lap:~$ git remote add origin git-private@prod:/home/git-private/myrepo.git
   mark@lap:~$ git push -u origin all
   

7. To securely backup encrypted versions of your remove, see my Recipe for setting up backups with tarsnap (OpenBSD 5.7).

Recipe for setting up backups with tarsnap (OpenBSD 5.7)

Tarsnap is a wicked smart backup service. (The author started studying math at university at age 13!) It only saves the deltas since your last backup, which means you can always do a full backup but you use much less space. Tarsnap (somehow) stores enough info that it makes every backup act like a full backup; for example, even if you delete all previous backups the last one will still be a full.

It provides you a simple command-line client (which makes it easy to script in cron), is secure (Hi, NSA!) and costs a bit more than the base Amazon S3 service it uses to store your files.

For the canonical directions, see http://www.tarsnap.com/gettingstarted.html.

Create tarsnap account and install client

1. Create your account: https://www.tarsnap.com/register.cgi
2. Click on the link in the "Tarsnap registration confirmation" email
3. Compile the tarsnap client (you must compile)

   # ftp https://www.tarsnap.com/download/tarsnap-autoconf-1.0.35.tgz
   # tar xzvf tarsnap-autoconf-1.0.35.tgz
   # cd tarsnap-autoconf-1.0.35
   # ./configure 
   ...
   # make all install clean
   

4. Login to your account at https://www.tarsnap.com/manage.cgi, then add money:
   • Click "Add funds to your account" link
   • Enter a payment amount of $5 (he shows account balances to 18 decimal places ... in one day, my balance went from $5.000000000000000000 to $4.986442789553223132. :)
   • Enter a credit card or use PayPal.
   • In a few minutes, you will get a "Tarsnap payment processed" confirmation email.

Create tarsnap key and make your first backup

Note: KEEP YOUR TARSNAP KEY. If you lose it, you cannot get your backed up data. Yes, that is correct.

What I did was encrypt it using my keybase.io account and then copied that encrypted file around to a bunch of different places.

5. $ tarsnap-keygen --keyfile /root/tarsnap.key --user mkbucc@gexample.com --machine office
6. (From another host ...)

   $ scp office:/root/tarsnap.key .
   $ brew install keybase
   $ keybase login
   Your keybase username or email: markb
   Your keybase login passphrase: ***********************************
   run scrypt [----------------------------------]
   info: Made directory '/Users/mark/.cache/keybase'
   info: Updated file: /Users/mark/.cache/keybase/session.json
   info: Updated file: /Users/mark/.config/keybase/config.json
   info: Creating temporary keyring dir: /Users/mark/.cache/keybase/tmp_keyrings
   $ keybase encrypt markb tarsnap.key
   $ scp tarsnap.key.asc 
   $ cp tarsnap.key.asc ~/Documents
   $ keybase logout
   info: Removing file: /Users/mark/.cache/keybase/session.json
   $
   

7. Ok, back on server ... create the first backup.

   # mkdir /var/cache/tarsnap
   # cat > /usr/local/bin/backup
   #! /bin/sh -e
   
   /usr/local/bin/tarsnap --keyfile /root/tarsnap.key  --cachedir /var/cache/tarsnap -c  -f $(hostname)-$(date +%s) /home
   ^D
   $ chmod +x  /usr/local/bin/backup
   $ backup
   ... wait, first one takes a while ...
   $ du -sh /var/cache/tarsnap
   196K    /var/cache/tarsnap
   $
   

8. Wrapper script to list backups.

   $ cat > /usr/local/bin/backuplist
   #! /bin/sh -e
   
   /usr/local/bin/tarsnap --keyfile /root/tarsnap.key --list-archives | sort
   ^D
   $ chmod +x /usr/local/bin/backuplist
   # backuplist 
   office-1435712869
   #
   

9. Finally, add to daily cron job

   # cat >> /etc/daily.local
   #! /bin/sh
   
   /usr/local/bin/backup
   ^D
   #
   

It took me longer to write up this blog entry than it did to set this up. Literally.

Enjoy!

OSX, Microsoft Intellipoint Mouse (Explorer Touch) and Acme editor

Doh!

My last post was about trying to get a three-button mouse that works in OSX for the acme editor. I thought I had to use xwindows.

But it turns out, all I had to do was configure the mouse to have OSX handle the middle click. Like so:

On the Microsoft Explorer Touch mouse, you also need to press the middle button more towards the back of the mouse than towards the front to get that middle click to fire.

This is the mouse I am thinking about getting for work, where I do use Linux and so will use i3 and X-Windows, so that work is not all for naught.

Virtual Box (Ubuntu 14.04), i3 (window manager), and acme

My quest to use three-button mouse with acme on mac book continues.

1. download 14.04 desktop iso
2. create new virtual box server
3. update
   1. tell ubuntu to update as part of install
   2. then run built-in updater
   3. then apt-get update/upgrade
   4. install virtualbox guest additions
   5. Create virtual box snapshot (so you can roll back to this state if you need to)

Install software

1. apt-get install i3 git
2. mkdir $HOME/src ; cd $HOME/src
3. git clone https://github.com/9fans/plan9port.git
4. cd plan9port
5. ./INSTALL

Configure Mouse

I'm using Microsoft IntelliPoint mouse, as the middle button is decent. (I found wheels as middle button too hard to click.) Make sure to configure mouse so that middle button is "Handled by Mac OS".

To check, in Ubuntu terminal window, run xev then click in the white window that comes up. The mouse events will echo to the terminal. Buttons 1, 2, and 3 should be left, middle, and right respectively.

i3 Cheat Sheet

full screen

Alt-f

open menu

Alt-d

stack windows

Alt-s

tab windows

Alt-w

logout

Alt-Shift-e

quit window

Alt-Shift-q

Success

I now have a full three-button mouse for using acme editor in OSX.

Install XMonad on OSX

• OSX: 10.9.5 (Mavericks)
• XMonad: 0.11.1
• XQuartz: 2.7.7 (xorg-server 1.15.2)

Steps

1. Download 195MB Haskel platform (Haskell Platform 2014.2.0.0 64bit.signed.pkg) from https://www.haskell.org/platform/mac.html
2. Double-click on pkg file and install.
   
   Note the installer opens an brief intro at file:///Library/Haskell/doc/start.html


3. Make sure XQuartz.app runs. It should open a terminal in X-Windows
   
4. Update cabal (the Haskel package manager). In OSX terminal (not X11 terminal):

   $ cabal update
   Downloading the latest package list from hackage.haskell.org
   Note: there is a new version of cabal-install available.
   To upgrade, run: cabal install cabal-install
   $ cabal install cabal-install
   Resolving dependencies...
   Configuring cabal-install-1.22.6.0...
   Building cabal-install-1.22.6.0...
   Installed cabal-install-1.22.6.0
   $
   

5. Install xmonad.

   $ cabal install xmonad
   ...
   [28 of 28] Compiling Graphics.X11     ( Graphics/X11.hs, dist/build/Graphics/X11.o )
   ld: library not found for -lXss
   clang: error: linker command failed with exit code 1 (use -v to see invocation)
   cabal: Error: some packages failed to install:
   X11-1.6.1.2 failed during the building phase. The exception was:
   ExitFailure 1
   xmonad-0.11.1 depends on X11-1.6.1.2 which failed to install.
   $ 
   

   Googling leads to this page (https://github.com/haskell-pkg-janitors/X11/issues/24) with the comment to try this:
   

   $ LIBRARY_PATH=/opt/X11/lib:$LIBRARY_PATH cabal install X11
   Resolving dependencies...
   Configuring X11-1.6.1.2...
   Building X11-1.6.1.2...
   Warning:
   /var/folders/cy/4988f14n4r35fp39wrt1pfq80000gn/T/pkgConf-X11-1.6.117333.2:
   Unrecognized field data-dir on line 35
   /var/folders/cy/4988f14n4r35fp39wrt1pfq80000gn/T/pkgConf-X11-1.6.117333.2:
   Unrecognized field key on line 4
   Installed X11-1.6.1.2
   $
   

   And we are good
   

   #% cabal install xmonad
   Resolving dependencies...
   Downloading xmonad-0.11.1...
   Configuring xmonad-0.11.1...
   Building xmonad-0.11.1...
   Warning:
   /var/folders/cy/4988f14n4r35fp39wrt1pfq80000gn/T/pkgConf-xmonad-0.1118683.1:
   Unrecognized field data-dir on line 28
   /var/folders/cy/4988f14n4r35fp39wrt1pfq80000gn/T/pkgConf-xmonad-0.1118683.1:
   Unrecognized field key on line 4
   Installed xmonad-0.11.1
   #
   

6. Add symlink. For some reason, after installing with cabal, xmonad binary ended in ~/.cabal/bin not ~/Library/Haskell/bin

   $ cd $HOME
   $ ln .cabal/bin/xmonad Library/Haskell/bin/xmonad
   

7. Create ~/.xinitrc.d/90-xmonad.sh

   $ mkdir ~/.xinitrc.d
   $ cat > ~/.xinitrc.d/90-xmonad.sh
   #! /bin/sh
   USERWM=$HOME/Library/Haskell/bin/xmonad
   ^D
   $ chmod +x ~/.xinitrc.d/90-xmonad.sh
   

8. Configure XQuartz
   
   
9. And open xmonad: alt-tab to make XQuartz active, then Option-Command-Shigt to make xmonad full screen.

Using rlwrap with sam editor

Recently, I've started using the sam editor. I've been using vim for the past 15 years, but recently tried Acme. Acme rocks, but I after a few weeks I missed using keyboard navigation, search and replace, etc. So I "fell back" to sam.

I really like the heavy use of regular expressions in sam, in particular the addressing. For example, here's how to print the full function you are in to the screen: -/^func/;/^}/-+.

I've noticed that since I started using sam, I think more of the names used in my code than the visual display of the code. This happens because you start to operate tell sam to do thing like move the function SendCommand to the end of the file, or 0/^func SendCommand/;/^}/-+ m $

One thing I found annoying is not having a command history for commands like that. I started keeping a text file off to the side where I could copy common commands from and then paste into sam (ala Acme), but I remembered rlwrap so I tried that.

It works, with the exception of the tab key. So instead of typing line indents, I just type the new code and use gofmt (,x|gofmt) or with rlwrap, the following keystroke sequence:

esc-key
/
gof
enter-key
enter-key

which, except for the gof, is in my finger muscle memory.

I am going to try it full-time now; here's how I set it up:

1. If you prefer vi keybindings for history (as I do), put these lines in your $HOME/.inputrc:

   set editing-mode vi
   set keymap vi
   

2. Add an alias to $HOME/.zshrc (or .bashrc, or whatever)

   alias sam="rlwrap sam -d"