Sunday, March 27, 2016

dets:select_delete: exception error: bad argument

When first learning about how to delete records from an Erlang dets table, I kept getting this error from the dets:select_delete() function:

Erlang/OTP 18 [erts-7.2.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Eshell V7.2.1  (abort with ^G)
1> c(dets_select_delete).  
{ok,dets_select_delete}
2> dets_select_delete:test().
** exception error: bad argument
     in function  dets:select_delete/2
        called as dets:select_delete(test,[{{'_','_','$1'},[{'<','$1',3}],[]}])
3>

Here is the code:

-module(dets_select_delete).
-export([test/0]).

% Test syntax of match clause.
test() ->
    %delete_dets(),
    {ok, test} = dets:open_file(test, [{type, set}, {file, "test.dets"}]),
    % Add some data
    ok = dets:insert(test, {a, 1}),
    ok = dets:insert(test, {b, 2}),
    ok = dets:insert(test, {c, 3}),
    Match = [{{'_','_','$1'},[{'<','$1',3}],[]}],
    DeletedN = dets:select_delete(test, Match),
    DeletedN.

The issue was with the return value clause:

    Match = [{{'_','_','$1'},[{'<','$1',3}],[]}],
                                           ^^^^
                                             |
                                             +-- wrong return value clause.

This is the correct match specification:

    Match = [{{'_','_','$1'},[{'<','$1',3}],[true]}],
                                           ^^^^^^^
                                             |
                                             +-- right return value clause.

I was confused by the examples "Match specifications in Erlang" [http://erlang.org/doc/apps/erts/match_spec.html], the simplest of which always have an empty return clause. Since the match spec is kind of complex, I thought it was a problem in my match spec.

I sorted this out by using the shell, and verifying my match spec was correct with the dets:select/2. Once I knew it was correct, I went back and read the docs to select_delete again, and there it is, clear as day:

Deletes each object from the table Name such that applying the match specification MatchSpec to the object returns the value true.

select_delete(Name, MatchSpec) -> N | {error, Reason}, Retrieved March 27, 2016.

Saturday, February 6, 2016

Erlang: The equals sign is a matching operator

In all previous programming languages I have used before Erlang, the equals sign means assignment. When you see
X = 1;
this reads as "assign the value 1 (one) to X."

Erlang is different. In Erlang, you read this like a mathematical statement: "find the value of x that makes the left-hand side of the equation equal to the right-hand side."

This simple difference in the meaning of = quickly gets more useful. For example, consider this:

"/" ++ Path = HttpRequest:get(path).

In Erlang, this says "find the value of Path that makes this statement true". This is nearly equivalent to the following Java code:

String Path = HttpRequest.getPathInfo().substring[1:];

I say nearly equivalent, because there is something else going on in the Erlang statement. The Erlang equals sign is stating that the equality holds, so a more precise Java translation is:

String Path;
if (HttpRequest.getPathInfo().startsWith("/")) {
   Path = HttpRequest.getPathInfo().substring[1:];
} else {
   String emsg = "no match of right hand side value \"%s \"";
   throw new IllegalStateException(emsg.format(HttpRequest.getPathInfo());
}

That one line of Erlang both asserts a pre-condition and binds a variable to a value.

This technique works in function arguments:

$ cat t.erl
-module(t).
-export([test/1]).

test("/" ++ X) -> X.
$ erl
1> c(t).
{ok,t}
2> t:test("abc").
** exception error: no function clause matching t:test("abc") (t.erl, line 4)
3> t:test("/abc").
"abc"
4>

And for other types:

4> [{X,Y}] = [{1,2}, {2,3}].
** exception error: no match of right hand side value [{1,2},{2,3}]
5> [{X,Y}] = [{1,2}].       
[{1,2}]
6> 

What I am finding as I code more in Erlang is that I now code only the happy path. This is a fun way to write code! I like to write safe code, but having to catch and handle all the error cases sucks the fun out of it. With Erlang, I get both: I write the happy path and I get pre-condition assertions built-in.

And going back to the first example, what are you going to do in a Java Servlet running in a container so buggy that the CGI PATH_INFO variable is missing it's leading slash? Crashing and then restarting the Servlet seems like a reasonable action. That's exactly what Erlang does by combining pattern matching and Supervisor behaviors.

Thursday, July 23, 2015

Proxy golang web server with relayd on OpenBSD 5.7

If you are new to OpenBSD, you may wonder why the httpd daemon that in the base install does not provide the ability to proxy web requests. The reason is that this capability is already provided in base by another daemon called relayd.

This post gives you a quick recipe for how to setup relayd to proxy http requests to a golang web server running as a non-priviledged user. Note that this barely scratches the surface of what relayd can do; for more info, you can read

Steps:

  1. Add a new user, accepting defaults.
  2. As that user, start up golang web server, listening on port 8080.
    $ cat > srv.go
    package main
    
    import (
            "fmt"
            "log"
            "net/http"
            "time"
    )
    
    func withalog(h http.Handler) http.Handler {
            f := func(w http.ResponseWriter, r *http.Request) {
                    fmt.Printf(
                            "%s - - [%s] \"%s %s %s\"\n",
                            r.Header["X-Forwarded-For"],
                            //r.RemoteAddr,
                            time.Now().Format("02/Jan/2006 03:04:05 EST"),
                            r.Method,
                            r.URL,
                            r.Proto)
                    h.ServeHTTP(w, r)
            }
            return http.HandlerFunc(f)
    }
    
    func main() {
            handler := http.FileServer(http.Dir("/usr/share/doc"))
            log.Fatal(http.ListenAndServe(":8080", withalog(handler)))
    }
    ^D
    $ go build srv.go
    $ ./srv &
    $ curl http://127.0.0.1:8080/../..
    [] - - [04/Jul/2015 10:20:21 EST] "GET / HTTP/1.1"
    <pre>
    <a href="mg/">mg/
    </pre>
    $
    
  3. Configure relayd to forward 80 to 8080
    # cat > /etc/relayd.conf
    http protocol "littleproto" {
            return error
            match request header append "X-Forwarded-For" \
                value "$REMOTE_ADDR"
    }
    
    relay myproxy {
            listen on 192.168.30.6 port 80
            protocol "littleproto"
            forward to 127.0.0.1 port 8080
    }
    ^D
    # relayd -n
    configuration OK
    #
    
  4. Set it to start on reboot.
    # echo relayd_flags="" >> /etc/rc.conf.local
    #
    
  5. Start it and test.
    # relayd
    # curl http://192.168.30.6
    <pre>
    <a href="mg/">mg/
    </pre>
    

Questions

What does return error do in http protocol?
Tells relayd to return an error pages. If you don't specify the style option (see man page), this is the default styling:
If I just want a http reverse proxy, do I need to configure PF?
No.
What's the differences between relay httpproxy {... and redirect www {... , both of which are in /etc/examples/relayd.conf.
Not 100% sure. From reading relay.conf(5), it says that a redirection does stateful forwarding, and a relay is for general purpose TCP proxy.

Wednesday, July 15, 2015

Your own private git remote

I have some documents (writing, financial data, some code) that I want to keep private. This describes how to setup a git remote repository on your home network, and to back it up to the cloud (daily) with tarsnap.

Definitions:

office
The host that holds your remote git repo.
lap
The computer where you do your programming.

Here are the steps:

  1. Create a git-private user on office. On OpenBSD, that is
    adduser git-private
  2. Create ssh key-pair on lap
    mark@lap:~$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/mark/.ssh/id_rsa): /home/mark/.ssh/id_rsa_git-private
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /home/mark/.ssh/id_rsa_git-private.
    Your public key has been saved in /home/mark/.ssh/id_rsa_git-private.pub.
    The key fingerprint is:
    cc:c7:c8:ab:36:9b:1a:d1:61:d1:2f:9a:06:4b:97:16 mark@lap
    The key's randomart image is:
    +--[ RSA 2048]----+
    |      ..         |
    |      E..        |
    |      oo .       |
    |    oo+=.o.      |
    |   ..=.oS.o      |
    |    ..+  o       |
    |    ..  .        |
    |     .oo         |
    |    .o+o         |
    +-----------------+
    mark@lap:~$
    
  3. Authorize key.
    scp ~/.ssh/id_rsa_git-private.pub git-private@office:~/.ssh/authorized_keys
  4. On lap, associate private key with office
    cat >> ~/.ssh/config
    Host            office
     Hostname office
     IdentityFile    ~/.ssh/id_rsa_git-private
     User            git-private
    ^D
    
  5. Set up bare remote git repo on office.
    mark@lap:~$ ssh git-private@office
    git-private@office:~$ git init --bare myrepo.git
    
  6. On lap, add remove repo as remote.
    mark@lap:~$ git remove -v
    mark@lap:~$ git remote add origin git-private@prod:/home/git-private/myrepo.git
    mark@lap:~$ git push -u origin all
    
  7. To securely backup encrypted versions of your remove, see my Recipe for setting up backups with tarsnap (OpenBSD 5.7).

Thursday, July 2, 2015

Recipe for setting up backups with tarsnap (OpenBSD 5.7)

Tarsnap is a wicked smart backup service. (The author started studying math at university at age 13!) It only saves the deltas since your last backup, which means you can always do a full backup but you use much less space. Tarsnap (somehow) stores enough info that it makes every backup act like a full backup; for example, even if you delete all previous backups the last one will still be a full.

It provides you a simple command-line client (which makes it easy to script in cron), is secure (Hi, NSA!) and costs a bit more than the base Amazon S3 service it uses to store your files.

For the canonical directions, see http://www.tarsnap.com/gettingstarted.html.

Create tarsnap account and install client

  1. Create your account: https://www.tarsnap.com/register.cgi
  2. Click on the link in the "Tarsnap registration confirmation" email
  3. Compile the tarsnap client (you must compile)
    # ftp https://www.tarsnap.com/download/tarsnap-autoconf-1.0.35.tgz
    # tar xzvf tarsnap-autoconf-1.0.35.tgz
    # cd tarsnap-autoconf-1.0.35
    # ./configure 
    ...
    # make all install clean
    
  4. Login to your account at https://www.tarsnap.com/manage.cgi, then add money:
    • Click "Add funds to your account" link
    • Enter a payment amount of $5 (he shows account balances to 18 decimal places ... in one day, my balance went from $5.000000000000000000 to $4.986442789553223132. :)
    • Enter a credit card or use PayPal.
    • In a few minutes, you will get a "Tarsnap payment processed" confirmation email.

Create tarsnap key and make your first backup

Note: KEEP YOUR TARSNAP KEY. If you lose it, you cannot get your backed up data. Yes, that is correct.

What I did was encrypt it using my keybase.io account and then copied that encrypted file around to a bunch of different places.

  1. $ tarsnap-keygen --keyfile /root/tarsnap.key --user mkbucc@gexample.com --machine office
  2. (From another host ...)
    $ scp office:/root/tarsnap.key .
    $ brew install keybase
    $ keybase login
    Your keybase username or email: markb
    Your keybase login passphrase: ***********************************
    run scrypt [----------------------------------]
    info: Made directory '/Users/mark/.cache/keybase'
    info: Updated file: /Users/mark/.cache/keybase/session.json
    info: Updated file: /Users/mark/.config/keybase/config.json
    info: Creating temporary keyring dir: /Users/mark/.cache/keybase/tmp_keyrings
    $ keybase encrypt markb tarsnap.key
    $ scp tarsnap.key.asc 
    $ cp tarsnap.key.asc ~/Documents
    $ keybase logout
    info: Removing file: /Users/mark/.cache/keybase/session.json
    $
    
  3. Ok, back on server ... create the first backup.
    # mkdir /var/cache/tarsnap
    # cat > /usr/local/bin/backup
    #! /bin/sh -e
    
    /usr/local/bin/tarsnap --keyfile /root/tarsnap.key  --cachedir /var/cache/tarsnap -c  -f $(hostname)-$(date +%s) /home
    ^D
    $ chmod +x  /usr/local/bin/backup
    $ backup
    ... wait, first one takes a while ...
    $ du -sh /var/cache/tarsnap
    196K    /var/cache/tarsnap
    $
    
  4. Wrapper script to list backups.
    $ cat > /usr/local/bin/backuplist
    #! /bin/sh -e
    
    /usr/local/bin/tarsnap --keyfile /root/tarsnap.key --list-archives | sort
    ^D
    $ chmod +x /usr/local/bin/backuplist
    # backuplist 
    office-1435712869
    #
    
  5. Finally, add to daily cron job
    # cat >> /etc/daily.local
    #! /bin/sh
    
    /usr/local/bin/backup
    ^D
    #
    

It took me longer to write up this blog entry than it did to set this up. Literally.

Enjoy!

Wednesday, July 1, 2015

OSX, Microsoft Intellipoint Mouse (Explorer Touch) and Acme editor

Doh!

My last post was about trying to get a three-button mouse that works in OSX for the acme editor. I thought I had to use xwindows.

But it turns out, all I had to do was configure the mouse to have OSX handle the middle click. Like so:

On the Microsoft Explorer Touch mouse, you also need to press the middle button more towards the back of the mouse than towards the front to get that middle click to fire.

This is the mouse I am thinking about getting for work, where I do use Linux and so will use i3 and X-Windows, so that work is not all for naught.

Wednesday, June 24, 2015

Virtual Box (Ubuntu 14.04), i3 (window manager), and acme

My quest to use three-button mouse with acme on mac book continues.

  1. download 14.04 desktop iso
  2. create new virtual box server
  3. update
    1. tell ubuntu to update as part of install
    2. then run built-in updater
    3. then apt-get update/upgrade
    4. install virtualbox guest additions
    5. Create virtual box snapshot (so you can roll back to this state if you need to)

Install software

  1. apt-get install i3 git
  2. mkdir $HOME/src ; cd $HOME/src
  3. git clone https://github.com/9fans/plan9port.git
  4. cd plan9port
  5. ./INSTALL

Configure Mouse

I'm using Microsoft IntelliPoint mouse, as the middle button is decent. (I found wheels as middle button too hard to click.) Make sure to configure mouse so that middle button is "Handled by Mac OS".

To check, in Ubuntu terminal window, run xev then click in the white window that comes up. The mouse events will echo to the terminal. Buttons 1, 2, and 3 should be left, middle, and right respectively.

i3 Cheat Sheet

full screen
Alt-f
open menu
Alt-d
stack windows
Alt-s
tab windows
Alt-w
logout
Alt-Shift-e
quit window
Alt-Shift-q

Success

I now have a full three-button mouse for using acme editor in OSX.