Introducing Wala

Posted in code wala pgp crypto storage http rust

This little project is heavily inspired by the "Single-Owner Chunk" (previously "Mutable Resource") concept from the Swarm project [1].

Dubbed wala [2], the rust application provides two features:

  1. Store any uploaded data under the SHA256 digest of the content itself. We call this "content addressed" storage.
  2. Create a link alias to the content using a keyword and a cryptographic identity.

First, let's briefly outline what the first item on the list is.

Curl up and digest

Say an instance of wala is listnening on localhost:8000.

A simple demonstration of the content addressed storage could be:

$ curl -X PUT http://localhost:8000 --data "foo"
2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
$ curl -X GET http://localhost:8000/2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
foo
$ echo -n "foo" | sha256sum
2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae  -

In short, anything uploaded to the endpoint will be stored under and can be retrieved by its hash.

That also means that if the content changes, so does the location.

Get to the pointer

Now let's say you want a permanent link to content that is changing.

If you are the only user of the instance, this would be straightforward thing. We could just use a keyword in the url.

But if you are not, then authentication is needed to make sure that one user doesn't overwrite the another user's permanent link.

This is usually accomplished with a username and password stored and controlled server-side.

But we can also do this by simply using public key cryptography instead.

Claiming what is yours

Since I have a weakness for PGP, the first implementation of authentication in wala has been implemented using the "sequoia openpgp" library.

The procedure is straightforward enough:

  1. Sign the content with your pgp key.
  2. Add an Authorization header [3] with your public key and signature.
  3. Upload the content with an arbitrary keyword. [4]

If the signature over the content matches the public key, then a symbolic link to the content will be created on the server. That symbolic link will be a digest of the key fingerprint and the keyword.

The content can then be accessed both using the content address and the symbolic link.

In wala, the symbolic link is referred to as a mutable reference. We will use this term from now on.

It ain't pretty, but it works

Let's demonstrate creating such an entry in wala using some commonly available tools. [5]

We will use a fictional gnupg private key with the fingerprint F3FAF668E82EF5124D5187BAEF26F4682343F692 and the keyword foo to create the mutable reference.

$ sig=`echo -n "bar" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0`
$ pub=`gpg --export F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 - w 0`
$ curl -v -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data bar
*   Trying 127.0.0.1:8000...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to localhost (127.0.0.1) port 8000 (#0)
> PUT /foo HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.85.0
> Accept: */*
> Authorization: PUBSIG pgp:mQGNBF+hSOgBDACpkPQEjADjnQtjmAsdPYpx5N+OMJBYj1DAoIYsDtV6vbcBJQt94Om3xl7RBhv9m2oLgzPsiRwjCEFRWyNSu0BUp5CFjcXfm0S4K2egx4erFnTnSSC9S6tmVNrVNEXvScE6sKAnmJ7JNX1ExJuEiWPbUDRWJ1hoI9+AR+8EONeJRLo/j0Np+S4IFDn0PsxdT+SB0GY0z2cEgjvjoPr4lW9IAb8Ft9TDYp+mOzejn1Fg7CuIrlBRSAv+sj7bVQw15dh1SpbwtS5xxubCa8ExEGI4ByXmeXdR0KZJ+EA5ksO0iSsQ/6ipSOdSg+i0niOClFNm1P/OhbUsYAxCUfiX654FMn2zoxVBEjJ3e7l0pH7ktodaxEctPofQLBA9LSDUIejqJsU0npw/DHDD2uvxG+/A6lgV9L8ETlvgp8RzeOCf2bHuiKYYz87txvkFwsXgU1+TZxbk+mtCBbngsVPLNarY/KGkVJL+yhcHRD0Pl4wXUd6auQuY6vQ9AuKiCT1We2sAEQEAAbQeTWVyIE1hbiA8bWVybWFuQGdyZXlza3VsbC5jb20+iQHUBBMBCAA+FiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAl+hSOgCGwMFCQPCZwAFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AACgkQ7yb0aCND9pLwiwwAhFJbAyUK05TJKfDz81757N472STtB8sfr0auwmRr8Zs1utHRVM0b/jkjTuo4uJNr7YVVKTKgE7+rJ+pwhm3wlTQ44LVLjByWAi/7NWg3E9b2elm+qkfgm/RfFt3vkuOxGSyZyIFFh+/twv6iABPvr6w7MZwrFaS0UP3g1VGa5TFqg6KNxod9H/gPLxv45lutXf3VvBZTJpr1pxn7aLHlFzEyIgNZbP/N1QF44GSrN/k0DfL631sZjauUXaZXbi5xGsKKCYwJ1g3q587pi6mTdTV3n0hKgVuipO8hGy5++YeOv+hXsCxDwyZ+Shv+qavd/SapxYgCdEueuwONIFfsIsWCd3SCcjKXicTTEFMu8nvBmf7xuo2hv6vEOxoijlXV+4LkGrskdB8ZMg8PywEx6DLmDokgnAhTLrTc1ShbkOtQ3yNjjyFK7BDpqobsJal6d8SpbhccUJLepaSmsk0CgJsTjhAl6EwX0EYgTo3kP5fScqrbD8VwQaT8CcE4rCV4uQGNBF+hSOgBDADHtpTT1k4x+6FN5OeURpKAaIsoPHghkJ2lb6yWmESCa+DaR6GXAKlbd0L9UMcXLqnaCn4SpZvbf8hP4fJRgWdRl5uVN/rmyVbZLUVjM8NcVdFRIrTsNyu4mLBmydc3iA/90sCTEOj9e7DSvxLmmLFjpwM5xXLd6z0l6+9G+woNmARXVS3V/RryFntyKC3ATCqVlJoQBG45Tj2gMIunpadTJXWmdioooeGW3sLeUv5MM98mSB4SjKRlJqGPNjx5lO6MmJbZeXZ/L/aO6EsXUQD2h82Wphll4rpGYWPiHTCYqZYiqNYr6E3xUpzcvWVp3uCYVJWP6Ds117p7BoyKVz00yxC9ledF3eppktZWqFVowCMihQE3676L3DDTZsnJf1/8xKUh5U2Mj3lBvjlvCECKi00qo8b1mn/OklQjJ5T4WzTrH6X+/zpez8ZkmtcOayHdUKD/64roZ9dXbXG/hp5A+UWj8oSVYKg2QNAwAnZ+aiZ2KVRE/Y61DCgFg6Ccx/cAEQEAAYkBvAQYAQgAJhYhBPP69mjoLvUSTVGHuu8m9GgjQ/aSBQJfoUjoAhsMBQkDwmcAAAoJEO8m9GgjQ/aSIPcL/3jqL2A2SmC+s0BO4vMPEfCpa2gZ/vo1azzjUieZu5WhIxb5ik0V6T75EW5F0OeZj9qXI06gW+IM8+C6ImUgaR3l47UjBiBPq+uKO9QuT/nOtbSs2dXoTNCLMQN7MlrdUBix+lnqZZGSDgh6n/uVyAYw8Sh4c3/3thHUiR7xzVKGxAKDT8LoVjhHshTzYuQq8MqlfvwVI4eESLaryQ+Y+j5+VLDzSLgPAnnIqF/ui2JQjefJxm/VLoYNaPAGdqoz/u/R0Tmz94bZUfLjgQaDoUpnxYywK2JGlf3mPZ3PNWjxJzuQTF5Ge5bz/TylnRYIyBT7KD7oaKHO62fhDbYPJ4f94iZN4B6nnTAeP34zFDlkUbX4AHudXU7bvxT5OUk9x9c2tj7xwxQHaEhq2+JsYW0EVw27RLhbymnBfLjVVUktNF0nQGvU2TEocw4pr2ZkDHQkSnlbNa4kujlL7VzbpnEgyOmi5er9GaIuVSVADovBu+pz/Ov1y/3jUe8hZ/KleQ==:iQGzBAABCAAdFiEE8/r2aOgu9RJNUYe67yb0aCND9pIFAmM9guIACgkQ7yb0aCND9pIEdwwAiLLqFlrKu0UsQebfuUP07cvGbYy9LfbCMsQj/3/pG/zl7q2mSl2YdXOalbaYD2uyGU/sm7J/+qQZXGyIjmDA7F53sNVAXTuYnrcKmYIzAmzW4lUAzfWA7yL55MtbR/eNUE1rqp/Gu/ejj1OedLyxi+tGFcXUHU0q8EnjQnzfHCJVzOa3PGMIX10NiXPjrF2pafAyE7q2ogwkKZdjJi+8tyAw0tviu4CRGOVlsNZlF+yxePZh55XdRZLCEt4n6mnJrccu0C22rM9R2dEReqGLAj8t/WhACI+UyNXtL+hICnu9y6wjk4spoMr0s0pqTQ76SMwfmRFzk11uZ+ge846hArcUxE27+AeBf9Q1IwT5Ypsc0Efm9ZPoJvA2ggcJv1Yyb58Ggfmd02xPW4EQ8MOEMLA/ZoAhOm3t3wATPNFG1ucm/o+NFNDpF7HNby+Savqv2NrbNwDMlWvFzRmER2+AIO0CIG2HVJScMEn7UkjF8jIm+ba3BIAXbz2FUZ3dytFF
> Content-Length: 3
> Content-Type: application/x-www-form-urlencoded
>
} [3 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: tiny-http (Rust)
< Date: Wed,  5 Oct 2022 13:15:37 GMT
< Content-Type: text/plain; charset=UTF-8
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: OPTIONS, PUT, GET
< Access-Control-Allow-Headers: Content-Type,Authorization,X-Filename
< Content-Length: 64
<
{ [64 bytes data]
100    67  100    64  100     3   3548    166 --:--:-- --:--:-- --:--:--  3722100    67  100    64  100     3   3534    165 --:--:-- --:--:-- --:--:--  3722
* Connection #0 to host localhost left intact
32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
$ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
bar
$ echo -n bar | sha256sum
fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9
$ curl -X GET http://localhost:8000/fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9
bar

Now, by changing the data and signature, the data under the mutable resource changes:

$ sig=`echo -n "xyzzy" | gpg -b -u F3FAF668E82EF5124D5187BAEF26F4682343F692 | base64 -w 0`
$ curl -X PUT http://localhost:8000/foo -H "Authorization: PUBSIG:$pub:$sig" --data xyzzy
32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
$ curl -X GET http://localhost:8000/32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2
xyzzy

Building your identity

We can also recreate the symbolic link hash using local tools:

$ t=`mktemp`
$ d=`echo -n foo | sha256sum | awk '{ printf "%s",$1; }' | sed -e 's/../\\\\x&/g'`
$ echo -ne $d > $t
$ k=`echo -n F3FAF668E82EF5124D5187BAEF26F4682343F692 | sed -e 's/../\\\\x&/g'`
$ echo -ne $k >> $t
$ cat $t | sha256sum
32ea47ecc5a3ee2576aab00c2f30eaabc2592d56e19f5c82fe4f7cf5874632b2  -

In other words, the mutable reference is constructed using the following recipe.

  1. Calculate binary value R of the SHA256 digest of the keyword (sha256(foo) -> 0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AE)
  2. Calculate binary value K of the key fingerprint (0xF3FAF668E82EF5124D5187BAEF26F4682343F692).
  3. Calculate binary value of SHA256 digest of R | K (sha256(0x2C26B46B68FFC68FF99B453C1D30413413422D706483BFA0F98A5E886266E7AEF3FAF668E82EF5124D5187BAEF26F4682343F692) -> 0x32EA47ECC5A3EE2576AAB00C2F30EAABC2592D56E19F5C82FE4F7CF5874632B2)

That means that anyone who knows the keyword and the public key of the uploader can calculate the mutable reference themselves, and retrieve the data behind it.

[1]Refer to section 4.3 of The Book of Swarm for a description of "feeds" and "single owner chunks." Swarm uses signatures to allow propagation of the data in the network. wala similarly uses signatures to accept update of resources "owned" by the holder of the private keys of an identity.
[2]At the time of writing wala is at version 0.1.1
[3]A custom authorization scheme PUBSIG has been invented for this purpose. The format is Authorization: PUBSIG pgp:key-fingerprint-in-base64>:<signature-in-base64>. Currently this is strictly just authentication, as there is no feature in wala (yet) to use access control lists to determine which public keys to allow PUTs from.
[4]If you do not specify a keyword, the keyword value in the mutable resource reference will be the sha256 hash of an empty value (0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855)
[5]wala comes with a binary tool wala_send that lets you specify a key fingerprint in your default pgp keyring and a keyword as arguments when you upload data. A lot less messy, but a lot less educational, too.