Catching the Blockchain Train

The Lightning Network Rookie Guide

2018-03-18T21:22:00+01:00

This article is part 1 of the new "Catching the BlockChain train" series: "The Lightning Network Rookie Guide".

Oh my, a new series! Tell me more!

After five episodes about IPFS, it's time for another exciting technology in blockchain world: Lightning Networks. From here on just Lightning Network (singular, I'll explain why later) or LN for short.

A quick test to see if you paid attention: LN. Did you expand this in your mind to Lightning Network? Well done, you passed!

BTW if you missed the episodes on IPFS, start here.

I like the idea to mix hands-on (playing with SW tools) with reading about and understanding the technology. This is reflected in the outline of this Rookie Guide*:

Play with the LN as an end user: this episode.
Understanding the basics of the LN.
Preparation: set up a Bitcoin full-node and a local block explorer.
LN hands-on: installing LN nodes, more wallets, apps, buying a t-shirt on mainnet, etc.
Understanding the LN better: diving into the white paper and the spec.

*) If the term rookie offends you: I'm talking about myself of course. I'm figuring this all out while I write these posts turning myself from a rookie into a beginner.

Satoshi's Vision!

Before we start, here's a glimpse of the genius of Satoshi Nakamoto: Satoshi's Vision! (Payment Channels / LN). That was in 2010!

We have a long way to go before we understand his specific statement:

nTimeLock does the reverse. It's an open transaction that can be replaced with new versions until the deadline. It can't be recorded until it locks. The highest version when the deadline hits gets recorded. It could be used, for example, to write an escrow transaction that will automatically permanently lock and go through unless it is revoked before the deadline. The feature isn't enabled or used yet, but the support is there so it could be implemented later.

but we'll get there, one step at the time.

Apparently, he was at it again in 2013: This email, composed in April of 2013 by Mike Hearn, quotes Satoshi talking about what is now known as the #LightningNetwork.

What's up with all the hype about the Lightning Network?

If you have been following Bitcoin tech, you are most likely aware of the Lightning Network. If not: it is a proposed solution to increase the scalability of the Bitcoin network by allowing "off-chain" transactions in a trustless way.

There has been a lot of talk about the LN lately, probably because there are a couple of implementations and it seems to work.

People are already using it on top of the main Bitcoin network. And three days ago the first LN node for mainnet was announced.

Here is a cool visualization of the current mainnet LN. When we are halfway this rookie guide I hope you and I to have our own node in there!

Now, why is the Lightning Network so exciting? I'm a fanboy for quite a while now, but this podcast with Elizabeth Stark and Andreas Antonopoulos got me really hooked: Let's Talk Bitcoin! #357 Real Lightning with Elizabeth Stark.

I encourage you to listen to the whole podcast, but if you don't, no worries, here are the points that stand out (IMHO at least):

Scalability improvements for Bitcoin: speedy and low-fee transactions as Bitcoin was intended/advertised are now possible.
Micropayments: for the first time in history there will be technology that makes small, cross-border, almost fee-free, payments possible.
Interoperability: one LN for all blockchains. Just as the internet is a network of networks, we can see the LN as the blockchain of blockchains. Standardization is underway so that the different blockchains can work against a common interface. These specs are named BOLTs (like BIPs for Bitcoin).
Cross chain swaps: this interoperability facilitates atomic swaps from, e.g., Bitcoin to Ethereum or any blockchain that interfaces with this overlaying LN. Genuinely decentralized exchanges will then be possible.
Application layer: on top of the LN (layer two above Bitcoin e.a. which represent layer one) there is layer three which will be the application layer. Here is where the zapps or lapps live.
Developer platform: the LN opens up a whole world of possibilities for developers. The LND implementation offers a gRPC interface that makes writing (z|l)apps easier. We will dive into that later for sure!
Developer community: there are 100s of developers active in this space, so let's hurry up and become one of them. They hang out in the Lightning Labs slack and get their documentation here.

Now if that doesn't get you excited... (it probably means you have a fun and meaningful life).

OK then, let's see how it looks like from the end-user perspective...

Lightning Network wallets

Before diving into the workings of the LN protocol and before setting up our own LN node let's try out some wallets that are available right now:

Lighting Wallet
HTLC.me
Eclair Wallet
Zap Desktop Wallet
Lightning Desktop App

First get some testnet bitcoin

We use the wallets on the bitcoin testnet, so to play along you'll need some testnet bitcoin (tBTC). As in the old days for mainnet bitcoin, you can get them at a faucet.

This faucet and this one also worked for me. If you have trouble getting testnet coins, please tweet to me, and I'll send you some.

Note that testnet addresses have m, n, or 2 as a prefix.

Testnet bitcoins haven't (and never will have) any value, but it still feels exciting to see a free 200.000.000 satoshis coming in :)

Lightning Wallet

This Lightning Wallet is for Android only; you can download it here: Bitcoin + Lightning Wallet.

I love this wallet as it is a one-guy project in the person of Anton Kumaigorodskiy and it has some innovative features.

This wallet has an excellent online user manual, let's walk through it:

Setting up a Bitcoin wallet: this app is very convenient as it is an SPV Bitcoin node with a Lightning node built on top of it. Follow the instructions in the manual and send some testnet coins to your wallet. Note that you need a wait for the transaction to be confirmed a couple of times to proceed.

Next: Using the Lightning wallet.

Select the Lightning tab and click Open new channel. Now select a channel or search for one of the recommended channels by Anton. I picked oh hi mark as it made me feel very welcome. Then fund the channel (hey, you can fund the channel with a sub-satoshi resolution!) and wait for two confirmations.

These confirmations can take a while as it is an actual bitcoin transaction. The amount you funded determines the maximum of the sum of the payments done through this channel. We'll dive into all this in the next episode of this guide.

As soon as we have an active payment channel, we can buy something.

My first purchase was an article at yalls.org for 100 test-satoshi, then left a comment for 21 test-satoshi. Micropayments are a reality!

I also bought a Blockaccino at starblocks. A few more places where you can spend tBTC here.

What happens here is that the payment is routed over a series of payment channels through a number of nodes. And all in a trustless way, with super low fees and very very fast. Pretty amazing no?

Watch the whole process in this Mobile Lightning wallet demo.

Anton also created another useful video: When Lightning goes wrong. It describes the various scenarios when channels are closed or when payments get stuck. These are the scenarios he describes:

A mutual (cooperative) closing transaction: both peers of the channel agree to close the channel, and the remaining balance is settled on the blockchain without any delays. Both peers need to be online for this to work.
Forced channel closing if a peer does not respond or becomes uncooperative. The peer that forces the channel to close is "punished" by a very long waiting time for their bitcoin to become spendable.
Your peer wants to steal from you by sending a previous commitment transaction to the blockchain. We see that the peer is punished for that and gets less than he would have had he played by the rules.

We'll find out in the next episode how all this is accomplished.

A last cool feature of this wallet consists of two parts: the app on your phone and a server that offloads some of the work from the mobile app. It's named Olympus, and there's a video about it here.

To receive a Lightning Network payment, we need another peer so let's move on to the next LN wallet...

HTLC.me

HTLC.me is the easiest way to get started, as you don't need to install anything and it gives you a bit tBTC to play with.

From the user's perspective this is not a wallet, but a centralized front-end to a Lightning Network node. It keeps track of your balance, and you can access it by applying the recovery secret they provide the first time you activate your "wallet".

Anyway, now that I created a wallet on HTLC.me and funded it with some tBTC, I can use it to test the receiving function on Anton's Lightning Wallet:

From Anton's Lightning Wallet user guide:

Receiving Lightning funds

In order to receive Lightning payments, some conditions must be met:

Nothing can be received immediately after creating a new payment channel, as ‘room’ for incoming funds has to be made by spending some funds first. A payment channel can be thought of as a full bottle of water: in order to pour something in one first has to pour something out.

Each channel implicitly contains a reserve which is unspendable and typically takes about 1% of the channel’s capacity. You must spend that reserve before receiving is allowed. Unspendable channel reserve is the reason you see a negative receive limit when a new channel is full. It indicates how much you need to spend before anything can be received through the channel.

Every payment request is disposable, they can’t be fulfilled multiple times. So you will need to issue a new individual payment request for every incoming payment you wish to receive.

Your Lightning wallet needs to be online in order to receive funds.

Next to that, there is also a risk receiving payments on a mobile app that is not always online. The peer can attempt to steal from you by sending a previous commitment transaction. Your LN client can only correct this if it is online at least once per day.

After making some space in the channel from my mobile app, I gave it a shot. Although the steps are very smooth, the actual payment failed as HTLC.me couldn't establish a connection to my mobile wallet.

Let's try again with another wallet!

Eclair Wallet

The Eclair wallet is also for Android only. It was announced in this blog post.

It works more or less the same as Anton's Lightning Wallet. One difference is that it doesn't allow receiving tBTC as explained in the announcement above.

The Eclair Wallet exposes some of the technical details like the Node id (Lightning Network's public identifier) of the wallet and the connected peers. Node addresses have the form of some-public-key@ip-address:port, e.g., 03dc39d7f43720c2c0f86778dfd2a77049fa4a44b4f0a8afb62f3921567de41375@213.133.99.89:9735.

Speaking of peers, the Eclair wallet allows connecting to multiple peers (aka more than one payment channel). If you know the Node id you can look it up in this brilliant explorer: https://explorer.acinq.co/#/n/03dc39d7f43720c2c0f86778dfd2a77049fa4a44b4f0a8afb62f3921567de41375.

From the wallet, you can scan that Node id and add it as another peer.

Let me sit down a bit to let this sink in... It is a cliche, but we are living in the future. Visa et al. should be scared.

Now, let's try again if we can receive with Anton's Lightning Wallet. I have both wallets connected to the same node to make it a bit easier (is it?). Unfortunately, we have no influence which channel the Eclair wallet uses.

A little while later I wake up in the now: one forever pending payment and another one failed. Visa can breath normal again.

Let's try with another wallet, maybe we have more luck there.

Zap Desktop Wallet

The Zap wallet is a different beast; it uses a full lnd node.

This wallet was first announced in this article: Announcing Zap: A Lightning Network Wallet. It has an embedded video with a walk-through that gives a good idea how it all works. But the better option is to download the wallet and play along! (BTW be warned this app is not ready for real use, it uses all CPU now and then, so only proceed if you like the sound of your computer's fans).

Download the latest release for your platform here: releases.

When running it the first time it takes a long time before you can use it. While it syncs, you can watch a couple of video's: Zap Lightning Network Wallet Tutorial.

I find this wallet not as intuitive as the other ones I played with, but it compensates this with the fact it is powered by a full lightning node. That hopefully means we can finally see how receiving tBTC works.

Another try to pay from Zap to Anton's app failed again, and payments to the Zap wallet from either mobile app also resulted in a RouteNotFound. Not sure what is going on here, but receiving money is harder than spending it, just like in real life.

Let's move on to our last wallet...

Lightning Desktop App

The latest release of this wallet (still massively in development) can be downloaded here.

It was announced here: Announcing Our Lightning Desktop App, Now Available for Testing.

Unfortunately, I couldn't get it to work on my Mac. I created an issue, so when things get fixed (or they point me out what is wrong on my side) I'll update this article.

First Conclusions

Making payments is unreal, fast and cheap!

Adding a peer node by reading a QR-code on the Lightning Network Explorer is like magic!

Receiving payments works very smooth if it worked... All my attempts failed, but OTOH, wallets that are not always online shouldn't receive payments in the first place. We have full nodes running on a server for that. Soon to be seen on this blog!

Of all the wallets both mobile wallets are my favorite, as it is more natural to pay with a QR scan from your phone than copy-pasting payment requests.

The next episode will be a bit less hands-on. I will dive deeper into the LN protocol, and hopefully, I can share that with you in an easy to understand way.

Ten terrible attempts to make IPFS human-friendly

2017-09-14T16:44:00+02:00

This article is part 6 of the Blockchain train journal, start reading here: Catching the Blockchain Train.

IPFS human-friendly naming; is it possible?

Did you see the traffic-magnet-title I came up with? It is a bit over the top; we're just looking at how the addressing of content can be made a bit easier than ipfs cat /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV.

As mentioned in the previous post, adding some mutable "label" to point to (immutable) IPFS content is sometimes desired.

An example is the content of this blog: the original address (hash) for the home page /ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/ is different from the current (ah no, already expired!) home page /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/.

Mutable links are essential, but not enough for us humans. They are hard to read, let alone remember.

What we'll do in this episode is to explore different ways to:

resolve this problem of adding mutable links, and
to make the links more human-friendly, while
trying to stay decentralized where possible.

We'll see the following methods in action:

The IPFS gateway
IPNS
DNS TXT IPNS Records
Browser extensions
Namecoin
Blockstack ID
Ethereum Name Service
Proquint Pronounceable Identifiers
Name shorteners
Filecoin

TL;DR To make IPFS more human-friendly, in the context of a decentralized IPFS powered website, there is currently no other way than sacrificing a part of decentralization using HTTP-to-IPFS gateways and old school DNS. Naming systems in the blockchain sphere (Blockstack, Namecoin, EthNames) are possible candidates to improve the UX while keeping it 100% decentralized, but at this point of time none of these work out of the box. Filecoin might be the way to go, but that is still far away.

The IPFS gateway

The most obvious step to make content available on the old skool internet with a regular browser is to make use of the IPFS gateway. These gateways speak HTTP and access content through their IPFS node.

Any gateway will do, but the gateway where content is pinned is of course faster. Examples of accessing the previous (before this post) home page of this blog:

http://decentralized.blog:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/
https://gateway.ipfs.io/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/
http://localhost:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/ (if you have a local IPFS node running)

So while that works, it has a couple of problems:

when the content of the home page changes these URL's still point to old content
the hash is not very human-friendly (a.k.a. ugly)
it introduced an element of centralization (sort of, there is no dependence on a particular single gateway)

IPNS

To solve the first of these three problems we can, of course, use IPNS. Any time the content changes we publish the latest content hash like:

ipfs name publish QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
Published to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV

Now we can access the home page through any gateway but with the mutable IPNS link: http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/

Much better, but still ugly...

Note: IPNS is still a bit shaky and forgets published names after about 12 hours. You might want to run a cron job to republish every 8 hours or so.

DNS TXT IPNS Records

Now, let's get rid of the hash. The white paper mentions the use of a DNS TXT record. This is what Wikipedia has to say about it:

A TXT record (short for text record) is a type of resource record in the Domain Name System (DNS) used to provide the ability to associate some arbitrary and unformatted text with a host or other name, such as human readable information about a server, network, data center, and other accounting information.

So the owner of a domain can associate any data to this domain by adding a TXT record. It is like a key-value store where only the owner of the domain can write, and the world can read.

This is used by IPNS to look up the /ipns/<peerID> path if you insert a valid domain as the path.

This is how IPNS normally resolves:

# Publish
ipfs name publish QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV
Published to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV

# Normally IPFS resolves the peerID hash
$ ipfs name resolve /ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN
/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV

To be able to resolve a domain we need to add a dnslink to the TXT record for that domain.

After doing so for the domain name of this blog (decentralized.blog) pointing to the /ipfs link

$ dig txt decentralized.blog
.
.
;; ANSWER SECTION:
decentralized.blog. 900 IN  TXT "dnslink=/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV"

we can resolve by domain name

$ ipfs name resolve decentralized.blog
/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV

which allows us to use a much nicer address:

/ipns/decentralized.blog

This works, but every time I change the content, I have to change the DNS TXT record and wait for propagation. The cool thing is that I can make the DNS TXT record point to a /ipns link, and simply do a ipfs name publish when content has changed:

$ dig TXT decentralized.blog
.
.
;; ANSWER SECTION:
decentralized.blog. 900 IN  TXT "dnslink=/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN"

# points to /ipns path now

Note, I should have done this in the first place, but I was led on the wrong track by this example in the white paper:

# this DNS TXT record
ipfs.benet.ai. TXT "ipfs=XLF2ipQ4jD3U ..."
# behaves as symlink
ln -s /ipns/XLF2ipQ4jD3U /ipns/fs.benet.ai

That is probably a typo, although the ipfs.io domain also points to an immutable /ipfs path for some reason:

$ dig TXT ipfs.io
.
.
;; ANSWER SECTION:
ipfs.io.        60  IN  TXT "dnslink=/ipfs/QmPCawMTd7csXKf7QVr2B1QRDZxdPeWxtE4EpkDRYtJWty"

More about how IPFS resolves paths here.

So now we have a human-friendly URL but at the cost of adding more centralization (the DNS).

Browser extensions

I was hoping there was a browser extension that allowed me to browse a web of /ipfs and /ipns paths, but the extensions that do exist are not made for that purpose.

Because it is still interesting, I'll mention it anyway. The most up to date extension is ipfs-companion. There are browser extensions available for Firefox and Chrome.

One of the cool things it does is helping to distribute data by redirecting requests to the public gateway to your local node.

Another exciting project is the Beaker browser that used to support IPFS, but not anymore. It is a browser for distributed content based on Dat.

Namecoin

What is Namecoin?

Namecoin is a very interesting protocol based on blockchain technology. I won't go into it here in much depth, because I want to start exploring the mother of all blockchains first (i.e., Bitcoin, but you knew that).

I just realized, this is episode 6 of Catching the Blockchain Train, and we haven't seen a single blockchain yet! And worst, we are not going into that here either... Please bear with me, and we will get there eventually.

So, Namecoin. This is what they say about themselves on the website:

Namecoin is an experimental open-source technology which improves decentralization, security, censorship resistance, privacy, and speed of certain components of the Internet infrastructure such as DNS and identities.

(For the technically minded, Namecoin is a key/value pair registration and transfer system based on the Bitcoin technology.)

Bitcoin frees money – Namecoin frees DNS, identities, and other technologies.

The part about DNS might be able to help us here. Namecoin facilitates the .bit TLD, which has its own portal.

Getting a .bit domain

To get started we need to get hold of a .bit name. I rented one a while ago at an exchange, but I now see that there is a better way: using a Namecoin client. If you go the exchange way, realize that you don't have the private keys to control the domain (change value in the KV pair), so you don't own it.

To get your own .bit domain name, follow the steps at Register and Configure .bit Domains.

You'll see that you need a bit of namecoin (NMC) to get your .bit domain, but mining is not really an option (it is done alongside Bitcoin mining and hard), but you can also buy some. Here is a list of exchanges where you can buy NMC, Namecoin Markets, but I'd recommend ShapeShift if you already own some bitcoin or altcoin. You need 0.02 NMC which is about nothing. You can use this namecoin explorer to see if your desired name is still available.

How a .bit lookup works

.bit domains have a unix-path equivalent like this: d/somedotbitname. These paths are the key of a record stored in the Namecoin's blockchain. So to see what the value is you have to have access to a Namecoin node, and ask for the value of the record with key d/somedotbitname.

That value can be anything, but in practice, it is either a reference to nameservers (standard DNS) or directly to an IP address, or to a Bitmessage address. Others add their email and say the .bit address is for sale.

We need access to a namecoin node to query it, but there are online REST API's (introducing a central point of failure) we can use, like webbtc:

# Bitmessage address value
$ curl http://namecoin.webbtc.com/name/d/mysite.json
{
  "name": "d/mysite",
  "value": "BM-2cUyrUNq91XqdPSKvcHSytuED9nnTazf5r",
  "txid": "78e212c47b41b3ae80413a7064c7bee044f8f2bc27362e9c37d5474b68bfd9e3",
  "address": "NFE3ED8C3BGRXuzjBRD3YGJV6jrmbS5h3Q",
  "expires_in": 23292
}

# IP address value
$ curl http://namecoin.webbtc.com/name/d/dappersoftware.json
{
  "name": "d/dappersoftware",
  "value": "{\"ip\":\"207.111.216.146\",\"map\":{\"*\":{\"ip\":\"207.111.216.146\"}}}",
  "txid": "da72c6b0cf85c94d7b2d4f2f8534d1cdafa47ca94fd00489bf205f00c3fc1cb2",
  "address": "N2hrunEcP3PNxdKvihMKCoEnvLUcCcBhKw",
  "expires_in": 4344
}

How browsing .bit websites works

Obviously the normal DNS system doesn't know what to do with a .bit domain, so in a browser that won't resolve to anything. The Namecoin team provides a layer on top op the namecoin node to be able to browse .bit domains: ncdns. I'll leave it at that since we are diverging from our goal...

How do we resolve a .bit domain to an IPFS hash?

What are we trying to do again? So much exciting stuff is going on that we almost got lost in the shapeshiftbitmessagewebbtcncdnschain. Focus!

The goal of this exploration is to be able to pass around a human-friendly address like dweb.bit to bring up an IPFS-powered website. That would be possible if the IPNS system knew how to read the value of dweb.bit in the Namecoin blockchain and interpret it and resolve it to an IPFS address.

It's just like we saw with the DNS TXT record trick, but now the IPFS node would need to have access to the Namecoin blockchain (or a centralized public API like webbtc we saw before).

So: /ipns/dweb.bit needs to resolve to /ipfs/<some multihash>. The value field of a .bit record can be anything, so that is easy to solve. The other part would be that the IPNS resolver implementation needs to recognize .bit domains and access the Namecoin network somehow. This was not implemented as can be easily shown:

$ ipfs name resolve dweb.bit
Error: Could not resolve name.

The IPFS-Namecoin integration is discussed here. Personally, I would start by providing a central .bit lookup service on ipfs.io, and extend that later with a Namecoin resolver implementation in the ipfs code. It doesn't look like it has priority now in the IPFS project though.

In any case, I'm ready: dweb.bit points to { "ipns": "/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN" }.

Blockstack ID

Another contestant for the naming layer that is mentioned in Juan Benet's presentations is Blockstack.

What is Blockstack?

This is what Wikipedia has to say about it:

Blockstack is the first implementation of a decentralized DNS system on top of the Bitcoin blockchain. It combines DNS functionality with public key infrastructure and is primarily meant to be used by new blockchain applications.

That sounds very similar to what Namecoin does, but it uses the Bitcoin blockchain instead of its own fork. In the past, it used the Namecoin blockchain, but the idea is that it should be blockchain agnostic.

On the homepage of Blockstack a bigger goal is stated:

A New Internet for Decentralized Apps

Blockstack is a new decentralized internet where users own their data and apps run locally. A browser portal is all that’s needed to get started.

But for now, let's limit our focus to the decentralized DNS system.

How to register a Blockstack ID

The naming system is called Blockstack ID, and the TLD is .id. You can get your own .id at onename or use the terminal as described here: Blockstack CLI.

Let's try that:

# First install blockstack with pip (Python2 only, booh!)

# Create a wallet; not mentioned in the documentation, but required to do first
$ blockstack setup 

# Check price for the desired name, the shorter the name, the more expensive
$ blockstack price dweb.id
{
    "name_price": {
        "btc": 0.016,
        "satoshis": 1600000
    },
    "preorder_tx_fee": {
        "btc": 0.00219904,
        "satoshis": 219904
    },
    "register_tx_fee": {
        "btc": 0.0020583,
        "satoshis": 205830
    },
    "total_estimated_cost": {
        "btc": 0.02213352,
        "satoshis": 2213352
    },
    "total_tx_fees": 613352,
    "update_tx_fee": {
        "btc": 0.00187618,
        "satoshis": 187618
    }
}

# Get a bitcoin address to pay the fee to
$ blockstack deposit
{
    "address": "3BmAHjCLgELuRdg3jM1MMgm5twXPWWxr7s",
    "message": "Send bitcoins to the address specified."
}

# Start the API server; also not documented, but the error on register is helpful
$ blockstack api start
{
    "status": true
}

# Now register a name
$ blockstack register dweb.id
Calculating total registration costs for dweb.id...
Registering dweb.id will cost about 0.02213352 BTC.
Use `blockstack price dweb.id` for a cost breakdown

The entire process takes 48 confirmations, or about 5 hours.
You need to have Internet access during this time period, so
this program can send the right transactions at the right
times.

Continue? (y/N): y
{
    "message": "Name queued for registration.  The process takes several hours.  You can check the status with `blockstack info`.",
    "success": true,
    "transaction_hash": "c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315"
}

# Check the status
$ blockstack info
{
    "cli_version": "0.14.4.2",
    "consensus_hash": "47f015a3959a80e7ce9c5fb425833860",
    "last_block_processed": 482064,
    "last_block_seen": 482070,
    "queues": {
        "preorder": [
            {
                "confirmations": 0,
                "name": "dweb.id",
                "tx_hash": "c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315"
            }
        ]
    },
    "server_alive": true,
    "server_host": "node.blockstack.org",
    "server_port": 6264,
    "server_version": "0.14.4.0"
}

This is a Bitcoin transaction which can be inspected here c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315.

About 5 hours later the registration was completed:

# Check if we own the dweb.id name now
$ blockstack names
{
    "addresses": [
        {
            "address": "3PR8Js7pLt5Bagk5MZHffXoR43EpQrJHxT",
            "names_owned": [
                "pors.id",
                "dweb.id"
            ]
        }
    ],
    "names_owned": [
        "error"
    ]
}
# Hmm, sort of I guess :)

All in all, it seems more like an identity thing, but the .id domains can also be used as a DNS record (BNS in this case). Blockstack also introduced namespaces, the bit after the dot (now id), which is obviously expensive.

How a .id lookup works

There is a command for lookups in the client:

$ blockstack lookup timblee.id
{
    "profile": {
        "@type": "Person",
        "account": [
            {
                "@type": "Account",
                "identifier": "timbl",
                "proofType": "http",
                "proofUrl": "https://gist.github.com/timbl/04e8ac7c81cd2dee2f51a5e8c672188d",
                "service": "github"
            },
            {
                "@type": "Account",
                "identifier": "timberners_lee",
                "proofType": "http",
                "proofUrl": "https://twitter.com/timberners_lee/status/740677355950080001",
                "service": "twitter"
            }
        ],
        "image": [
            {
                "@type": "ImageObject",
                "contentUrl": "https://s3.amazonaws.com/97p/lUU.jpeg",
                "name": "cover"
            }
        ]
    },
    "zonefile": "$ORIGIN timblee.id\n$TTL 3600\n_http._tcp URI 10 1 \"https://blockstack.s3.amazonaws.com/timblee.id\"\n"
}

The zonefile bit is the interesting part in our use case; we can store routing information in there.

Also, there is an API (it seems a bit shaky, but it works for some accounts): https://core.blockstack.org/v2/users/werner. For pors.id the API doesn't work, but the blockstack explorer does return data for this account.

So how can we use all this?

How do we resolve a .id domain to an IPFS hash?

As said, the zone file entry seems to be the best spot to store routing information. This can be done with the CLI client with the blockstack update command. But to what value do we need to set it? That all depends on what the Blockstack and IPFS teams have agreed on.

A bit of Googling and digging in the Blockstack and IPFS forums didn't give me any answers and my hope was set on this duo-presentation Muneeb Ali & Juan Benet: Blockstack IPFS "CTO Briefing" at CONSTRUCT 2017, but not a word about integration there (at least not the addressing bit).

There is some work about the integration between Blockstack and IPFS as a data storage layer. But that's not what we are looking for here.

So just as with Namecoin, there is no way to use Blockstack as the name layer for IPFS.

But again, I want to be ready when they are so I updated my zonefile:

$ echo '{"ipns":"ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN"}' > new_zone_file.txt
$ blockstack update dweb.id new_zone_file.txt
{
    "message": "Name queued for update.  The process takes ~1 hour.  You can check the status with `blockstack info`.",
    "success": true,
    "transaction_hash": "026c739761ec6ed60119b4d80da2ccf81f274d558f53f377d6862a6020d767b8",
    "zonefile_hash": "4db474dd1dda7502c6152d248b7a24302f6e104a"
}

See the resulting Blockstackified zone file here.

I will revisit Blockstack in depth in a future post, as soon as we got a better understanding of blockchain technology in general.

Ethereum Name Service

We start to see a trend here: very cool blockchain powered name systems, but no support in IPFS. Let's see how ENS fares.

What is ENS?

From the documentation:

ENS is the Ethereum Name Service, a distributed, open, and extensible naming system based on the Ethereum blockchain.

ENS can be used to resolve a wide variety of resources. The initial standard for ENS defines resolution for Ethereum addresses, but the system is extensible by design, allowing more resource types to be resolved in future without the core components of ENS requiring upgrades.

We haven't looked at Ethereum yet, but that will, of course, be the case in an upcoming post (multiple posts most likely). It is next to Bitcoin the most important blockchain around.

In short: Ethereum is building a decentralized virtual machine where you can run applications. From the Ethereum website:

Ethereum is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference.

These apps run on a custom built blockchain, an enormously powerful shared global infrastructure that can move value around and represent the ownership of property. This enables developers to create markets, store registries of debts or promises, move funds in accordance with instructions given long in the past (like a will or a futures contract) and many other things that have not been invented yet, all without a middle man or counterparty risk.

As said, more on that later, for now we just look at the naming system.

To understand the ENS basics, I recommend to:

read ENS documentation introduction and
watch ENS Ethereum Domain Name System, slides here.

The currently available TLD is .eth. Let's try to get one...

How to register a .eth name

The registration for a .eth name is set up as an auction. The details are not really relevant, but you can read more about it here: registrar.ens.domains.

This is also the officail app, but it didn't work for me for some reason. This one works much nicer: myetherwallet. It operates in combination with an Ethereum browser or this browser extension: metamask.

You need some Ether (the Ethereum currency) and a bit patience, and you will be the owner of a .eth address.

Progress can be followed here etherscan.io.

How a .eth lookup works

There is a javascript library that allows you to hook into the ENS system. This code needs to run on the Ethereum virtual machine. Currently, there seems no other way to interact with it.

How do we resolve a .eth domain to an IPFS hash?

Well, we don't. Although IPFS is mentioned once in the ENS documentation (in the intro) there is no code that makes integration possible.

Intermission: now what?

As we have seen so far, there is no (easy) way to use blockchain-based names to address IPFS content. We want to be able to pass around .eth, .id, and .bit addresses and know it will resolve in a trusted way to the correct piece of content. I.e., the content that the owner of such an address wants it to point to.

One way forward that will certainly happen for Ethereum and Blockstack is that IPFS will be used as one of the supported file systems. In that case, the user lives in the Ethereum or Blockstack world and will be able to use .eth or .id addresses.

The alternative, where IPFS itself resolves these addresses is harder because IPFS nodes need trusted access to nodes of the other network. For example, this proof of concept relies on having access to a Namecoin node on the same machine where the IPFS node runs. This is obviously not feasible (who wants to run a node for every naming system that comes out next to its IPFS node?).

A proposal to solve this that I like is pluggable ipns resolvers. I have added my own 2 cents to it, let's see where it leads. But till there is a solution, we still have a couple of contestants, starting with...

Proquint Pronounceable Identifiers

Another round another chance!

From the white paper:

There have always been schemes to encode binary into pronounceable words. IPNS supports Proquint. Thus:
# this proquint phrase
/ipns/dahih-dolij-sozuk-vosah-luvar-fuluh
# will resolve to corresponding
/ipns/KhAwNprxYVxKqpDZ

I'm not sure if this is an improvement, especially because the definition of "pronounceable" is stretched a bit here.

As an aside, this remembers me of my first open source contribution Sort-of-pronounceable password generator back in 2000. There was no github yet, so the source code is gone, but someone put it in a pastebin 11 years later (I found it when I Googled for my own name, ahem). To make sure this historic piece of code never gets lost I added it to IPFS.

Back to Proquint! This is actually implemented in IPFS. So let's give it a try:

$ ipfs name resolve -r /ipns/dahih-dolij-sozuk-vosah-luvar-fuluh
EU�o�C.��M�
# yuk

So yeah that works, but /ipns/KhAwNprxYVxKqpDZ is not something valid.

You can replay it with these online tools:

To encode a valid IPNS peerId is a bit more work to figure out. I'm not interested in that now and also lazy, so I'll leave that to anyone who likes to dive into that :)

Another solution mentioned in the white paper is...

Name shorteners

From the white paper:

Services are bound to spring up that will provide name shortening as a service, offering up their namespaces to users. This is similar to what we see today with DNS and Web URLs:
# User can get a link from
/ipns/shorten.er/foobar
# To her own namespace
/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm

So this is based on the ipns feature it can resolve domains via a DNS TXT record.

Resolving goes like this:

A request for /ipns/shorten.er/foobar is resolved to /ipns/<hash-of-shortener-service-provider>/foobar.
This file /ipns/<hash-of-shortener-service-provider>/foobar points to the peerID address of foobar, like: /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm

Now, this pointing is not as easy as it sounds. This was probably the intended use of what is described in the white paper as Peer Links:

As encouraged by SFS, users can link other users' Objects directly into their own Objects (namespace, home, etc). This has the benefit of also creating a web of trust (and supports the old Certificate Authority model):
# Alice links to bob Bob
ipfs link /<alice-pk-hash>/friends/bob /<bob-pk-hash>
# Eve links to Alice
ipfs link /<eve-pk-hash/friends/alice /<alice-pk-hash>
# Eve also has access to Bob
/<eve-pk-hash/friends/alice/friends/bob
# access Verisign certified domains
/<verisign-pk-hash>/foo.com

Unfortunately, the ipfs link command is not implemented yet which probably explains why no-one implemented an IPFS name shortener.

There is a way to do something similar to peer links with ipfs files, but it is not suitable to build a name resolver. See here for my search for a solution: Are Peer Links supported?.

Again no success here, but there might be hope for the future...

Filecoin

As mentioned in a previous post, Filecoin will be the cryptocurrency to help incentivize the file storage marketplace. This means that there will be a blockchain available in the IPFS network. The primary function of this blockchain is to pay and get paid for storage, but since it's there, it can also be used for other functionality. Like identity or a name system.

In other words, just like Namecoin, Blockstack and ENS provide blockchain enabled identity/naming, Filecoin can do the same. We'll have to see how this all plays out in the next months (years?) but this seems like an attractive option since the IPFS nodes and the Filecoin nodes will most likely be one and the same or at least tightly integrated.

The Filecoin white paper doesn't mention anything about naming, but it is still very early to say anything definite about it.

This is an fascinating introduction Filecoin | White Paper Breakdown and Token Sale Analysis although he gets into a hate speech for a couple of minutes about the ICO (which I think is not completely undeserved).

OK, more than enough about friendly naming I'd say! Time to move on to the next step in our journey to decentralizing this blog (more soon in the next episode...).

Understanding the IPFS White Paper part 2

2017-08-21T17:44:00+02:00

This article is part 5 of the Blockchain train journal, start reading here: Catching the Blockchain Train.

The IPFS White Paper: IPFS Design

The IPFS stack is visualized as follows:

or with more detail:

I borrowed both images from presentations by Juan Benet (the BDFL of IPFS).

The IPFS design in the white paper goes more or less through these layers, bottom-up:

The IPFS Protocol is divided into a stack of sub-protocols responsible for different functionality:

Identities - manage node identity generation and verification.

Network - manages connections to other peers, uses various underlying network protocols. Configurable.

Routing - maintains information to locate specific peers and objects. Responds to both local and remote queries. Defaults to a DHT, but is swappable.

Exchange - a novel block exchange protocol (BitSwap) that governs efficient block distribution. Modelled as a market, weakly incentivizes data replication. Trade Strategies swappable.

Objects - a Merkle DAG of content-addressed immutable objects with links. Used to represent arbitrary data structures, e.g. file hierarchies and communication systems.

Files - versioned file system hierarchy inspired by Git.

Naming - A self-certifying mutable name system.

Here's my alternative naming of these sub-protocols:

Identities: name those nodes
Network: talk to other clients
Routing: announce and find stuff
Exchange: give and take
Objects: organize the data
Files: uh?
Naming: adding mutability

Let's go through them and see if we can increase our understanding of IPFS a bit!

Identities: name those nodes

IPFS is a P2P network of clients; there is no central server. These clients are the nodes of the network and need a way to be identified by the other nodes. If you just number the nodes 1,2,3,... anyone can add a node with an existing ID and claim to be that node. To prevent that some cryptography is needed. IPFS does it like this:

generate a PKI key pair (public + private key)
hash the public key
the resulting hash is the NodeId

All this is done during the init phase of a node: ipfs init > the resulting keys are stored in ~/.ipfs/config and returns the NodeId.

When two nodes start communicating the following happens:

exchange public keys
check if: hash(other.PublicKey) == other.NodeId
if so, we have identified the other node and can e.g. request for data objects
if not, we disconnect from the "fake" node

The actual hashing algorithm is not specified in the white paper, read the note about that here:

Rather than locking the system to a particular set of function choices, IPFS favors self-describing values. Hash digest values are stored in multihash format, which includes a short header specifying the hash function used, and the digest length in bytes.

Example:

\\\

This allows the system to (a) choose the best function for the use case (e.g. stronger security vs faster performance), and (b) evolve as function choices change. Self-describing values allow using different parameter choices compatibly.

These multihashes are part of a whole family of self-describing hashes, and it is brilliant, check it out: multiformats.

Network: talk to other clients

The summary is this: IPFS works on top of any network (see the image above).

Interesting here is the network addressing to connect to a peer. IPFS uses multiaddr formatting for that. You can see it in action when starting a node:

Swarm listening on /ip4/127.0.0.1/tcp/4001

Swarm listening on /ip4/172.17.0.1/tcp/4001

Swarm listening on /ip4/185.24.123.123/tcp/4001

Swarm listening on /ip6/2a02:1234:9:0:21a:4aff:fed4:da32/tcp/4001

Swarm listening on /ip6/::1/tcp/4001

API server listening on /ip4/127.0.0.1/tcp/5001

Gateway (read-only) server listening on /ip4/0.0.0.0/tcp/8080

Routing: announce and find stuff

The routing layer is based on a DHT, as discussed in the previous episode, and its purpose is to:

announce that this node has some data (a block as discussed in the next chapter), or
find which nodes have some specific data (by referring to the multihash of a block), and
if the data is small enough (=< 1KB) the DHT stores the data as its value.

The command line interface and API don't expose the complete routing interface as specified in the white paper. What does work:

# tell the DHT we have this specific content:
$ ipfs dht provide QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG

# ask for peers who have the content:
$ ipfs dht findprovs QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG
QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
QmczCvhy6unZEVC5ukR3BC3BPxYie4jBeRApTUKq97ZnEo
QmPM3WzZ3q1SR3pzXuGPHD7e6z3eWEWCnAvrNw7Wegyc8o
QmPKSqtkDmYsSxQbqNkZG1AmVnZSFyv5WF7fFE2YebNBFG
QmPMJ3HLLLBtW53R2YixhVHmVb3pkR2bHni3qbqq23vBSv
QmPNHJJphV1TB6Z99L6r9Y7bKaCBUQ67X17hicnEmsgWDJ
QmPNhiqGg81o2Perk2i7VNvvVuuLLUMKDxMNwVauP8r5Yv
QmPQJRgP3Vxi52Ho7HfnYdiCRJTRM1TXwgEnyjcwcLuKfb
QmNNxr1ZoyPbwNe2CvYz1CVyvSNWsE8WNwDWQ9t9BDjnj5
QmNT744VjtRFpDYB25EVLx7ha1zAVDKsd3qFjxfQLjPEXq
QmNWwGRWTYeut6qvKDhJBuEJZnbqMPMfuF81MPvHvPBX89
QmNZM5NmzZNPkvH2kPXDYNAB1cAeBNfxLyM9B1crgt3VeJ
QmNZRDzSJybdf4rmt972SH4U9TF6sEK8q2NSEJpEt7SkTp
QmNZdBUV9QXytVcPjcYM8i9AG22G2qwjZmh4ZwpJs9KvXi
QmNbSJ9okrwMphfjudiXVeE7QWkJiEe4JHHiKT8L4Pv7z5
QmNdqMkVqLTsJWj7Ja3oKwLNWcAYUkRjSZPg22B7rvKFMr
QmNfyHTzAetJGBFTRkXXHe5om13Qj4LLjd9SDwJ87T6vCK
QmNmrRTP5sJMUkobujpVXzzjpLACBTzf9weND6prUjdstW
QmNkGG9EZrq699KnjbENARLUg3HwRBC7nkojnmYY8joBXL
QmP6CHbxjvu5dxdJLGNmDZATdu3TizkRZ6cD9TUQsn4oxY

# Get all multiaddr's for a peer
$ ipfs dht findpeer QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
/ip4/192.168.1.14/tcp/4001
/ip6/::1/tcp/4001
/ip4/127.0.0.1/tcp/4001
/ip4/1.2.3.4/tcp/37665

ipfs put and ipfs get only work for ipns records in the API. Maybe storing small data on the DHT itself was not implemented (yet)?

Exchange: give and take

Data is broken up into blocks, and the exchange layer is responsible for distributing these blocks. It looks like BitTorrent, but it's different, so the protocol warrants its own name: BitSwap.

The main difference is that wherein BitTorrent blocks are traded with peers looking for blocks of the same file (torrent swarm), in BitSwap blocks are traded cross-file. So one big swarm for all IPFS data.

BitSwap is modeled as a marketplace that incentivizes data replication. The way this is implemented is called the BitSwap Strategy, and the white paper describes a feasible strategy and also states that the strategy can be replaced by another strategy. One such a bartering system can be based on a virtual currency, which is where FileCoin comes in.

Of course, each node can decide on its own strategy, so the generally used strategy must be resilient against abuse. When most nodes are set up to have some fair way of bartering it will work something like this:

when peers connect, they exchange which blocks they have (have_list) and which blocks they are looking for (want_list)
to decide if a node will actually share data, it will apply its BitSwap Strategy
this strategy is based on previous data exchanges between these two peers
when peers exchange blocks they keep track of the amount of data they share (builds credit) and the amount of data they receive (builds debt)
this accounting between two peers is kept track of in the BitSwap Ledger
if a peer has credit (shared more than received), our node will send the requested block
if a peer has debt, our node will share or not share, depending on a deterministic function where the chance of sharing becomes smaller when the debt is bigger
a data exchange always starts with the exchange of the ledger, if it is not identical our node disconnects

So this is set up kind of cool I think: game theory in action! The white paper further describes some edge cases like what to do if I have no blocks to barter with? The answer is simply to collect blocks that your peers are looking for, so you have something to trade.

Now let's have a look how we can poke around in the innards of the BitSwap protocol.

The command-line interface has a section blocks and a section bitswap; those sound relevant :)

To see bitswap in action, I'm going to request a large file Qmdsrpg2oXZTWGjat98VgpFQb5u1Vdw5Gun2rgQ2Xhxa2t which is a video (download it to see what video!):

# ask for the file
$ ipfs get Qmdsrpg2oXZTWGjat98VgpFQb5u1Vdw5Gun2rgQ2Xhxa2t

# in a seperate terminal, after requesting the file, I inspect the "bitswap wantlist"
$ ipfs bitswap wantlist
QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff
QmUmDEBm9a8MYyqRdb3YQnoqPmqAo4cEWdKQErirFJdSWD
QmY5VJPbsRZzFCTMrFBx2qtZiyyeLhsjBysyfC1fx2gE9S
QmdbzYgyhqUNCNL8xU2HTSKwao1ck2Gmi5U1ygjQuJd92b
QmbZDe5Dcv9mJr8fiqp5aJL2cbyu64tgzwCS2Vy4P3krCL
QmRjzMzVeYRE5b6tDF3sTXMV1sTffno92uL3WwuFavBrWQ
QmPavzEJQw8atvErXQis6C6GF7DRFbb95doAaFkHe9M38u
QmY9fs1Pkr3nV7RkbGdfGh3q8HuKtMMCCUp22AAbwPYnrS
QmUtxZkuJuyydd124Z2cfx6jXMAMpcXZRF96QMAsXc2y6c
QmbYDTJkmLqMm6ojdL6pLP7C8mMVfVPnUxn3yp8HzXDcXf
QmbW9MZ7cwn8svpixosAuC7GQmUXDTZRuxJ8dJp6HyJzCS
QmdCLGWsYQFhi9y3BmkhUreX2S799iWGyJqvnbK9dzB55c
Qmc7EvnBPf2mPCUCfvjcsaQGLEakBbUN9iycnyrLF3b2or
Qmd1mNnDQPf1BAjFqDHjiLe4g4ZFPAheQCniYkbQPosjDE
QmPip8XzQhJFd487WWw7D8aBuGLwXtohciPtUDSnxpvMFR
QmZn5NAPEDtptMb3ybaMEdcVaoxWHs7rKQ4H5UBcyHiqTZ
.
.
.

# find a node where we have debt
$ ipfs dht findprovs Qmdsrpg2oXZTWGjat98VgpFQb5u1Vdw5Gun2rgQ2Xhxa2t
QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z
QmUh2KnjAvgEbJFSd5JZws4CNvt6LbC4C1sRpBgCbZQiqD
Qmc9pBLfKSwWboKHMvmKx1P7Z738CojuUXkPA1dsPrvSw2
QmZFhGyS2W833nKKkbqZAU2uSvBbWUytDJkKBHimwRmhd6
QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic
Qmbut9Ywz9YEDrz8ySBSgWyJk41Uvm2QJPhwDJzJyGFsD6

# try one to see if we have downloaded from that node
$ ipfs bitswap ledger QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
Ledger for <peer.ID SoLMeW>
Debt ratio: 0.000000
Exchanges:  11
Bytes sent: 0
Bytes received: 2883738

Thank you QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3; what a generous peer you are!

Now, have a look at the block commands:

# Let's pick a block from the wantlist above
$ ipfs block stat QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff
Key: QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff
Size: 262158

$ ipfs block get QmYEqofNsPNQEa7yNx93KgDycmrzbFkr5oc3NMKXMxx5ff > slice_of_a_movie
# results in a binary file of 262 KB

We'll have another look at how blocks fit in in the next chapter.

The three layers of the stack we described so far (network, routing, exchange) are implemented in libp2p.

Let's climb up the stack to the core of IPFS...

Objects: organize the data

Now it gets fascinating. You could summarize IPFS as: Distributed, authenticated, hash-linked data structures. These hash-linked data structures are where the Merkle DAG comes in (remember our previous episode?).

To create any data structure, IPFS offers a flexible and powerful solution:

organize the data in a graph, where we call the nodes of the graph objects
these objects can contain data (any sort of data, transparent to IPFS) and/or links to other objects
these links - Merkle Links - are simply the cryptographic hash of the target object

This way of organizing data has a couple of useful properties (quoting from the white paper):

Content Addressing: all content is uniquely identified by its multihash checksum, including links.

Tamper resistance: all content is verified with its checksum. If data is tampered with or corrupted, IPFS detects it.

Deduplication: all objects that hold the exact same content are equal, and only stored once. This is particularly useful with index objects, such as git trees and commits, or common portions of data.

To get a feel for IPFS objects, check out this objects visualization example.

Another nifty feature is the use of unix-style paths, where a Merkle DAG has the structure:

/ipfs/<hash-of-object>/<named-path-to-object

We'll see an example below.

This is really all there is to it. Lets see it in action by replaying some examples from the quick-start:

$ mkdir foo
$ mkdir foo/bar
$ echo "baz" > foo/baz
$ echo "baz" > foo/bar/baz
$ tree foo/
foo/
├── bar
│   └── baz
└── baz
$ ipfs add -r foo
added QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR foo/bar/baz
added QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR foo/baz
added QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN foo/bar
added QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm foo

# the last hash is the root-node, we can access objects through their path starting at the root, like:
$ ipfs cat /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm/bar/baz
baz

# To inspect an object identified by a hash, we do
$ ipfs object get /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm
{  
  "Links":[  
    {  
      "Name":"bar",
      "Hash":"QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN",
      "Size":61
    },
    {  
      "Name":"baz",
      "Hash":"QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR",
      "Size":12
    }
  ],
  "Data":"\u0008\u0001"
}

# The above object has no data (except the mysterious \u0008\u0001) and two links

# If you're just interested in the links, use "refs":
$ ipfs refs QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm
QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN
QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR

# Now a leaf object without links
$ ipfs object get /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm/bar/baz
{  
  "Links":[  

  ],
  "Data":"\u0008\u0002\u0012\u0004baz\n\u0018\u0004"
}

# The string 'baz' is somewhere in there :)

The Unicode characters that show up in the data field are the result of serialization of the data. IPFS uses protobuf for that I think. Correct me if I'm wrong :)

At the time I'm writing this there is an experimental alternative for the ipfs object commands: ipfs dag:

$ ipfs dag get QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm
{  
  "data":"CAE=",
  "links":[  
    {  
      "Cid":{  
        "/":"QmeBpzHngbHes9hoPjfDCmpNHGztkmZFRX4Yp9ftKcXZDN"
      },
      "Name":"bar",
      "Size":61
    },
    {  
      "Cid":{  
        "/":"QmWLdkp93sNxGRjnFHPaYg8tCQ35NBY3XPn6KiETd3Z4WR"
      },
      "Name":"baz",
      "Size":12
    }
  ]
}

$ ipfs dag get /ipfs/QmdcYvbv8FSBfbq1VVSfbjLokVaBYRLKHShpnXu3crd3Gm/bar/baz
{  
  "data":"CAISBGJhegoYBA==",
  "links":[  

  ]
}

We see a couple of differences there, but let's not get into that. Both outputs follow the IPFS object format from the white paper. One interesting bit is the "Cid" that shows up; this refers to the newer Content IDentifier.

Another feature that is mentioned is the possibility to pin objects, which results in storage of these objects in the file system of the local node. The current go implementation of ipfs stores it in a leveldb database under the ~/.ipfs/datastore directory. We have seen pinning in action in a previous post.

The last part of this chapter mentions the availability of object level encryption. This is not implemented yet: status wip (Work in Progress; I had to look it up as well). The project page is here: ipfs keystore proposal.

The ipfs dag command hints to something new...

Intermission: IPLD

If you studied the images at the start of this post carefully, you are probably wondering, what is IPLD and how does it fit in? According to the white paper, it doesn't fit in, as it isn't mentioned at all!

My guess is that IPLD is not mentioned because it was introduced later, but it more or less maps to the Objects chapter in the paper. IPLD is broader, more general, than what the white paper specifies. Hey Juan, update the white paper will ya! :-)

If you don't want to wait for the updated white paper, have a look here: the IPLD website (Inter Planetary Linked Data), the IPLD specs and the IPLD implementations.

And this video is an excellent introduction: Juan Benet: Enter the Merkle Forest.

But if you don't feel like reading/watching more: IPLD is more or less the same as what is described in the "Objects" and "Files" chapters here.

Moving on to the next chapter in the white paper...

Files: uh?

On top of the Merkle DAG objects IPFS defines a Git-like file system with versioning, with the following elements:

blob: there is just data in blobs and it represents the concept of a file in IPFS. No links in blobs
list: lists are also a representation of an IPFS file, but consisting of multiple blobs and/or lists
tree: a collection of blobs, lists and/or trees: acts as a directory
commit: a snapshot of the history in a tree (just like a git commit).

Now I hear you thinking: aren't these blobs, lists, and trees the same things as what we saw in the Mergle DAG? We had objects there with data, with or without links, and nice Unix-like file paths.

I heard you thinking that because I thought the same thing when I arrived at this chapter. After searching around a bit I started to get the feeling that this layer was discarded and IPLD stops at the "objects" layer, and everything on top of that is open to whatever implementation. If an expert is reading this and thinks I have it all wrong: please let me know, and I'll correct it with the new insight.

Now, what about the commit file type? The title of the white paper is "IPFS - Content Addressed, Versioned, P2P File System", but the versioning hasn't been implemented yet it seems.

There is some brainstorming going on about versioning here and here.

That leaves one more layer to go...

Naming: adding mutability

Since links in IPFS are content addressable (a cryptographic hash over the content represents the block or object of content), data is immutable by definition. It can only be replaced by another version of the content, and it, therefore, gets a new "address".

The solution is to create "labels" or "pointers" (just like git branches and tags) to immutable content. These labels can be used to represent the latest version of an object (or graph of objects).

In IPFS this pointer can be created using the Self-Certified Filesystems I described in the previous post. It is named IPNS and works like this:

The root address of a node is /ipns/<NodeId>
The content it points to can be changed by publishing an IPFS object to this address
By publishing, the owner of the node (the person who knows the secret key that was generated with ipfs init) cryptographically signs this "pointer".
This enables other users to verify the authenticity of the object published by the owner.
Just like IPFS paths, IPNS paths also start with a hash, followed by a Unix-like path.
IPNS records are announced and resolved via the DHT.

I already showed the actual execution of the ipfs publish command in the post Getting to know IPFS.

This chapter in the white paper also describes some methods to make addresses more human-friendly, but I'll leave that in store for the next episode which will be hands-on again. We gotta get rid of these hashes in the addresses and make it all work nicely in our good old browsers: Ten terrible attempts to make IPFS human-friendly

Let me know what you think of this post by tweeting to me @pors!

Understanding the IPFS White Paper part 1

2017-08-10T11:11:00+02:00

This article is part 4 of the Blockchain train journal, start reading here: Catching the Blockchain Train.

Oh no, a White Paper!

The crypto currency/blockchain world loves its white papers, and IPFS is no exception. It started with the famous Bitcoin: A Peer-to-Peer Electronic Cash System white paper by Satoshi Nakamoto (which in turn references to another white paper you don't want to know about). We will have a look at the Bitcoin white paper in a future post when we dive into crypto currencies.

Two things you need to know about white papers:

They are distributed in PDF format (because it is harder to change than HTML!)
white papers are hard, scary and contain at least one mathematical formula.

The IPFS white paper is here: IPFS - Content Addressed, Versioned, P2P File System (DRAFT 3). And, as it should, hosted on IPFS.

To understand how IPFS works it's a good idea to walk through this white paper step by step. I'll limit this post to chapter 2 of the white paper, having a look at the underlying technologies:

Distributed Hash Tables
Block Exchanges - BitTorrent
Version Control Systems - Git
Self-Certified Filesystems - SFS

Mapped on a ISO like stack:

OK, here we go!

Distributed Hash Tables (DHT)

What is a DHT?

A DHT is like a Python dict or Perl hash (if you have the key, you can retrieve the value), but the data is distributed over multiple nodes. The Wikipedia article Distributed hash table gives a good introduction.

In the case of IPFS, the key is a hash over the content. So ask an IPFS node for the content with hash QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ and the IPFS node will lookup in the DHT which nodes have the content.

How the specific value is found efficiently (fast, with a little as possible network requests) and how to manage the DHT so that changes (nodes that enter/leave the network, or new entries to the table) are absorbed easily is different for the various DHT implementations that exist.

One such an implementation is called Pastry, and I like these two videos where the routing (how to find values) and the dynamics (how to handle node additions/removals) are explained really well.

Kademlia and friends

Back to the white paper. Three DHT implementations are mentioned explicitly, starting with Kademlia. Kademlia has a white paper of its own but let's not go there...

Kademlia is the DHT protocol that is used in almost all popular P2P systems and again the Wikipedia article on Kedemlia is a great introduction.

In short, Kedemlia uses the ID of the nodes to get step by step closer to the node with the desired hash (from the Wikipedia article):

When searching for some value, the algorithm needs to know the associated key and explores the network in several steps. Each step will find nodes that are closer to the key until the contacted node returns the value or no more closer nodes are found. This is very efficient: Like many other DHTs, Kademlia contacts only O(log(n)) nodes during the search out of a total of n nodes in the system.

The details can start to get quite complicated and I don't think it won't add to our goal of understanding IPFS, so we move on. If you like this sort of thing: this is a nice introduction and if you want to dig deeper the link to the Kademlia white paper is ^^^.

IPFS mentions two other DHT implementations that are added to standard Kedemlia:

Coral DSHT: this improves the lookup performance and decreases resource use.
S/Kademlia: makes Kademlia more resistant against malicious attacks.

The IPFS DHT in practice

The DHT is used in IPFS for routing, in other words:

to announce added data to the network
and help locate data that is requested by any node.

The white paper states:

Small values (equal to or less than 1KB) are stored directly on the DHT. For values larger, the DHT stores references, which are the NodeIds of peers who can serve the block.

So let's have a look if we can access the DHT directly and add and retrieve small data blocks.

$ ipfs daemon # make sure this runs (and install/setup ipfs first)

$ echo 'my tiny text' | ipfs add # add content to the node, smaller than 1KB
QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF

$ ipfs cat QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF # just check
my tiny text

$ ipfs dht get /ipfs/QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF
# returns not found

So now we have that text there we would like to access it from the DHT, but apparently, the only ipfs dht get request that is supported is for keys starting with /ipns/.

OK, and so we create an IPNS and see if we can query that directly:

$ ipfs name publish QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF # point a IPNS address to our content
Published to QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6: /ipfs/QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF

$ ipfs resolve QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6 # check it
# never returns, hmm IPNS doesn't seem to be ready for production

$ ipfs dht get /ipns/QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6 # same thing but directly
# does return binary data starting with 4/ipfs/QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF

It sort of does work, but it is limited to IPNS for now it seems.

What else can we do with the DHT?

Use the DHT to find out which nodes can provide some data:

$ ipfs dht findprovs QmfQKcXMLGvCxx6opNDwb1ptD1LJER6MPHdsMHCB1CXpFF
QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN

# ^^^ there are both my nodes, so that works
# Now ask the DHT for the address of the first peer that was returned
$ ipfs dht findpeer QmYebHWdWStasXWZQiXuFacckKC33HTbicXPkdSi5Yfpz6
/ip4/176.92.234.78/tcp/4001
/ip4/85.74.239.218/tcp/38689
/ip4/127.0.0.1/tcp/4001
/ip4/192.168.1.30/tcp/4001
/ip6/::1/tcp/4001
/ip4/192.168.1.3/tcp/4001
/ip4/176.92.234.78/tcp/40443

# Oooh nice! We will visit this address notation in the next episode.

All in all, we saw a bit of the DHT in action, but not as intuitive and functional as I hoped for.

Time to move on to the next exciting technology applied by IPFS:

BitTorrent

How does BitTorrent work?

We all know about BitTorrent, but some of us (yeah, me too) need to dig deeper to really get it. This presentation is a great introduction: Feross Aboukhadijeh: WebTorrent - JSConf.Asia 2014. The speaker talks about how he implemented a BitTorrent client that can run in the browser, thanks to WebRTC (which is a cool technology that might come in handy later on for our project). It also explains the working of the DHT again to drive it home.

BitTorrent and IPFS

The exchange of data (blocks) in IPFS is inspired by BitTorrent but is not 100% BitTorrent. The white paper mentions two BitTorrent features that IPFS uses:

tit-for-tat strategy (if you don't share, you won't receive either)
get rare pieces first (improves performance and more, see the first PDF above)

A notable difference is that where in BitTorrent each file has a separate swarm of peers (forming a P2P network with each other) where IPFS is one big swarm of peers for all data. The IPFS BitTorrent variety is called BitSwap and I'll go over that in the next episode.

Let's try if we can see something of the swarm in action.

# Make sure you have the daemon running
$ ipfs swarm peers
/ip4/104.131.131.82/tcp/4001/ipfs/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
/ip4/104.236.151.122/tcp/4001/ipfs/QmSoLju6m7xTh3DuokvT3886QRYqxAzb1kShaanJgW36yx
/ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
/ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3

When you just start the daemon you get connected to a couple of "seed peers". After a while the number of peers grows rapidly.

Another command to query the swarm:

$ ipfs swarm addrs
QmNRuQrwtGgeVQgndTny4A4Rukg7GR7vzDJrVJxUBfqevk (4)
        /ip4/127.0.0.1/tcp/4001
        /ip4/172.31.41.39/tcp/4001
        /ip4/35.167.26.28/tcp/4001
        /ip6/::1/tcp/4001
QmNSJohkvvBVmHeN7KUZnm4X84GA6Znbv6ZmvsTAjbw3AB (5)
        /ip4/10.0.1.8/tcp/4001
        /ip4/127.0.0.1/tcp/4001
        /ip4/174.44.163.74/tcp/16012
        /ip6/::1/tcp/4001
        /ip6/fd15:462e:f8fd:695e:9161:27dd:7f78:d242/tcp/4001
QmNTJyhCYcbv5GdnqtdEwTfJCgY6pV4PJiNTtAmuoxnQak (3)
        /ip4/127.0.0.1/tcp/4001
        /ip4/94.176.232.68/tcp/4001
        /ip6/::1/tcp/4001
QmNTZy7TfXvsHczwwV3EYbxRZN5gthgicG9GiroD7C4ZrP (4)
        /ip4/127.0.0.1/tcp/4001
        /ip4/172.20.255.127/tcp/4001
        /ip4/54.229.227.53/tcp/4001
        /ip6/::1/tcp/4001
        .
        .
        .

These are the addresses in the swarm the node knows of, the hash on top is the peerId.

With this information, you can connect to a peer like

$ ipfs swarm connect /ip4/114.91.202.180/tcp/34746/ipfs/QmfTgdg6GkqJtUrWAYo69GjcLrjQq9LjTjgW3KZ1ux1X6U
connect QmfTgdg6GkqJtUrWAYo69GjcLrjQq9LjTjgW3KZ1ux1X6U success

More about BitSwap later...

Let's take a break

Pfew, that was a lot of theory and dry material, and we are not even halfway! So to let it all sink in and to free up the brain for the rest go for a walk in the park and listen to this podcast: The Quiet Master of Cryptocurrency — Nick Szabo. It goes all over the place, but I found it very interesting. It zooms out from bit level to things like "what is money?".

Highly recommended listening, and when you're done...

Version Control Systems - Git

The white paper's entire section on Version Control Systems is reproduced here:

Version Control Systems provide facilities to model files changing over time and distribute different versions efficiently.

The popular version control system Git provides a powerful Merkle DAG object model that captures changes to a filesystem tree in a distributed-friendly way.

Immutable objects represent Files (blob), Directories (tree), and Changes (commit).

Objects are content-addressed, by the cryptographic hash of their contents.

Links to other objects are embedded, forming a Merkle DAG. This provides many useful integrity and workflow properties.

Most versioning metadata (branches, tags, etc.) are simply pointer references, and thus inexpensive to create and update.

Version changes only update references or add objects.

Distributing version changes to other users is simply transferring objects and updating remote references.

Let's step through these lines one by one to understand it:

Ad 1. Immutable objects represent Files (blob), Directories (tree), and Changes (commit).

Git only adds data. So blobs, trees, and commits are immutable. What the last version of either of these is, is determined by special references (see point 4 and 5).

Ad 2. Objects are content-addressed, by the cryptographic hash of their contents.

In git, the file or directory name is not used when they are referenced. Git hashes the content (or listing or commit) with SHA1 and uses these hashes in its database. This article makes it very insightful how that works: Git under the hood (Start reading at "Now let’s look at how Git does all this")

Ad 3. Links to other objects are embedded, forming a Merkle DAG. This provides many useful integrity and workflow properties.

Oh my, a Merkle DAG! No worries, it is actually not as scary as it sounds.

A Merkle tree is a binary tree where the parent contains the hash of the concatenation of the hashes of the two children. This explains the integrity property: any change in a data block results in a change of the root node. With just a little bit of meta-data (uncles and parents, which can be untrusted) and a trusted root node, we can verify the validity of the block.

Now, a Merkle DAG is not the same as a Merkle Tree. The difference is explained here: Merkle DAG and JRFC 20 - Merkle DAG

In short: the Merkle DAG is more general as it is not a binary tree but a graph, and any node can contain data, not just the leaf nodes as in the Merkle Tree.

It's still all a bit vague, but I'll address it again when we have a look at the IPFS Merkle DAG.

Ad 4. and 5 Most versioning metadata (branches, tags, etc.) are simply pointer references, and thus inexpensive to create and update. Version changes only update references or add objects.

This is visualized here: Git for Computer Scientists where the branch, the HEAD, and tags are simply references to commits.

Ad 6. Distributing version changes to other users is simply transferring objects and updating remote references.

What it says here, no need to explain I think :)

Self-Certified Filesystems - SFS

This is used to implement the IPNS name system for IPFS. It allows us to generate an address for a remote filesystem, where the user can verify the validity of the address.

From the white paper:

SFS introduced a technique for building Self-Certified Filesystems: addressing remote filesystems using the following scheme
/sfs/<Location>:<HostID>
where Location is the server network address, and:
HostID = hash(public_key || Location)
Thus the name of an SFS file system certifies its server.

I think that speaks for itself, we will see it in action in the next episode: Understanding the IPFS White Paper part 2

Let me know what you think of this post by tweeting to me @pors!

Getting to know IPFS

2017-08-03T19:11:00+02:00

This article is part 3 of the Blockchain train journal, start reading here: Catching the Blockchain Train.

Let's read a bit (reading is good for you!)

I already listed this article in the previous episode, but I mention it again because it is a great place to start to understand IPFS and its context: HTTP is obsolete. It's time for the distributed, permanent web.

A couple of things we can pick up from it (a bit simplified):

IPFS consists of a network of peer-to-peer nodes (aka computers that talk to each other directly)
These nodes can store content (any type of file)
Content is represented by a hash and is immutable (if the content changes, so does the hash)
A node can request content from other nodes by using this hash. This is pretty cool: there is a permanent relation between the hash and the content itself. Unlike the current web where the content behind a URL can change.
Nodes can decide to store a copy of any content
The more nodes store a piece of content the harder it is to get rid of it (the permanent web).
The network also gets faster that way, similar to bittorrent getting faster when the number of seeders goes up (IPFS is partly based on the bittorrent protocol, but one of the differences is that it prevents duplicate pieces of identical content)

A bit more detail:

Content (files) is broken up into blocks, and each block gets a hash
Duplicate blocks (identical content but different hash) are removed from the network
IPFS has a directory concept (another hash) that points to the hashes of the content inside
Git like version control is used for blocks
To solve the problem of finding the last version of some content is solved by IPNS
IPNS allows anyone to create a unique public link to any content that is out there.
So an IPFS hash points to some immutable content, regardless the version; the IPNS address points to some file or directory determined by the creator of that address
This IPNS address is implemented as a pubkeyhash which is similar to the public address in a Bitcoin wallet.
IPNS hashes are long and ugly. Also, browsers don't speak IPFS. There are a couple of solutions to bridge the gap to the current time that we will look into later.

To read and discover more about IPFS, this page on github is a great starting point ipfs/ipfs. Make sure to check out the quick summary.

OK, it is all still a bit abstract I guess, so time to play.

Terminal time (move those fingers buddy!)

Now we understand the basics lets start playing a bit.

To get up and running and to be able to play along, first install (I recommend the ipfs-update approach) and follow the getting started.

Now it is time to set the first step into making this blog decentralized. BTW what a great domain this site has! Well, as long as you are reading the centralized version :)

I uploaded the statically generated website to my server and it looks like this:

├── archives.html
├── author
│   └── mark-pors.html
├── authors.html
├── catching-the-blockchain-train.html
├── categories.html
├── category
│   └── blockchain-train-journal.html
├── feeds
│   ├── all.atom.xml
│   └── blockchain-train-journal.atom.xml
├── getting-to-know-ipfs.html
├── index.html
├── picking-a-decentralized-storage-system.html
├── tags.html
└── theme
    ├── css
     ... // ...
            └── youtube.png

To shoot a file into the interplanetary file system, we simply do:

$ ipfs init # this creates your node's peer ID
$ ipfs daemon # start your local node
$ ipfs add catching-the-blockchain-train.html
added QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4 catching-the-blockchain-train.html

Now my node can serve this file and tell the network there is a piece of content (my first blog post!) available, just in case someone requests this hash: QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4.

So, in your terminal:

ipfs cat QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4

should return the same as (for as long as I host it over HTTP of course):

curl http://decentralized.blog/catching-the-blockchain-train.html

Notice that when you request the file for the second time through ipfs, it gets it very fast. That is because your node now has a copy of the content locally. Pretty cool huh?

The ipfs node comes with an HTTP gateway, so we can also access the file in a browser: http://decentralized.blog:8080/ipfs/QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4

This is what happened here (from the getting started document):

The gateway served a file from your computer. The gateway queried the DHT, found your machine, requested the file, your machine sent it to the gateway, and the gateway sent it to your browser.

Note: this obviously introduces a central server into the picture, but at least anyone can run such a gateway. So it is still sort-of-decentralized. We have to look for something better later on.

Decentralize it! At last some progress

To push the whole website onto the interplanetary file system can't be more simple, just add the -r flag and point to the directory containing all static files.

$ ipfs add -r ../../www/dcb
added QmWu4hsywXoSrw5JRhjUadyMnDuxuvcgExwVD62cWeLVjb dcb/archives.html
added QmUCJ6z2EfikXEDUYbxfjdkrX2zw62XCAM93HHC61Qrmwg dcb/author/mark-pors.html
added QmV13vUiKU64oHoxbp5MN7bnehraW46de82PLvndbisn64 dcb/authors.html
added QmXyZcrThrfWQSTKzPiNT4Nd2RcqcVQ3tr7rmFqHYZ3fq4 dcb/catching-the-blockchain-train.html
added QmRiWHfYLQEJoKz2dDq2sXnW2keiKZG1nHbnaN55TTXN8b dcb/categories.html
added QmSCx2eV3BjHJWoXMBc7rKakPv3fAcewyroFCRmovcsYSy dcb/category/blockchain-train-journal.html
added QmVxVLT22KLGasBLxzSv6yXofx3GVXmWEd3HqSjPp1X6Zz dcb/feeds/all.atom.xml
added QmdKJY5VVd8Qm5xsLt8Vj5XwvTbe5mCQhK6DnZ9LkqTHwz dcb/feeds/blockchain-train-journal.atom.xml
added QmYpyQ3MXYZum3q2E9pRxUWnJXK74cLvFbuMDYukK9ivK1 dcb/index.html
added QmXkfv9ZXb9evxPXTMJC8fvgeZymYpTQo3CVFv7mDoSRTN dcb/picking-a-decentralized-storage-system.html
added QmSyuSjmzhwkoj6qH7s4MmKeB98Ub1DZ1Hm5anBBnR9yVu dcb/tags.html
added QmSfLETTMQc1b9cBJENH1S4NK5qniXSf79KSP8h5CbU3U8 dcb/theme/css/main.css
... // ...
added QmU8CmEYWV5aUuqoToq1xiyqftdY8MHQ8MdViA2ex23uyV dcb/theme/images/icons/youtube.png
added QmYBVeJkBRdJYj8jr3YTGWE537h26YqcfjvbX4ttjZ65X2 dcb/author
added QmPwQPX86RQi6MrZQigh2tWgztcbf46EjhRg1hzMzELvHN dcb/category
added QmfRxCPp417G3NTqEpBmwtGwtUKFzgQjBRvxNBq88HhC9F dcb/feeds
added QmYUnPcDPcnKA3WFembvNYG48m1oUWa7QLNWZxrkLv1vcE dcb/theme/css
added QmfBq8BDXxhu9cLAG4qhxqY6K5xDQCBWyoYqX5fTmQbfwW dcb/theme/images/icons
added Qme4Dt6vQ1eAVcEs7dY55cAMhTsRJxTakANhZjX9EKvGfF dcb/theme/images
added QmciTMAUiEqMpar2u9n1dJNJStmrP6fjMESyH6ZS5eo56H dcb/theme
added QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ dcb

Note: my first try failed when I tried to add a symlink to the network, that doesn't work.

See that the hash for dcb/catching-the-blockchain-train.html is the same as we previously got? That's the de-duplication in action.

Now when we take the bottom hash from this list and feed it to the gateway, we see the blog: http://decentralized.blog:8080/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/. All relative links work just fine; it's beautiful!

In case my gateway is down, just try the one provided by IPFS: https://gateway.ipfs.io/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/ or your local gateway: http://127.0.0.1:8080/ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/. Now is that cool or what?

There is still a small issue: it is the previous version of the blog because I'm still writing this article and when I upload it when I'm done the hash changes, but then the new hash won't be in the article...

There is a solution for that:

A permanent address for this blog with IPNS

The usage of IPNS is explained here: The Inter-Planetary Naming System.

So in our case, on the node that runs on my server, I do:

$ ipfs name publish QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ
Published to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ

This results in an address for the blog that won't change: QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN.

This address points (when I executed the command just right now) to the directory containing the static files (index.html is served when it is in that directory BTW).

When you read this, the address QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN is still the home page address, but it will contain new content (I will do another publish when I'm done writing this post).

So our static blog is now decentralized and available through http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/. Note the ipns part instead of ipfs this time.

IPFS is pretty cool, and it makes our lives much easier. Before we move on to improving our blog I'd like to look a little bit under the hood the understand the workings of IPFS (and IPNS) a bit more...

Read on here: Understanding the IPFS White Paper part 1.

Picking a Decentralized Storage System

2017-08-02T19:11:00+02:00

This article is part 2 of the Blockchain train journal, start reading here: Catching the Blockchain Train.

Some reading material to get started

A couple of articles that I found useful when learning about decentralized storage:

Comparing decentralized storage systems

Our decentralized blog pages are in need of some sort of decentralized hosting. To build that from scratch seems like a lot of work and will probably result in a blockchain train crash.

So let's see what is already out there we could use for our project.

I searched a bit and came up with:

Storj
SIA
MaidSafe
IPFS
ZeroNet
Ethereum Swarm
BigchainDB
did I forget a serious contender? Please let me know.

I’m not yet qualified to compare these technologies (still trying to catch the train!), but I’ll add some observations for each:

Storj

From the home page:

Blockchain-based, end-to-end encrypted, distributed object storage, where only you have access to your data.

That doesn't sound like something that is handy for a blog, where we want content to be public.

Storj looks like a great product though, it is a bit like Resilio Sync, but with a blockchain added to create a marketplace for buying or renting out disk space on the network.

SIA

SIA is another Dropbox killer, more or less the same as Storj.

MaidSafe

Where Storj and SIA pitch their service as a faster and cheaper way to store your data, Maidsafe promotes their SAFE network by scaring you. Cloud providers, government, and hackers can not be trusted is the message in their video. Apart from the funny name, it is very similar to the two services above.

IPFS

Now, this is a different beast. The first three services are incentive platforms, which means they have their own cryptocurrency to support their marketplace. IPFS also has such a thing, FileCoin, but it is neatly split from IPFS which is more like a decentralized storage protocol.

IPFS stands for interplanetary file system, but they hide the original name from their homepage. Too funny for a serious business!

On the home page it says:

IPFS is the Distributed Web

A peer-to-peer hypermedia protocol to make the web faster, safer, and more open.

IPFS seems like an great option for our project.

ZeroNet

ZeroNet doesn't have a cheesy video on their home page, so I had to actually read. However, their awesome presentation is 100x better than the four videos from the others combined: ZeroNet preso.

They also seem to operate a lot less like a serious business compared to the others, showing more of the idealistic open source spirit.

On the home page:

Open, free and uncensorable websites, using Bitcoin cryptography and BitTorrent network

Oh, oh, that is what we are trying to do here! A competitor! Nah, let's just keep an eye on the code, maybe we can learn a thing or two.

Ethereum Swarm

On our trip to become blockchain developers, we have no other choice than to dive into Ethereum. We will address that in a future episode. But their decentralized file storage system, named Swarm, must be considered here.

It took me a while to find the source code for this project, but here it is!

From the documentation:

Swarm is a distributed storage platform and content distribution service, a native base layer service of the ethereum web 3 stack. The primary objective of Swarm is to provide a sufficiently decentralized and redundant store of Ethereum’s public record, in particular to store and distribute dapp code and data as well as block chain data.

It seems that Swarm is an incentive platform enforced with Ethereum smart contracts, so not the best choice for our purpose. We gotta stay hands-on here. We want to add a coin later ourselves!

BigchainDB

From their site:

BigchainDB is complementary to decentralized storage, processing and communication building blocks. It can be used side by side with higher-level decentralized computing platforms and applications, and protocols for identity, financial assets, intellectual property and sidechains. BigchainDB fills a gap in the decentralized stack.

This is a brilliant solution, almost too good to be true. It is not just decentralized storage, but a full blown database.

It is built on top of MongoDB and decentralized, so data is immutable, and there is no central authority.

For our current project it might be a bit overkill, but let's keep an eye on it.

The winning decentralized storage system

OK, I’ll be honest, I already picked one before I did this quick research what is around :-)

The winner is IPFS! Yay!

The runner-up we might have a look at at a later stage: BigchainDB!

Even in retrospect, IPFS seems like a good choice: all competitors have their "flaws" in the context of this project. By applying "selection by deduction", only IPFS remains.

After this very scientific approach in picking IPFS, let's hope it is indeed useful for us.

Let's move on to part 3: Getting to know IPFS

Catching the Blockchain Train

2017-08-01T13:34:00+02:00

What is this and for who?

This is a blog for techies who (almost) missed the blockchain train. Hmm, I also need to write a song "the blockchain train," it just sounds so good!

I'm a bit late to the blockchain/Bitcoin party, and when you made it past the previous line I assume you are too!

No worries, there is still plenty of time to add blockchain technology to your skillset. I am confident that what we see happening now is just the very beginning of a decentralized Internet. So even though we feel we are late, we can still catch the blockchain train. Especially because you and I are pretty smart! Right?

My approach to educating myself is two-fold:

First by reading as much as I can about the blockchain, Bitcoin, smart contracts, Ethereum, dapps, ipfs, and all of that.
In parallel, I want to apply what I learn by turning this blog into a decentralized app (dapp) as much as possible. I will work in "naive mode" and start building and using technologies I found on my path, and backtrack (you know, refactor) when I see it was the wrong approach.

The code will, of course, be open source and available here: blockchaintrain.

BTW this is not a tutorial; it's just a journal of my activities.

What I'm reading

I'm reading three books in parallel right now (next to the online article here and there):

Mastering Bitcoin: Programming the Open Blockchain: a thorough technical introduction into the world of Bitcoin. I got stuck halfway when I read it a couple of years ago. This time I mean to finish it.
Blockchain for dummies: yeah, I know...
Decentralized Applications: Harnessing Bitcoin's Blockchain Technology: not very well received, but I like it so far.

I'll keep you updated on other books or articles I read.

Decentralized blog design part 1

The first version of this blog (you're reading it now) is a statically generated blog with Pelican*, and I uploaded the files to my server with SSH.

So my progress towards a fully decentralized blog is 0/100!

Excellent! That means there is still a lot to discover.

In the next article, I will describe how I made a first step: load the content of the blog from a decentralized file system.

*: if you don't like Python, you can use Hugo (Go) or Jekyll (Ruby) or any one of these. Let me know if you follow along with a different site generator, and I'll add it to this blog.

What can you expect in the next episodes?

Using a decentralized file system
From static pages to dynamic content (so from files to a dapp)
Supporting identity
Introduction of colored coins or sidechains or maybe even a mini-ICO
Whatever seems like a useful feature to add...

Wanna join me?

If you want to play along and contribute, please leave a note in the comments! Uhm, as soon as I have figured out how to support comments in a decentralized manner. Till then, tweet to me @pors!

Continue reading here, part 2 of the Blockchain train journal: Picking a Decentralized Storage System.