Setup your own private Proof-of-Authority Ethereum network with Geth

Goal: step by step guide to help you setup a local private ethereum network using the Proof-of-Authority consensus engine (also named clique).

In a nutshell: we will setup two nodes on the same machine, creating a peer-to-peer network on our localhost. In addition to the two nodes, a bootnode (discovery service) will also be setup.

It took me quite some time and extensive research and googling to finally have a solid ethereum development environment for testing my smart contracts and my DApps.

In this post, I’ve decided to share how I am setting a Proof-of-Authority network using the clique consensus engine of Geth. It’s my way to thank the community by giving back and hopefully making life easier for anyone willing exploring the Ethereum universe.

OS and Software

My OS is CentOS 7.

For the Ethereum client, I am using Geth (the Go implementation of the Ethereum protocole). I believe that Geth is easy to install with plenty of great tutorials out there, so I am not gonna cover any installation here. I am currently running Geth Version: 1.8.15-unstable:

[furnace@localhost devnet]$ geth version
WARN [09-04|12:07:26.031] Sanitizing cache to Go's GC limits       provided=1024 updated=613
Geth
Version: 1.8.15-unstable
Git Commit: c1c003e4ff36c22d67662ca661fc78cde850d401
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.11
Operating System: linux
GOPATH=
GOROOT=/usr/local/go
[furnace@localhost devnet]$

1. Let's get started

1.0 Overview

Let’s start by the end… For clarity, this is what you are supposed to get when you will have completed Chapter 1.

[furnace@localhost devnet]$ tree -L 2
.

0 directories, 0 files
[furnace@localhost devnet]$

1.1 create a workspace

[furnace@localhost devnet]$ mkdir devnet
[furnace@localhost devnet]$ cd devnet
[furnace@localhost devnet]$ mkdir node1 node2
[furnace@localhost devnet]$

1.2 create your accounts

The accounts (also called wallet) hold a private-public key pair that are required for interacting with any blockchain. Any mining node (strictly speaking the nodes will not be mining but voting) needs to be able to sign transactions (using their private key) and to identify itself on the network (the address is derived from the public key). Therefore we need at least two accounts, one per node.

In Geth jargon, a voting node is called a Sealer.

for node 1:

[furnace@localhost devnet]$ geth --datadir node1 account new
WARN [09-04|12:17:34.774] Sanitizing cache to Go's GC limits       provided=1024 updated=613
INFO [09-04|12:17:34.775] Maximum peer count                       ETH=25 LES=0 total=25
Your new account is locked with a password. Please give a password. Do not forget this password.
Passphrase: pwdnode1 (for example)
Repeat passphrase: pwdnode1
Address: {bac5564cc4f7528ebb6150270ee63b3fa1641b17}
[furnace@localhost devnet]$

for node 2:

[furnace@localhost linux]$ geth --datadir node2 account new
WARN [09-04|12:19:48.058] Sanitizing cache to Go's GC limits       provided=1024 updated=613
INFO [09-04|12:19:48.058] Maximum peer count                       ETH=25 LES=0 total=25
Your new account is locked with a password. Please give a password. Do not forget this password.
Passphrase: pwdnode2 (for example)
Repeat passphrase: pwdnode2
Address: {8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08}
[furnace@localhost linux]$

This creates the keystore/ folder containing your account file. Notice that the last part of the file name in keystore/ is the address of your account (also printed in the terminal just above).

I suggest to copy these two addresses from the terminal screen and to save them in a text file. That will ease some copy-pasting job later on. However remember that you can read those addesses from the UTC-datetime-address file in keystore/.

[furnace@localhost devnet]$ echo 'bac5564cc4f7528ebb6150270ee63b3fa1641b17' >> accounts.txt
[furnace@localhost devnet]$ echo '8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08' >> accounts.txt
[furnace@localhost devnet]$

For each node, I propose to save your password in a file. That will ease some process for later on (such as unlocking your account)

[furnace@localhost devnet]$ echo 'pwdnode1' > node1/password.txt
[furnace@localhost devnet]$ echo 'pwdnode2' > node2/password.txt
[furnace@localhost devnet]$

1.3 create your Genesis file

A genesis file is the file used to initialize the blockchain. The very first block, called the genesis block, is crafted based on the parameters in the genesis.json file.

Geth comes with a bunch of exectuables such as puppeth or bootnode . You can find the complete list on the Geth github. Puppeth removes the pain of creating a genesis file from scratch (and does much more). Start puppeth:

[furnace@localhost devnet]$ puppeth
+-----------------------------------------------------------+
| Welcome to puppeth, your Ethereum private network manager |
|                                                           |
| This tool lets you create a new Ethereum network down to  |
| the genesis block, bootnodes, miners and ethstats servers |
| without the hassle that it would normally entail.         |
|                                                           |
| Puppeth uses SSH to dial in to remote servers, and builds |
| its network components out of Docker containers using the |
| docker-compose toolset.                                   |
+-----------------------------------------------------------+

Please specify a network name to administer (no spaces or hyphens, please)
>

and happily answer the questions (every value can be updated by hand later on, so don’t spent too much time engineering it for your first trials).

Please specify a network name to administer (no spaces or hyphens, please)
> devnet

Sweet, you can set this via --network=devnet next time!

INFO [09-04|12:31:23.261] Administering Ethereum network           name=devnet
WARN [09-04|12:31:23.261] No previous configurations found         path=/home/furnace/.puppeth/devnet

What would you like to do? (default = stats)
 1. Show network stats
 2. Configure new genesis
 3. Track new remote server
 4. Deploy network components
> 2

Which consensus engine to use? (default = clique)
 1. Ethash - proof-of-work
 2. Clique - proof-of-authority
> 2

How many seconds should blocks take? (default = 15)
> 10 // for example

Which accounts are allowed to seal? (mandatory at least one)
> 0xbac5564cc4f7528ebb6150270ee63b3fa1641b17
> 0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08
> 0x

Which accounts should be pre-funded? (advisable at least one)
> 0xbac5564cc4f7528ebb6150270ee63b3fa1641b17 // free ethers !
> 0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08
> 0x

Specify your chain/network ID if you want an explicit one (default = random)
> 1515 // for example. Do not use anything from 1 to 10
INFO [09-04|12:33:03.083] Configured new genesis block

What would you like to do? (default = stats)
 1. Show network stats
 2. Manage existing genesis
 3. Track new remote server
 4. Deploy network components
> 2

 1. Modify existing fork rules
 2. Export genesis configuration
 3. Remove genesis configuration
> 2

Which file to save the genesis into? (default = devnet.json)
> genesis.json
INFO [09-04|12:33:32.988] Exported existing genesis block

What would you like to do? (default = stats)
 1. Show network stats
 2. Manage existing genesis
 3. Track new remote server
 4. Deploy network components
> ^C // ctrl+C to quit puppeth
[furnace@localhost devnet]$

PoA doesn’t have mining rewards

So I would highly suggest that you allocate some ethers (defined in the unit of wei) to a bunch of addresses in the genesis file, otherwise you’ll hand up without any ether and thus will not be able to pay for your transactions. You could have a gasPrice of zero but that sometimes leads to undesired behavior from the nodes that could go under the radar (like not broadcasting pending transaction depending on the config of the other nodes on the network). I encourage you nevertheless to play with every parameter :)

1.4 Initialize your nodes

Now that we have the genesis.json file, let’s forge the genesis block ! Each node MUST be initialize with the SAME genesis file.

[furnace@localhost devnet]$ geth --datadir node1 init genesis.json
WARN [09-04|12:41:13.796] Sanitizing cache to Go's GC limits       provided=1024 updated=613
INFO [09-04|12:41:13.798] Maximum peer count                       ETH=25 LES=0 total=25
INFO [09-04|12:41:13.799] Allocated cache and file handles         database=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon.com/poa-private-network/devnet/node1/geth/chaindata cache=16 handles=16
INFO [09-04|12:41:13.802] Writing custom genesis block
INFO [09-04|12:41:13.809] Persisted trie from memory database      nodes=358 size=52.27kB time=1.123878ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [09-04|12:41:13.809] Successfully wrote genesis state         database=chaindata                                                                                                              hash=73c7c0…9ac634
INFO [09-04|12:41:13.809] Allocated cache and file handles         database=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon.com/poa-private-network/devnet/node1/geth/lightchaindata cache=16 handles=16
INFO [09-04|12:41:13.813] Writing custom genesis block
INFO [09-04|12:41:13.817] Persisted trie from memory database      nodes=358 size=52.27kB time=909.821µs  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [09-04|12:41:13.817] Successfully wrote genesis state         database=lightchaindata                                                                                                              hash=73c7c0…9ac634
[furnace@localhost devnet]$

[furnace@localhost devnet]$ geth --datadir node2 init genesis.json
WARN [09-04|12:44:51.531] Sanitizing cache to Go's GC limits       provided=1024 updated=613
INFO [09-04|12:44:51.533] Maximum peer count                       ETH=25 LES=0 total=25
INFO [09-04|12:44:51.533] Allocated cache and file handles         database=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon.com/poa-private-network/devnet/node2/geth/chaindata cache=16 handles=16
INFO [09-04|12:44:51.536] Writing custom genesis block
INFO [09-04|12:44:51.542] Persisted trie from memory database      nodes=358 size=52.27kB time=1.110326ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [09-04|12:44:51.542] Successfully wrote genesis state         database=chaindata                                                                                                              hash=73c7c0…9ac634
INFO [09-04|12:44:51.542] Allocated cache and file handles         database=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon.com/poa-private-network/devnet/node2/geth/lightchaindata cache=16 handles=16
INFO [09-04|12:44:51.546] Writing custom genesis block
INFO [09-04|12:44:51.550] Persisted trie from memory database      nodes=358 size=52.27kB time=879.024µs  gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [09-04|12:44:51.550] Successfully wrote genesis state         database=lightchaindata                                                                                                              hash=73c7c0…9ac634
[furnace@localhost devnet]$

tada ! done.

Side note: how does your node know about the genesis parameters when joining the Ethereum Mainnet or the Ropsten testnet, or the Rinkeby testnet ? They are already defined in the source code in params/config.go.

1.5 Create a bootnode

A bootnode only purpose is to helping nodes discovering each others (remember, the Ethereum blockchain is a peer-to-peer network). Nodes could have dynamic IP, being turned off, and on again. The bootnode is usually ran on a static IP and thus acts like a pub where nodes know they will find their mates.

Initialize the bootnode:

[furnace@localhost devnet]$ bootnode -genkey boot.key
[furnace@localhost devnet]$

This creates a value called the enode uniquely identifying your bootnode (more on this soon) and we store this enode in the boot.key file.

1.6 Midway celebration

Congrats ! Chapter 1 is done :) try

[furnace@localhost devnet]$ tree -L 2
.
├── accounts.txt
├── boot.key
├── genesis.json
├── node1
│   ├── geth
│   ├── keystore
│   └── password.txt
└── node2
    ├── geth
    ├── keystore
    └── password.txt

6 directories, 5 files
[furnace@localhost devnet]$

and compare the output with the section 1.0. Hopefully you should get the same tree.

At this point the setup is done and we are ready to make this blockchain live.

2. Make it live

2.1 Start the bootnode service

[furnace@localhost devnet]$ bootnode -nodekey boot.key -verbosity 9 -addr :30310
INFO [09-04|13:02:53.293] UDP listener up                          self=enode://bde0739c7d9f213054575574d6e8e1df21147066a666bc939f29634dac947ad5c715a990d878264fe91a83c9fa0495911c824fe43329570c8d0510c8fa3243f2@[::]:30310

I like to have some verbosity for my bootnode as it is nice to see when the nodes are playing ping-pong on the network (meaning it’s working!).

Feel free to use any port you like but please avoid the mainstream ones (like 80 for HTTP). 30303 is used for the public ethereum networks.

2.2 Starting your nodes

Big time ! Finally (but usually here the troubles arrive too).

Everything in one huge command ! I am gonna cover some options but please do your homework and refer to the doc.

starting node 1

[furnace@localhost devnet]$ geth --datadir node1/ --syncmode 'full' --port 30311 --rpc --rpcaddr 'localhost' --rpcport 8501 --rpcapi 'personal,db,eth,net,web3,txpool,miner' --bootnodes 'enode://bde0739c7d9f213054575574d6e8e1df21147066a666bc939f29634dac947ad5c715a990d878264fe91a83c9fa0495911c824fe43329570c8d0510c8fa3243f2@127.0.0.1:30310' --networkid 1515 --gasprice '1' -unlock '0xbac5564cc4f7528ebb6150270ee63b3fa1641b17' --password node1/password.txt --mine
WARN [09-04|13:39:47.264] Sanitizing cache to Go's GC limits       provided=1024 updated=613
INFO [09-04|13:39:47.264] Maximum peer count                       ETH=25 LES=0 total=25
INFO [09-04|13:39:47.265] Starting peer-to-peer node               instance=Geth/v1.8.15-unstable-c1c003e4/linux-amd64/go1.11
INFO [09-04|13:39:47.265] Allocated cache and file handles         database=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon/poa/devnet/node1/geth/chaindata cache=459 handles=1024
INFO [09-04|13:39:47.270] Initialised chain configuration          config="{ChainID: 1515 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: <nil> Engine: clique}"
INFO [09-04|13:39:47.270] Initialising Ethereum protocol           versions="[63 62]" network=1515
INFO [09-04|13:39:47.271] Loaded most recent local header          number=0 hash=73c7c0…9ac634 td=1
INFO [09-04|13:39:47.271] Loaded most recent local full block      number=0 hash=73c7c0…9ac634 td=1
INFO [09-04|13:39:47.271] Loaded most recent local fast block      number=0 hash=73c7c0…9ac634 td=1
INFO [09-04|13:39:47.271] Loaded local transaction journal         transactions=0 dropped=0
INFO [09-04|13:39:47.271] Regenerated local transaction journal    transactions=0 accounts=0
INFO [09-04|13:39:47.271] Starting P2P networking
INFO [09-04|13:39:49.378] UDP listener up                          self=enode://0863f00ba83494b69ce32fd45dd06a2b83497fa15ebe34e64ab47a74f21e1dfdcd15f8eb7e276f780c3f825659e4c8eaccb212f953a5c87f096bd6fd7e32491a@[::]:30311
INFO [09-04|13:39:49.379] IPC endpoint opened                      url=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon/poa/devnet/node1/geth.ipc
INFO [09-04|13:39:49.380] HTTP endpoint opened                     url=http://localhost:8501                                                                        cors= vhosts=localhost
INFO [09-04|13:39:49.409] RLPx listener up                         self=enode://0863f00ba83494b69ce32fd45dd06a2b83497fa15ebe34e64ab47a74f21e1dfdcd15f8eb7e276f780c3f825659e4c8eaccb212f953a5c87f096bd6fd7e32491a@[::]:30311
INFO [09-04|13:39:49.929] Unlocked account                         address=0xBaC5564cC4F7528eBb6150270EE63B3Fa1641B17
INFO [09-04|13:39:49.929] Transaction pool price threshold updated price=1
INFO [09-04|13:39:49.929] Transaction pool price threshold updated price=1
INFO [09-04|13:39:49.929] Etherbase automatically configured       address=0xBaC5564cC4F7528eBb6150270EE63B3Fa1641B17
INFO [09-04|13:39:49.930] Commit new mining work                   number=1 sealhash=406169…7dffd4 uncles=0 txs=0 gas=0 fees=0 elapsed=120.071µs
INFO [09-04|13:39:49.930] Successfully sealed new block            number=1 sealhash=406169…7dffd4 hash=1adc58…cf4be9 elapsed=436.829µs
INFO [09-04|13:39:49.930] 🔨 mined potential block                  number=1 hash=1adc58…cf4be9
INFO [09-04|13:39:49.930] Commit new mining work                   number=2 sealhash=45ab48…cf43cf uncles=0 txs=0 gas=0 fees=0 elapsed=240.863µs
INFO [09-04|13:39:49.930] Signed recently, must wait for others

• --syncmode 'full' helps preventing the error Discarded Bad Propagated Block.
• --port 30311 is the enode port for node1 and has to be different from the bootnode port (that is 30310 if you followed my command) because we are on a localhost. On a real network (on node per machine), use the same port.
• --rpcapi allows the listed modules to be used over RPC calls (see section 3.3 for an example). See the Geth Management APIs for more info. Be mindful about hacks as everyone can call your RPC methods if no firewall is protecting your node.
• --bootnodes tells your node at what address to find your bootnode. Replace [::] with the bootnode IP. No domain name are allowed! Only IPs. Check enode URL format.
• --networkId as defined in the genesis.json file. Please use the same id!
• --gasprice '1' I don’t like to pay on my own network :) be careful with gasprice. If your transactions are not being broadcasted to the network but only the node receiving the transactions is processing them, this means you sent a transaction with a gasprice that is not accepted (too low) by the other nodes on the network. No error will be return. If you have two nodes, only one will be processing the transactions. This is sneaky and reduces your network throughput by a factor 2.
• --unlock --password --mine tell the node to unlock this account, with the password in that file and to start mining (i.e. voting/sealing for Proof-of-Authority)
• --targetgaslimit value see the update in section 2.3.

same for node 2 (update parameters specific to the node)

[furnace@localhost devnet]$ geth --datadir node2/ --syncmode 'full' --port 30312 --rpc --rpcaddr 'localhost' --rpcport 8502 --rpcapi 'personal,db,eth,net,web3,txpool,miner' --bootnodes 'enode://bde0739c7d9f213054575574d6e8e1df21147066a666bc939f29634dac947ad5c715a990d878264fe91a83c9fa0495911c824fe43329570c8d0510c8fa3243f2@127.0.0.1:30310' --networkid 1515 --gasprice '1' -unlock '0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08' --password node2/password.txt --mine
WARN [09-04|13:40:13.884] Sanitizing cache to Go's GC limits       provided=1024 updated=613
INFO [09-04|13:40:13.884] Maximum peer count                       ETH=25 LES=0 total=25
INFO [09-04|13:40:13.885] Starting peer-to-peer node               instance=Geth/v1.8.15-unstable-c1c003e4/linux-amd64/go1.11
INFO [09-04|13:40:13.885] Allocated cache and file handles         database=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon/poa/devnet/node2/geth/chaindata cache=459 handles=1024
INFO [09-04|13:40:13.891] Initialised chain configuration          config="{ChainID: 1515 Homestead: 1 DAO: <nil> DAOSupport: false EIP150: 2 EIP155: 3 EIP158: 3 Byzantium: 4 Constantinople: <nil> Engine: clique}"
INFO [09-04|13:40:13.891] Initialising Ethereum protocol           versions="[63 62]" network=1515
INFO [09-04|13:40:13.892] Loaded most recent local header          number=0 hash=73c7c0…9ac634 td=1
INFO [09-04|13:40:13.892] Loaded most recent local full block      number=0 hash=73c7c0…9ac634 td=1
INFO [09-04|13:40:13.892] Loaded most recent local fast block      number=0 hash=73c7c0…9ac634 td=1
INFO [09-04|13:40:13.892] Loaded local transaction journal         transactions=0 dropped=0
INFO [09-04|13:40:13.892] Regenerated local transaction journal    transactions=0 accounts=0
INFO [09-04|13:40:13.892] Starting P2P networking
INFO [09-04|13:40:15.997] UDP listener up                          self=enode://999a764d33869166f74f725dddcc3bd33520d11505796293518079800dac1dee60754da7fc9e7477f04d3fad9adf32c320f9ba4ad46d39e0d93b5bf182283080@[::]:30312
INFO [09-04|13:40:15.998] IPC endpoint opened                      url=/home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon/poa/devnet/node2/geth.ipc
INFO [09-04|13:40:15.999] HTTP endpoint opened                     url=http://localhost:8502                                                                        cors= vhosts=localhost
INFO [09-04|13:40:16.028] RLPx listener up                         self=enode://999a764d33869166f74f725dddcc3bd33520d11505796293518079800dac1dee60754da7fc9e7477f04d3fad9adf32c320f9ba4ad46d39e0d93b5bf182283080@[::]:30312
INFO [09-04|13:40:16.557] Unlocked account                         address=0x8A6673DC85E544Bce6d4876f63CfcDC4E5C8Cc08
INFO [09-04|13:40:16.557] Transaction pool price threshold updated price=1
INFO [09-04|13:40:16.557] Transaction pool price threshold updated price=1
INFO [09-04|13:40:16.557] Etherbase automatically configured       address=0x8A6673DC85E544Bce6d4876f63CfcDC4E5C8Cc08
INFO [09-04|13:40:16.557] Commit new mining work                   number=1 sealhash=cfb980…7f93e2 uncles=0 txs=0 gas=0 fees=0 elapsed=82.213µs
INFO [09-04|13:40:16.667] Successfully sealed new block            number=1 sealhash=cfb980…7f93e2 hash=1844aa…45566e elapsed=109.635ms
INFO [09-04|13:40:16.667] 🔨 mined potential block                  number=1 hash=1844aa…45566e
INFO [09-04|13:40:16.667] Commit new mining work                   number=2 sealhash=c358ea…3a45c5 uncles=0 txs=0 gas=0 fees=0 elapsed=210.384µs
INFO [09-04|13:40:16.667] Signed recently, must wait for others
INFO [09-04|13:40:26.028] Block synchronisation started
INFO [09-04|13:40:26.028] Mining aborted due to sync
INFO [09-04|13:40:26.030] Imported new chain segment               blocks=1 txs=0 mgas=0.000 elapsed=439.659µs mgasps=0.000 number=1 hash=1adc58…cf4be9 cache=0.00B
INFO [09-04|13:40:26.031] Commit new mining work                   number=2 sealhash=74efc0…7e7d4a uncles=1 txs=0 gas=0 fees=0 elapsed=52.306µs
INFO [09-04|13:40:26.031] Successfully sealed new block            number=2 sealhash=74efc0…7e7d4a hash=a0faf0…46d9e3 elapsed=492.377µs
INFO [09-04|13:40:26.031] 🔨 mined potential block                  number=2 hash=a0faf0…46d9e3
INFO [09-04|13:40:26.032] Commit new mining work                   number=3 sealhash=25d534…bf8155 uncles=1 txs=0 gas=0 fees=0 elapsed=319.072µs
INFO [09-04|13:40:26.032] Signed recently, must wait for others
INFO [09-04|13:40:36.002] Imported new chain segment               blocks=1 txs=0 mgas=0.000 elapsed=176.7µs   mgasps=0.000 number=3 hash=9d8671…b6a367 cache=0.00B
INFO [09-04|13:40:36.002] Commit new mining work                   number=4 sealhash=dae53f…f55f65 uncles=1 txs=0 gas=0 fees=0 elapsed=59.942µs

At this point your bootnode should stream connections coming from node1 (port 30311) and node2 (port 30312) as shown in the upper terminal window. Node1 (middle terminal) and node2 (lower terminal) should be happily mining and signing blocks. Here I have a period of 1 second (defined in the genesis file) therefore the fast block creation.

[图片上传失败...(image-209569-1543368534581)]

2.3 Update your genesis file

I am sure you’ll want to modify some values in your genesis file. Go ahead ! However in order for those changes to become effective, we have to initialize a new blockchain. Here is the genesis file I am currently using:

{
    "config": {
        "chainId": 1515,
        "homesteadBlock": 1,
        "eip150Block": 2,
        "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "eip155Block": 3,
        "eip158Block": 3,
        "byzantiumBlock": 4,
        "clique": {
            "period": 1,
            "epoch": 30000
        }
    },
    "nonce": "0x0",
    "timestamp": "0x5a722c92",
    "extraData": "0x000000000000000000000000000000000000000000000000000000000000000008a58f09194e403d02a1928a7bf78646cfc260b087366ef81db496edd0ea2055ca605e8686eec1e60000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "gasLimit": "0x59A5380",
    "difficulty": "0x1",
    "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "coinbase": "0x0000000000000000000000000000000000000000",
    "alloc": {
        "08a58f09194e403d02a1928a7bf78646cfc260b0": {
            "balance": "0x200000000000000000000000000000000000000000000000000000000000000"
        },
        "87366ef81db496edd0ea2055ca605e8686eec1e6": {
            "balance": "0x200000000000000000000000000000000000000000000000000000000000000"
        },
        "F464A67CA59606f0fFE159092FF2F474d69FD675": {
            "balance": "0x200000000000000000000000000000000000000000000000000000000000000"
        }
    },
    "number": "0x0",
    "gasUsed": "0x0",
    "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000"
}

I’ve cleaned the empty addresses that puppeth includes when creating the file (at section 1.3). I’ve also added a third address that gets funded when the genesis block is created. Then I have changed the period from 15 second to 1 to get those blocks mined faster (be careful as one empty block weights 1024 bytes — here my chaindata/ folder gains 1024 bytes per second (and more if the blocks are not empty). Finally I’ve increased the gasLimit to allow for more transaction (trully speaking, computation) per block.

update: The gasLimit defined in the genesis file only applies to the genesis block ! The gasLimit of new blocks is DYNAMIC meaning its value is changing over time depending on how much gas was used in the parent (previous) block. The computation of the new gasLimit is done in the function CalcGasLimit (github source). If you want a constant gas Limit use the option --targetgaslimit intValue when running geth. I would recommend to set it equal to the gasLimit in the genesis file (the command option is an integer whereas the genesis value is hexadecimal) so that you get a constant gas limit that does not change over time anymore. Given the genesis file above with "gasLimit":"0x59A5380" , I am running my node with --targetgaslimit 94000000 for a constant gas limit across all blocks.

The field extraData contains the address that are allowed to seal (that’s why puppeth is nice to have).

I have investigate the impact of changing the period and the gasLimit on the number of transaction per second (transaction rate) that the blockchain can process. But that’s gonna be another article; link here.

When you are happy with your genesis file. Kill your nodes if they are running (ctrl C in the terminal). Then delete the folder geth/ in node1/ and geht/ in node2/ . Delete only geth/ folders!

Then initialize your nodes. From section 1.4 :

[furnace@localhost devnet]$ geth --datadir node1/ init genesis.json
[furnace@localhost devnet]$ geth --datadir node2/ init genesis.json

and start your nodes again with the commands in section 2.2

3. Interact with your nodes

Great your network is now live :) but how to connect to it and starting exploring ?

3.1 Open a Geth Javascript Console

The simplest and probably more straight forward way to play with a node is probably to attach a Geth javascript console to one of the nodes.

3.1.1 Through IPC

IPC (Inter-Process Communication) works only locally : you should be on the same machine as your node. Open an extra terminal and attach to your node.

To connect to node1:

[furnace@localhost devnet]$ geth attach node1/geth.ipc
WARN [09-04|14:16:21.530] Sanitizing cache to Go's GC limits       provided=1024 updated=613
Welcome to the Geth JavaScript console!

instance: Geth/v1.8.15-unstable-c1c003e4/linux-amd64/go1.11
coinbase: 0xbac5564cc4f7528ebb6150270ee63b3fa1641b17
at block: 3 (Tue, 04 Sep 2018 13:40:36 EDT)
 datadir: /home/furnace/bitbucket/zblockchain/ethereumcodes/linux/hackernoon/poa/devnet/node1
 modules: admin:1.0 clique:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

>

The file geth.ipc is created only when the node is running. So do not expect to find it if your node1 is off.

RPC gives access without restriction to all modules listed in the terminal : admin: 1.0 clique:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

3.1.2 Through RPC

RPC (Remote Procedure Call) works over the internet as HTTP requests. Therefore be careful when you open RPC to the outside world as everyone will have access to your node. For this reason RPC is disabled by default and when enabled it does not give access to all modules. In this guide we allowed RPC on our Geth node with the command --rpc and gave access to the modules personal,db,eth,net,web3,txpool,miner (from section 2.2).

To connect to node1 using RPC:

[furnace@localhost devnet]$ geth attach 'http://localhost:8501'
WARN [09-04|14:18:59.858] Sanitizing cache to Go's GC limits       provided=1024 updated=613
Welcome to the Geth JavaScript console!

instance: Geth/v1.8.15-unstable-c1c003e4/linux-amd64/go1.11
coinbase: 0xbac5564cc4f7528ebb6150270ee63b3fa1641b17
at block: 3 (Tue, 04 Sep 2018 13:40:36 EDT)
 modules: eth:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

>

3.1.3 Using the Geth Javascript Console

Here are some examples of methods

> net.version
"1515"
> eth.blockNumber
3
> eth.coinbase
"0xbac5564cc4f7528ebb6150270ee63b3fa1641b17"
> eth.sendTransaction({'from':eth.coinbase, 'to':'0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08', 'value':web3.toWei(3, 'ether')})
"0x66808c1cd481339656016c9672632c4e3167ea3503ba1170d8ee33df9fb67833"
> eth.getTransactionReceipt("0x66808c1cd481339656016c9672632c4e3167ea3503ba1170d8ee33df9fb67833")
null
> eth.getTransactionReceipt("0x66808c1cd481339656016c9672632c4e3167ea3503ba1170d8ee33df9fb67833")
null
> eth.sendTransaction({'from':eth.coinbase, 'to':'0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08', 'value':web3.toWei(0.00001, 'ether')})
"0xa4d5d46422f01ccc962ec3df5c546c1829e10036986ee35e68d3e5d5cb36f254"
> eth.getTransactionReceipt("0xa4d5d46422f01ccc962ec3df5c546c1829e10036986ee35e68d3e5d5cb36f254")
null
> eth.sendTransaction({'from':eth.coinbase, 'to':'0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08', 'value':web3.toWei(0.00001, 'ether')})
"0x17535d6b94c4afeafc0c340ca1558e5af58e21a0ff80113c37cc9e9e4ea15b99"
> eth.getTransactionReceipt("0x17535d6b94c4afeafc0c340ca1558e5af58e21a0ff80113c37cc9e9e4ea15b99")
{
  blockHash: "0x21cff6be11c3c2246973261c50929bccb3e7de10f76a7cc7343d920db14f6b04",
  blockNumber: 6,
  contractAddress: null,
  cumulativeGasUsed: 21000,
  from: "0xbac5564cc4f7528ebb6150270ee63b3fa1641b17",
  gasUsed: 21000,
  logs: [],
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  status: "0x1",
  to: "0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08",
  transactionHash: "0x17535d6b94c4afeafc0c340ca1558e5af58e21a0ff80113c37cc9e9e4ea15b99",
  transactionIndex: 0
}
> eth.getTransactionReceipt("0xa4d5d46422f01ccc962ec3df5c546c1829e10036986ee35e68d3e5d5cb36f254")
{
  blockHash: "0xa5678a258431440bb6961489e0f4074203e1f0753530a16c7044b479a1da4543",
  blockNumber: 5,
  contractAddress: null,
  cumulativeGasUsed: 42000,
  from: "0xbac5564cc4f7528ebb6150270ee63b3fa1641b17",
  gasUsed: 21000,
  logs: [],
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  status: "0x1",
  to: "0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08",
  transactionHash: "0xa4d5d46422f01ccc962ec3df5c546c1829e10036986ee35e68d3e5d5cb36f254",
  transactionIndex: 1
}
> eth.getTransactionReceipt("0x66808c1cd481339656016c9672632c4e3167ea3503ba1170d8ee33df9fb67833")
{
  blockHash: "0xa5678a258431440bb6961489e0f4074203e1f0753530a16c7044b479a1da4543",
  blockNumber: 5,
  contractAddress: null,
  cumulativeGasUsed: 21000,
  from: "0xbac5564cc4f7528ebb6150270ee63b3fa1641b17",
  gasUsed: 21000,
  logs: [],
  logsBloom: "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
  status: "0x1",
  to: "0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08",
  transactionHash: "0x66808c1cd481339656016c9672632c4e3167ea3503ba1170d8ee33df9fb67833",
  transactionIndex: 0
}
> exit // to quit the Geth javascript console
[furnace@localhost devnet]$

Note

When there is none miner or sealer node, the eth.getTransactionReceipt will return null. And after miner, some transactions would be packed in on block, such as above block number 5.

for the full list of methods, see Management APIs and JSON RPC API.

3.2 Using Mist

The Mist browser provides a graphical user interface for deploying and interacting with smart contracts and managing accounts. To connect Mist to your local private network over IPC, simply do :

devnet$ mist --rpc node1/geth.ipc

and over RPC (make sure RPC is enabled)

$ mist --rpc 'http://localhost:8501'

The procedure is exactly the same if you want to use the Ethereum wallet instead of mist. Just replace mist by ethereumwallet in the commands above.

3.3 Making RPC calls with your favorite programming language

In section 3.1, we saw how to interact with the Geth API by hand. Now let’s use our PC for what it is best at : automation.

The reference and by far for sending JSON-RPC requests to your node is the web3.js javascript library. I believe the internet of full of great tutorial and example on how to use the web3.js library. Therefore I am not gonna covert any of it here.

The JSON-RPC APIs are currently also being implemented in java with the web3.j library and in python with the web3.py library. Those libraries offer high-level methods for working with the ethereum blockchain just like web3.js.

However, it’s also possible to send raw JSON-RPC requests directly to your node. I think it is worth trying as it’s providing a valuable understanding on how those high-level libraries work under the hood.

Here is a simple example of sending a raw JSON-RPC request to my node using python 3:

[furnace@localhost devnet]$ python3.6
Python 3.6.5 (default, Apr 10 2018, 17:08:37)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import json
>>> session = requests.Session()
>>> method = 'eth_getTransactionCount'
>>> params = ["0x627306090abaB3A6e1400e9345bC60c78a8BEf57","latest"]
>>> PAYLOAD = {"jsonrpc":"2.0","method":method,"params":params,"id":67}
>>> PAYLOAD = json.dumps(PAYLOAD)
>>> headers = {'Content-type': 'application/json'}
>>> response = session.post('http://127.0.0.1:8501', data=PAYLOAD, headers=headers)
>>> response.content
b'{"jsonrpc":"2.0","id":67,"result":"0x0"}\n'
>>> json.loads(response.content)['result']
'0x0'
>>>

rpc.py

import requests
import json
session = requests.Session()
method = 'eth_getTransactionCount'
params = ["0x627306090abaB3A6e1400e9345bC60c78a8BEf57","latest"]
PAYLOAD = {"jsonrpc":"2.0","method":method,"params":params,"id":67}
PAYLOAD = json.dumps(PAYLOAD)
headers = {'Content-type': 'application/json'}
response = session.post('http://127.0.0.1:8501', data=PAYLOAD, headers=headers)
print(response.content)
print(json.loads(response.content)['result'])

[furnace@localhost devnet]$ python3.6 rpc.py
b'{"jsonrpc":"2.0","id":67,"result":"0x0"}\n'
0x0
[furnace@localhost devnet]$

[furnace@localhost devnet]$ curl -H "Content-Type: application/json" -X POST --data '{"jsonrpc":"2.0","method":"net_version","params":[],"id":67}' http://localhost:8501
{"jsonrpc":"2.0","id":67,"result":"1515"}
[furnace@localhost devnet]$

3.4 Deploy and test your smart contracts with Truffle on your private network

Development frameworks like Truffle (or Embark, Populus) are great tools for developing and testing smart contracts.

When you initialize a workspace with

[furnace@localhost hackernoon]$ truffle init

Truffle creates a series of files and folders to help you get started. I usually edit the truffle.js file as such

module.exports = {
    // See <http://truffleframework.com/docs/advanced/configuration>
    // to customize your Truffle configuration!
    networks: {
        devnet: {
            host: '127.0.0.1',
            port: 8501,
            gas: 4500000,           // should be add, else will Error: exceeds block gas limit
            network_id: '*'
        },
        ganache: {
            host: '127.0.0.1',
            port: 7545,
            network_id: '*'
        }
    }
};

then use the command

[furnace@localhost hackernoon]$ truffle deploy --network devnet
Using network 'devnet'.

Network up to date.
[furnace@localhost hackernoon]$

to deploy your smart contracts defined in migrations/X_deploy.js . Or for running your tests in test/

[furnace@localhost hackernoon]$ truffle test --network devnet
Using network 'devnet'.


  0 passing (0ms)

[furnace@localhost hackernoon]$

Usually the Ethereum Blockchain simulator Ganache is more than enough for running your tests. However I like to use my private blockchain for ultimate testing on a real node and not only on a simulator. With Ganache I believe that the layer of abstraction is too big, what is the beauty of it but also a danger as it requires no understanding what so ever of the complexity of a real node (transaction pool, gasPrice, gasLimit, broadcasting transactions between nodes, mining or voting, computation time, consensus engine, etc.).

4. Extra: Building a Resilient Network

When deploying this network on AWS, we thought about deploying 4 different nodes (each one with its own account) in two different availability zones for great robustness. WRONG ! If N sealers are defined in the genesis file, clique will only work if int(N/2+1) (source code) nodes are online. This means that if we define 5 sealers in the genesis file, we will need a minimum of 3 nodes to be mining (sealing) for the blockchain to work. If we define 4 sealers we’ll also need at least 3 mining nodes.

Ok cool so what is the problem ? Back to our task of creating a resilient network with 4 nodes in two availability zones, the blockchain will only work if 3 nodes are minnig. That is bad because if one availability zone goes down we are left with only 2 nodes, and the blockchain will be stuck in the state “Signed recently, must wait for others” . This bad setup is illustrated in the Figure below.

The solution is to run multiple times the same node in different availability zones. The simplest case is presented in the Figure below. This way the network is still working if one availability zone goes offline !

reminder: think about using a fix IP for the bootnodes ;)

What’s next ?

That’s pretty much it for this guide. If you understand everything here I believe you’re already on very good tracks and you have a solid foundation on which you can continue your journey with confidence.

You can start developing Dapps (Decentralized Applications) by grabbing a web3 library or by making your own custom JSON-RPC wrapper.

In this post, I explore how to use python for deploying and transacting with a smart contract using only raw HTTP requests.

Final Words

Congratulation if you made it until the end. I hope this guide is comprehensive and helped you on your journey. I welcome any feedback to improve this guide !

And a BIG Thank you to the community for all the documentation, tutorials, Q&A websites and guides out there.

Happy Hacking !

Appendix A

• geth 1.8 was released a few days after this guide was published and fortunately does not break anything. This post is then valid and was tested for both geth 1.7.3 and geth 1.8. Awesome :)
• Clique requires int(N/2+1) sealers (where N is the number of sealers defined in the genesis file — in extraData field) to be online in order to run.
• thx to Ivica Aracic for pointing out that clique PoA DOES WORK with a single node. For any reason I missed that and I apologize for the confusion. With a single node, we just need (A) create genesis file with only one sealer (only 1 address in extraData ) (B) create an account (C) init geth (D) run geth, unlock account and mine. No bootnode is required then.

Appendix B. Install python and library

[furnace@localhost ~]$ sudo python3.6 -m pip install --upgrade pip
[furnace@localhost ~]$ sudo python3.6 -m pip install requests

Appendix C. The whole steps

1. Prepare

1.1 Create workspace

[furnace@localhost devnet] cd devnet
[furnace@localhost devnet]

1.2 Create accounts

[furnace@localhost devnet] geth --datadir node1 account new

[furnace@localhost devnet] echo '8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08' >> accounts.txt

[furnace@localhost devnet] echo 'pwdnode2' > node2/password.txt

1.3 Create genesis file

By tool puppeth.

2. Play

The following can execute by script deploy.sh.

2.1 Clear nodes

[furnace@localhost devnet] rm -rf node2/geth

2.2 Init nodes

[furnace@localhost devnet] geth --datadir node2 init genesis.json

2.3 Create bootnode

[furnace@localhost devnet]$ bootnode -genkey boot.key

2.4 Startup bootnode

[furnace@localhost devnet]$ bootnode -nodekey boot.key -verbosity 9 -addr :30310

2.5 Startup node1

[furnace@localhost devnet]$ geth --datadir node1/ --syncmode 'full' --port 30311 --rpc --rpcaddr 0.0.0.0 --rpcport 8501 --rpcapi 'personal,db,eth,net,web3,txpool,miner' --rpccorsdomain "" --ws --wsaddr 0.0.0.0 --wsport 8601 --wsapi 'personal,db,eth,net,web3,txpool,miner' --wsorigins "" --bootnodes 'enode://bde0739c7d9f213054575574d6e8e1df21147066a666bc939f29634dac947ad5c715a990d878264fe91a83c9fa0495911c824fe43329570c8d0510c8fa3243f2@127.0.0.1:30310' --networkid 1515 --gasprice '1' -unlock '0xbac5564cc4f7528ebb6150270ee63b3fa1641b17' --password node1/password.txt --mine 2>>node1/eth_output.log &

2.6 Startup node2

[furnace@localhost devnet]$ geth --datadir node2/ --syncmode 'full' --port 30312 --rpc --rpcaddr 'localhost' --rpcport 8502 --rpcapi 'personal,db,eth,net,web3,txpool,miner' --ws --wsaddr 0.0.0.0 --wsport 8602 --wsapi 'personal,db,eth,net,web3,txpool,miner' --wsorigins "*" --bootnodes 'enode://bde0739c7d9f213054575574d6e8e1df21147066a666bc939f29634dac947ad5c715a990d878264fe91a83c9fa0495911c824fe43329570c8d0510c8fa3243f2@127.0.0.1:30310' --networkid 1515 --gasprice '1' -unlock '0x8a6673dc85e544bce6d4876f63cfcdc4e5c8cc08' --password node2/password.txt --mine 2>>node2/eth_output.log &

Reference

1. Setup your own private Proof-of-Authority Ethereum network with Geth, https://hackernoon.com/setup-your-own-private-proof-of-authority-ethereum-network-with-geth-9a0a3750cda8
2. Command Line Options, https://github.com/ethereum/go-ethereum/wiki/Command-Line-Options
3. Clique PoA protocol & Rinkeby PoA testnet #225, https://github.com/ethereum/EIPs/issues/225
4. Clique : Discarded bad propagated block#1 when syncing #14945, https://github.com/ethereum/go-ethereum/issues/14945
5. Management APIs, https://github.com/ethereum/go-ethereum/wiki/Management-APIs
6. JavaScript Console, https://github.com/ethereum/go-ethereum/wiki/JavaScript-Console
7. go-ethereum/core/block_validator.go, https://github.com/ethereum/go-ethereum/blob/master/core/block_validator.go#L106
8. JSON RPC, https://github.com/ethereum/wiki/wiki/JSON-RPC
9. Mist browser, https://github.com/ethereum/mist
10. Ethereum wallet, https://github.com/ethereum/mist/releases
11. web3.js, https://github.com/ethereum/web3.js/
12. web3j, https://github.com/web3j/web3j
13. web3.py, https://github.com/ethereum/web3.py
14. JSON-RPC 2.0 Specification, https://www.jsonrpc.org/specification
15. Truffle, https://truffleframework.com/
16. Embark, https://github.com/embark-framework/embark
17. Populus, https://github.com/ethereum/populus
18. Ganache, https://truffleframework.com/ganache
19. clique.go, https://github.com/ethereum/go-ethereum/blob/master/consensus/clique/clique.go#L622
20. What's a ÐApp?, https://www.stateofthedapps.com/whats-a-dapp
21. Ethereum: create raw JSON-RPC requests with Python for deploying and transacting with a smart contract, https://hackernoon.com/ethereum-create-raw-json-rpc-requests-with-python-for-deploying-and-transacting-with-a-smart-7ceafd6790d9

Contributor

1. Windstamp, https://github.com/windstamp