Each lab will consist of a small problem and details of how to proceed. Each lab is intended to give every student hands-on experience with the core concepts and technologies covered during the course.  A student may concentrate, as a team member, on one technology over another for the final project, but labs are designed to give each and every student exposure to all the technologies that come into play.  You need to submit labs to the TAs for grading--see submission instructions below.  Generally, unless otherwise specified, you will have one week to complete each assigned lab.

See the syllabus for information on grading.  Turning in lab assignments on time is required, without exception, and all late deliveries will be penalized, regardless of cause.  Submit your assignments to the subversion repository according to the directions on the syllabus page.

These solutions are to be writing in the programming language prescribed by the instructions.  The reason for this is simple.  One cannot write a Java RMI solution in Haskell.  That said, you will not need to be "programming" in Java (or Python or C++ or Go) in this lab, although you will be following instructions to build Docker containers and you will run executables in these languages in their respective containers.  The goal of this lab is to get you introduced to the networking capabilities of Docker and introduced to Java RMI (Problem 1) and Google gRPC (Problem 2), so that (Problem 3) you can distribute your blockchain across multiple containers and mine from multiple containers that are all communicating in a peer-to-peer fashion.

Lab 9   Due:  See individual due dates for each Problem below (NOTE:  To receive any points for Lab 9, Problem 3 must be submitted)

CHOOSE BETWEEN DOING PROBLEM 1 or PROBLEM 2 (DO NOT DO BOTH)

You are to do either Problem 1 or Problem 2.  Both are worth 1 out of 5 points for this lab.  Everyone will do Problem 3.  Problem 3 is worth 4 out of 5 points for this lab.  Your key decision point on whether to opt for Problem 1 or Problem 2 will be whether you are working in Java and would prefer the simplicity of Java RMI network communication (Problem 1) or you wish to explore Google gRPC (Problem 2).  Note that you can still do gRPC using Java as well (and instructions are below)...so this is totally up to you.  Everyone working in a language other than Java will likely opt for Problem 2 (although there are no rules here...you can still choose Problem 1 even if  you're programming in Haskell or Go....).

Problem 1:  Java RMI and Docker Networking:  Due: 6:00 pm, Tuesday, June 11, 2019 (revised 6/10)

BACKGROUND:

WHAT YOU NEED TO DO:

There are two shell scripts that you will use to document your work with this lab (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  Contact the TA if you're working on Windows.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab and subsequent labs, and will ignore any output that we see that is not part of the lab requirements.

Create a working directory (perhaps something like "~/mpcs56600/lab7" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

STEP 1:

For this lab problem, we are going to code a little container that will accept a synchronous call over RMI to a server.  That RMI Server will echo back the string sent from the RMI Client, and return it to the RMI Client running in a separate container. 

First, make sure docker is running either on your laptop (Option I from the first lab) or in your VM (Option II from the first lab).

STEP 2:

STEP 3:

$ docker ps                      >Problem1.out
$ docker diff RMIServer         >>Problem1.out
$ docker diff RMIClient         >>Problem1.out

Problem 2:  gRPC and Docker Networking:  Due: 6:00 pm, Tuesday, June 11, 2019 (revised 6/10)

BACKGROUND:

WHAT YOU NEED TO DO:

There are two shell scripts that you will use to document your work with this lab (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  Contact the TA if you're working on Windows.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab and subsequent labs, and will ignore any output that we see that is not part of the lab requirements.

Create a working directory (perhaps something like "~/mpcs56600/lab7" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

We will be working with containers that contains Google Protocol Buffers and example gRPC code (from google) for the following language bindings:

java [https://grpc.io/docs/tutorials/basic/java.html]
C++ [https://grpc.io/docs/tutorials/basic/c.html#generating-client-and-server-code]
go [https://grpc.io/docs/quickstart/go.html#install-protocol-buffers-v3]
python [https://grpc.io/docs/tutorials/basic/python.html]

There is a quickstart guide for each language (implementing simple HelloWorld):

gRPC Guides:  https://grpc.io/docs/guides/
gRPC Tutorials:  https://grpc.io/docs/tutorials/
C++:  https://grpc.io/docs/quickstart/cpp.html
Java:  https://grpc.io/docs/quickstart/java.html
Go:  https://grpc.io/docs/quickstart/go.html
Python:  https://grpc.io/docs/quickstart/python.html

These are the simplest introductions to the various language bindings.

STEP 1:

Download the file :  grpc-lab7.tar.bz2. from /home/mark/pub/56600/src.labs/LAB.7  It's over a gig in size, so it will take a minute or so.

bunzip2 it.  (install bunzip2 if you need to).

Load it into docker:

$ docker load < grpc-lab7.tar
18f9b4e2e1bc: Loading layer [==================================================>]  129.3MB/129.3MB
a021c4ee5b3a: Loading layer [==================================================>]  281.1MB/281.1MB
7656f8e9f9d5: Loading layer [==================================================>]  182.3MB/182.3MB
ecb7d1e409dd: Loading layer [==================================================>]  664.6MB/664.6MB
cb19c93edb8e: Loading layer [==================================================>]    208MB/208MB
f6ba9235b10c: Loading layer [==================================================>]  1.886GB/1.886GB
680f7dcf7877: Loading layer [==================================================>]  691.2kB/691.2kB
bcab93a01dd1: Loading layer [==================================================>]  234.4MB/234.4MB
Loaded image: grpc-lab7:mpcs56600

Now, run:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
grpc-lab7           mpcs56600           d1e7bbaa08a1        16 hours ago        3.53GB

Now, run a new docker image to create a GRPC_SERVER container:

docker run -it --hostname grpc_server --name GRPC_SERVER grpc-lab7:mpcs56600 /bin/bash

this creates a new container based on grpc:mpcs5660 called GRPC_SERVER:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED                  STATUS              PORTS               NAMES
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         Less than a second ago   Up 3 seconds                            GRPC_SERVER

Now, exec into your new GRPC_SERVER container:

$ docker exec -it GRPC_SERVER /bin/bash
root@grpc_server:/#

Run the C++ server:
cd to /src/grpc/examples/cpp/helloworld

root@grpc_server:/src/grpc/examples/cpp/helloworld# ./greeter_server
Server listening on 172.17.0.3:50051

Now, confirm that the server is functioning by opening another terminal window, exec'ing into the same container (as above), and changing into the same helloworld cpp directory, and executing:

root@grpc_server:/src/grpc/examples/cpp/helloworld# ./greeter_client
Greeter received: Hello world

You are looking to see Hello world printed out.  Great.

Now, run a python test:

$ docker run -it --hostname grpc_client_python --name GRPC_CLIENT_PYTHON --link GRPC_SERVER:grpc_server grpc-lab7:mpcs56600 /bin/bash

If you open another terminal window on your host, and run docker ps, you should see your new python container:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         41 seconds ago      Up 52 seconds                           GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         10 minutes ago      Up 10 minutes                           GRPC_SERVER

Now, execute the following cat command:

root@grpc_client_python:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    grpc_server grpc_server GRPC_SERVER
172.17.0.3    grpc_client_python

Notice that docker has kindly added a HOSTS link to your GRPC_SERVER which is aliased as "grpc_server".  This is nice and very kind of Docker, and the magic happened because you added the  --link GRPC_SERVER:grpc_server line in your docker run command for your GRPC_CLIENT_PYTHON above.

Well, let's see if this stuff works.  Execute:

root@grpc_client_python:/src/grpc/examples/python/helloworld# python greeter_client.py
Greeter client received: Hello you

root@grpc_client_python:/#

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
5098b903220e        grpc-lab7:mpcs56600      "/bin/bash"         8 minutes ago       Up 10 minutes                           GRPC_CLIENT_PYTHON
a0cad78cfbf2        grpc-lab7:mpcs56600      "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_GO
3a597e796815        grpc-lab7:mpcs56600      "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_JAVA
c0fe154dbc9a        grpc-lab7:mpcs56600      "/bin/bash"         2 hours ago         Up 2 hours                              GRPC_SERVER

In your python container you just ran, cd to the /src/grpc/examples/python/helloworld directory, and then execute:

root@grpc_client_python:/src/grpc/examples/python/helloworld# python greeter_client.py
Greeter client received: Hello you

This is wonderful!  The python client (you should of course examine the python client code) has done this to communicate to the GRPC_SERVER in the other container:

  channel = grpc.insecure_channel('grpc_server:50051')
  stub = helloworld_pb2_grpc.GreeterStub(channel)
  response = stub.SayHello(helloworld_pb2.HelloRequest(name='you'))
  print("Greeter client received: " + response.message)

It has created a channel to the grpc_server listening on port 50051, and has created an RPC stub, and has called the server's SayHello function (remember the server is written in C++) passing it the parameter "you".  The response back from the server is:  "Hello you".  You can of course change the name from the anonymous "you" to your own name.

Ok, so far, so good.  Now let's run a Go container.  First, exit out of the GRPC_CLIENT_PYTHON container.  Then, execute:

docker run -it --hostname grpc_client_go --name GRPC_CLIENT_GO --link GRPC_SERVER:grpc_server grpc-lab7:mpcs56600 /bin/bash

et voila!  There you are in your brand new GRPC_CLIENT_GO container:

root@grpc_client_go:/#

From that other free terminal, execute:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
cb1c098fbbf9        grpc-lab7:mpcs56600   "/bin/bash"         54 seconds ago      Up About a minute                       GRPC_CLIENT_GO
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         25 minutes ago      Up 25 minutes                           GRPC_SERVER

Whoa Nellie!  What happened to the Python container????  Well, when you exited it's bash shell, the container stopped.  No matter, it's still there:

$ docker ps -a
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS                     PORTS               NAMES
cb1c098fbbf9        grpc-lab7:mpcs56600   "/bin/bash"         2 minutes ago       Up 2 minutes                                   GRPC_CLIENT_GO
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         16 minutes ago      Exited (0) 2 minutes ago                       GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         26 minutes ago      Up 26 minutes                                  GRPC_SERVER

If you ever want to start it again, all you have to do is execute:

docker start GRPC_CLIENT_PYTHON in a terminal window and you're back in it.

$ docker start GRPC_CLIENT_PYTHON
GRPC_CLIENT_PYTHON

From another window:
$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
cb1c098fbbf9        grpc-lab7:mpcs56600   "/bin/bash"         3 minutes ago       Up 3 minutes                            GRPC_CLIENT_GO
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         18 minutes ago      Up 6 seconds                            GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         27 minutes ago      Up 28 minutes                           GRPC_SERVER

Then, in another free terminal window you can just exec into the running python container:

$ docker exec -it GRPC_CLIENT_PYTHON /bin/bash
root@grpc_client_python:/#

Ok, back to our Go container.  Get back into the window where you executed the docker run command for GRPC_CLIENT_GO.  It will have the prompt "root@grpc_client_go:/#".

Now, let's execute the Go client against our C++ server.  First, in the Go container, type once more:

root@grpc_client_go:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    grpc_server grpc_server GRPC_SERVER
172.17.0.3    grpc_client_go

There she is!  grpc_server again!  Excellent. 

Now, let's run the Go client against the C++ server:

Make sure we have our GOPATH set appropriately (should be...):
env |grep go
See:
PATH=/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOPATH=/root/go

If you see your PATH includes /usr/local/go/bin and your GOPATH set, you're "good to go".

cd to the go source directory:
cd ~/go/src

Now, do an "ls" on that directory, and you'll see several directories, including google.golang.org/

That directory contains the go language examples.

Execute:
ls google.golang.org/grpc/examples/helloworld

Here, you'll see the greeter code.  There's a little change we need to make.  cd to the google.golang.org/grpc/examples/helloworld/greeter_client directory:

root@grpc_client_go:~/go/src $ cd google.golang.org/grpc/examples/helloworld/greeter_client

Then, change this code in main.go:

const (
        address     = "localhost:50051"
        defaultName = "world"
)

to this:

const (
        /*address     = "localhost:50051"*/
        address     = "grpc_server:50051"
        defaultName = "world"
)

This way our go program will be hitting our C++ server in the other container.

Now build the go client:
go generate google.golang.org/grpc/examples/helloworld/greeter_client

If all goes well (this is Unix of course), you'll see nothing.  That's good news.

Now, you can execute the go client:

root@grpc_client_go:~/go/src/google.golang.org/grpc/examples/helloworld/greeter_client# go run main.go
2018/07/23 14:14:52 Greeting: Hello world

Hot diggity dog.  It worked.  Pro.

Now, we'll run a Java client.  Exit out of your Go container, and then execute:

docker run -it --hostname grpc_client_java --name GRPC_CLIENT_JAVA --link GRPC_SERVER:grpc_server grpc-lab7:mpcs56600 /bin/bash
root@grpc_client_java:/#

root@grpc_client_java:/# cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
ff00::0    ip6-mcastprefix
ff02::1    ip6-allnodes
ff02::2    ip6-allrouters
172.17.0.2    grpc_server grpc_server GRPC_SERVER
172.17.0.3    grpc_client_java

Note the actual IP address for the grpc_server (above it is "172.17.0.2").  You'll need that in a minute.

Now, we've got to go into the Java client and modify the server address (similar to what we did in our Go example above). 

Edit the file:  HelloWorldClient.java

In the main() method, we need to update the server IP address with the IP address for the grpc_server from the /etc/hosts file, so modify this line:

    HelloWorldClient client = new HelloWorldClient("localhost", 50051);

to this:

    //HelloWorldClient client = new HelloWorldClient("localhost", 50051);
    HelloWorldClient client = new HelloWorldClient("172.17.0.2", 50051);

Of course we're assuming your grpc_server address is "172.17.0.2".  You'll of course change it to whatever YOUR server's IP address is.

Now, cd back into your ~/java/grpc-java/examples directory, and rebuild the Java code:

Now, rebuild java client:
./gradlew installDist
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details

BUILD SUCCESSFUL in 12s
12 actionable tasks: 12 up-to-date

You're looking for that happy green BUILD SUCCESSFUL line.

Now, run the java client:
./build/install/examples/bin/hello-world-client

You should see something like this output from the java client:

root@grpc_client_java:~/java/grpc-java/examples# ./build/install/examples/bin/hello-world-client
Jul 23, 2018 2:40:01 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Will try to greet world ...
Jul 23, 2018 2:40:02 PM io.grpc.examples.helloworld.HelloWorldClient greet
INFO: Greeting: Hello world

Whatever the date, it's that last "INFO" line that's important.  That's the greeting from the C++ server.

Feel free to examine the Java source code that is generated in the following files and directories:
./src/main/java/io/grpc/examples/helloworld/*
./build/generated/source/proto/main/grpc/io/grpc/examples/helloworld/GreeterGrpc.java

This latter file is the java stub genereated by the protocol compiler.

Now with all your containers from this problem running (start them again if you've stopped them), execute the following from a prompt on your host system (NOT in a container):

$ docker ps >Problem2.out
$ docker diff GRPC_SERVER >>Problem2.out
$ docker diff GRPC_CLIENT_PYTHON>>Problem2.out
$ docker diff GRPC_CLIENT_JAVA >>Problem2.out
$ docker diff GRPC_CLIENT_GO >>Problem2.out

Exit out of your Java container. 

You can now stop your C++ server and exit out of the GRPC_SERVER container.

Ensure you have your terminal logs and submit these. You may need to type exit to stop one or more runscript sessions.

REMEMBER:  At any point now you can simply start and stop your containers, so, for example, if you wanted to restart your Java container, and then stop it again,  you'd just execute:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up 45 minutes                           GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_SERVER

$ docker start GRPC_CLIENT_JAVA
GRPC_CLIENT_JAVA

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
61095b85983b        grpc-lab7:mpcs56600   "/bin/bash"         21 minutes ago      Up 5 seconds                            GRPC_CLIENT_JAVA
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_SERVER

$ docker stop GRPC_CLIENT_JAVA
GRPC_CLIENT_JAVA

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
7f07214e3521        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_CLIENT_PYTHON
7e298f1b6e59        grpc-lab7:mpcs56600   "/bin/bash"         About an hour ago   Up About an hour                        GRPC_SERVER

Same with any of the other containers.

This concludes Problem 2.
 

Problem 3:  Distributing your Blockchain using Docker Networking and Docker Compose:  Due: 6:00 pm, Tuesday, June 11, 2019 (revised 6/10)

BACKGROUND:

Like all programming problems, learning a new technology is not an exercise in reading but rather an exercise in thinking and typing.  This lab is designed to give you  hands-on experience in some fundamental skills involved in Docker containerization.  You will generally find the References section below helpful in addition to the required and recommended reading.  When we talk about "Docker", we are talking specifically about the Stable Community Edition of Docker, which is the version we will be using in this class.  The Stable Community Edition provides the basic container engine and built-in orchestration, networking, and security.

In this lab problem, you will distribute your blockchain across multiple Docker containers and have them all inter-communicate in terms of broadcasting transactions and mining blocks. 

WHAT YOU NEED TO DO:

First, let's just pause for a moment and understand where you are and where you are going.  Right now, through Lab 8 you have a program (or two) that (a) generates new transactions and adds them to your blockchain and (b) mines them.  All you're going to do here is distribute this program so that the same program runs on multiple Docker containers ("full node peers"), so that multiple transactions are coming from multiple containers, and multiple containers are mining the transactions they hear about.

Make sure you have read and understood the implications of Chapter 8, The Bitcoin Network, of Antonopoulos.

As usual, anything not specifically prescribed or proscribed in the following instructions is left to your discretion (aka creative license) as to exactly how to implement.

Details:

The first thing you will need to decide is what distributed communication mechanism you are going to use.  Of course, you can choose to simply work with raw sockets.  This will prove engaging but also challenging, as you may have to worry about endianness, structural boundaries, string termination, communication faults, and all sorts of other things that may come into play.  That said, communicating over raw sockets requires nothing other than your programming language and its libraries.

The other options include Java RMI, Google gRPC, Windows Communication Foundation WCF/.NET, or any other mechanism you wish (you may NOT choose a shared file system such as NFS, etc., where you only really have a "single" blockchain that every peer jointly "sees") that will allow for the exchange of serialized "string" of data over the network between containers.  The exact mechanism of data sharing is entirely up to you, but in general, you will need to be able to serialize your transactions into some form of "byte stream", and transmit that stream between containers.    Using a mechanism such as gRPC or RMI will go a long way towards simplifying the handling of the serialization of your data.  Note your mileage may vary when trying to serialize an actual Transaction object as opposed to a simple string.  This will be left to you to ponder.  That said, the only thing you will really need to transfer are Transactions and Blocks and a few handshaking structures.

Vocabulary:  When we speak of a "Node" below, we mean a Docker container.  When we speak of a "Full Node", we mean a docker container running an instance of your Blockchain Mining Software you produced for Lab 8, now newly distributed (as described below)

STEP 1 (Time T₀):

Figure out what default port your bitcoin network will establish initial communication over, ie, the handshake port.  You can use any ephemeral PORT, such as 58333 (bitcoin's default for Mainnet is 8333), or 12345, etc.  Your choice. 

In one docker container, run a docker container whose "--name" is "DNS_SEED", which will contain a program (Registrar) which will listen on the ephemeral PORT you have chosen (e.g., 58333, 12345, etc.), i.e., whose only job is to listen for incoming connections (via whatever technology you are using for network connections--raw sockets, RMI, WCF, gRPC, etc.).  For your docker image, you are free to chose any docker image base that makes sense to you, an ubuntu-based image is always a good bet.



 

STEP 2 (Time T₁):

Each Full Node runs in a docker container that is linked to the DNS_SEED node in its docker run command (--link DNS_SEED).

The Registrar program running in the DNS_SEED node does not actually have to be running your blockchain mining software (a Full Node)...it can be a simple network server (running RMI, gRPC, WCF, raw socket) whose job it is to listen (on your ephemeral PORT) for incoming connections and to return a list of {0...1} "Full Nodes" on your coin network that have already registered with the Registrar program running on the DNS_SEED node.  Note that the first Full Node that registers (at time T₁) will receive null/0 nodes back from the DNS_SEED; however, its own IP address will have been registered with the DNS_SEED.  You can think of the DNS_SEED node as acting as if it is a DNS service.  A Full Node (such as Node A) as we are describing it is a distributed version of your blockchain mining software you developed for Lab 8. 




The first network call a newly-launched Full Node makes is a call to the DNS_SEED's register() function/method.  The Full Node initializes and sends (via the register() function) a serialized structure/object [Registration] that contains the following information:

struct/class Registration {
nVersion:  The version of your mining software, defaults for this Lab to 1.
nTime:  The current time on the joining node
addrMe:  The IP address of the joining node (e.g., "172.17.0.3" on your docker subnet)
}

Upon receipt, the DNS_SEED records this information in a list, especially noting the IP address of the registering Full Node in an internal datastore (can be non-persistent, i.e., an in-memory list or an actual database (e.g., MySQL, SQLExpress, etc., ... your choice).

Remember that it is the job of the Registrar to return a list of zero or one nodes ({0...1} "Full Nodes") that have previously registered with it (the Registrar always returns the latest registered node in its list or null/0 if it has no nodes in its list).  In our example, as of T₁, this is the very first Full Node registering with the DNS_SEED, so the Registrar running on the DNS_SEED will return zero or a null list (your call on what that means) indicating that this registering node is the first node to register.

After a Full Node (let's call this Node A...you can consider it anything you wish) first communicates with the DNS_SEED node and is registered, each Full Node launches its distributed blockchain mining server which begins to patiently listen for network calls to its handshake() function/method (explained below). 

STEP 3 (Time T₂):

As time goes by, a new Full Node (let's call this Node B) will run in a new docker container (again, always linked to DNS_SEED).  This new Full Node will do the same thing as the previous Full Node:  it will first register (Step 1 in the drawing) with the DNS_SEED server (as described above).



Once the newly-launched Full Node has registered (Step 1 in the drawing) with the Registrar running in the DNS_SEED (in our example this is Node "B"), if the node gets a non-null response from the DNS_SEED (which in this example it would, because the Registrar running on DNS_SEED would have returned the IP address for Node A, its last-registered node), the new Full Node ("B") will then send a handshake() message (i.e., call the handshake() function/method) on Node A's IP address it received back from the Registrar running in the DNS_SEED (Step 2 in our drawing), seeking to communicate with that IP address whose server (distributed blockchain mining software) is listening on port (say) 58333.  The handshake() function/method takes in a Handshake class/structure from the calling peer that resembles the Registration structure/class described above, viz.:


struct/class Handshake {
nVersion:  The version of your mining software, defaults for this lab to 1.
nTime:  The current time on the calling node
addrMe:  The IP address of the calling node (e.g., "172.17.0.5" on the docker subnet)
bestHeight:  The height of the blockchain tip as this node knows it (can be the height of the genesis block)
}

Note the addition of the "bestHeight" field, which is the height of the tip of Node B's blockchain (can be 0 if it only has the genesis block).  On receipt of a handshake() call, the receiving Full Node ("A") will record the IP address of the caller (Handshake.addrMe which will be Node B's IP address) in a list of its known peers (your call as to that form/structure).  The list of known peers is simply a list of IP addresses a Full Node maintains of other Full Nodes that have "shook hands" with it, i.e., called its handshake() method and provided their IP address via Handshake.addrMe.

In returning from the handshake() call, the receiving Full Node (the node whose Handshake() function was called, in this case, Node "A") will return to the calling node ("B") a list of all the nodes that it knows about, i.e., it's list of known peers, (at this time, that list will contain only the IP of node "B", because Node A does not know of any other nodes other than the handshaking node "B").  We will leave it to you to determine the structure of the returned list of known peers.  Node B will recognize its own IP returned from node A's list of known peers, and will not act further on it's own IP address.

STEP 4 (Time T₃):

Node A now has a single known peer:  Node B.  As time goes by, a new Full Node (let's call this Node C) will run in a new docker container (again, always linked to DNS_SEED).  This new Full Node will do the same thing as the previous Full Nodes:  it will first register with the DNS_SEED server (as described above, Step 1 in our drawing).

Once the new Full Node has registered with the DNS_SEED, if a node gets a non-null response from the DNS_SEED (which in this example it would, because the DNS_SEED would have returned the IP address for Node B (we can assume that the DNS_SEED simply returns it's "latest" addition to the network), the new Full Node ("C") will send a handshake() message (i.e., call the handshake() function/method) on the IP address it received back (it only receives one IP node back from the DNS_SEED), seeking to handshake() with that IP address whose server (distributed blockchain mining software) is listening on port (say) 58333, as described above (Step 2 in our drawing), which in our example, would be node "B". 




Once the third newly-launched Full Node has registered with the DNS_SEED and has conducted a handshake with Node B (from which it learned of Node A because node "B" returned its list of known peers, which includes node "A"), the newly-launched Full Node ("C") will also handshake with any other peer nodes it has learned from the node it just shook hands with (in our case, Node "B"), and thus it will call the handshake() function/method on Node "A" providing Node A it's IP Address (that of Node "C").

Note that a Full Node will only call every given node's handshake() function/method once...i.e., it will maintain knowledge of which nodes it has already shaken hands with (your call on how to handle this...i.e., a second list of handshakes or a "handshake" flag on the list of known peers, etc.).  If a Full Node discovers (via known peers) a Full Node with which it has not already shaken  hands, it will call that node's handshake() function/method.  In this way, all nodes will have eventually shaken hands with each other and it is never the case that a given node either (a) calls it's own handshake() function/method or (b) calls another Full Node's handshake() function/method more than once.

STEP 5 (Time T₄):

Once the third newly-launched Full Node has registered with the DNS_SEED and has conducted a handshake with Nodes B and now Node A (Node C learned of Node A from it's handshake with Node B), each node will now know of a minimum of two other nodes (known peers) on the network.  Upon learning of the existence of Node C, Node A will in turn call Node C's handshake() method, seeking to discover Node C's list of known peers, which it will add to its own list of known peers.

Note that in our example, we are not limiting the number of handshakes with other nodes because we are intentionally keeping it simple.  In a real blockchain implementation, any given node would not seek to know of "all" nodes on the network, just a select subset.  We are not now concerned with minimizing network traffic.

STEP 6 (Time T_5..n):

At this point, each node's mining activity can begin (your call on how this state is recognized...).  The distributed mining process will entail the simulated generation of new Transactions (as you have been doing in previous labs) and the attempted mining of a new block (as you did in Lab 8).  As each new mining node "discovers" a new transaction (periodic generation...your mileage may vary and you may feel free to tweak this, but start out with each node discovering a new transaction about every 3-5 seconds...try to make this random by alternatively sleeping within the bounds of 2-5 seconds--see python example below), it adds the newly-created transaction to its own memory pool and then broadcasts that transaction to all the other nodes on the network it knows about (by calling newTransactionBroadcast() on the other Full Nodes). 

So for example, if Node C "discovers" (generates) a new transaction, it will broadcast that transaction to the other nodes using something like the following structure/class:

struct/class NewTransaction {
    serialized transaction data
}

Each Full Node is listening for NewTransactionBroadcast() calls on your ephemeral PORT (or other port depending on your network solution, ie., RMI, WCF, etc.), and when a node receives a new transaction from the network, it checks first if the new transaction is already in it's memory pool of transactions, and if it's not, it will add the new transaction to its working memory pool of transactions it is adding to the block it is trying to mine.  For this lab, we will assume that all transactions broadcast are valid transactions.  The node then will propagate (gossip) the new transaction by calling NewTransactionBroadcast on all nodes in its list of known nodes.  A given node will never broadcast a given transaction more than once.

The difficulty you will use in your mining for this lab will be 0x1e200000 which will average a successful new block generation with the correct nonce of between 1 and 17 seconds (μ ~ 7 seconds with one miner).  0x1e200000 will yield a hex target of 0x0000200000000000000000000000000000000000000000000000000000000000 (decimal: 220855883097298041197912187592864814478435487109452369765200775161577472).

Once a new block is mined by a Full Node, it will immediately broadcast this new block (with all of its transaction IDs) as a new structure or class:

struct/class NewBlockMined {
    serialized block data of successful nonce (which will include Block and Header)
}

as:  NewTransactionBroadcast(myNewBlockMined)

As other nodes receive the newly-mined block solution.  For this lab, we will assume that all new blocks  broadcast are valid.  The node then will propagate (gossip) the new block by calling NewBlockBroadcast(newBlock) on all nodes in its list of known nodes.  A given node will never broadcast a given block more than once to each known node. Upon receipt of a new block broadcast, each mining node will simply add the new block to its blockchain.  Remember, each new Block is added on (bound to) to a specific prior block off of which the miner is working, which may be the genesis block in the case of the first mined block. 

Once a miner publishes a newly-mined block, each miner (including the miner that produced the correct result) will sleep for a random number of seconds (as integers) between 0 and 3 seconds, i.e., in the range {0..3}.  As an example, the python command print(random.randint(0,3)) will produce this result.  This will reduce the chance that miners will all produce blocks at "about the same time", since we assume all Full Nodes are running on the same physical processor(s) (as all containers are likely running on the same physical machine) and they will tend (through the round-robin load-balancing of the OS scheduler) to produce results at about the same time.  Adding this randomness will reduce this likelihood and produce more "clean winners", which should reduce the chance of forks, which we would like to avoid in this lab.

You may feel free to add further nodes if you wish, this is up to you if you're curious, but not required.  A Full Node is only to connect to a MAX of two other nodes in your network.

FINAL STEP:

Run it and debug it.

You will likely discover "timing-related" or "order-related" issues as you begin to run and debug your distributed solution.  You will need to think about issues, and how you will handle them, including:

What if I receive a transaction I already know about?  (ignore it?)
What if I'm mining a block and I receive a new transaction?  (add it and reset your nonce to 0?)
What data structure will I use to hold my "working set" of unspent transactions?  (lists and maps are always a good choice)
How will I serialize my transactions for networked communication?  (language-dependent)
How will I serialize my newly-mined blocks for networked communication?  (network-framework dependent, ie, Java RMI, Windows C# WCF, python sockets, etc.)
How do you handle multiple methods/functions listening on a single port (e.g., 58333)?  Or do you establish each listener on a separate port (i.e., use multiple ports)?  This may depend on the method of communication you choose (RMI, gRPC, WCF, sockets, etc.).  You are completely free to make this decision as you see fit...whatever is easier.
How will you handle forks which might occur?  You don't have to worry about handling forks unless you are encountering them.
How do you govern the "gossiping" between the nodes...that is to say, how do you make sure that transactions and blocks are not re-broadcast over and over again?

To prepare for submission, create a tarball of your DNS_SEED container and one of your Full Nodes for Problem 3:

Commit a new image of both using a command similar to (use of course the actual names you've used for this problem below):

docker commit DNS_SEED dns_seed:lab7
docker commit FULL_NODE full_node:lab7

Then, save the images to a tarball:

$ docker save -o DNSSeed.tar dns_seed:lab7
$ docker save -o FullNode.tar full_node:lab7

Then, zip them up:

$ bzip2 DNSSeed.tar
[be patient...may take a few secs]
$ bzip2 FullNode.tar
[be patient...may take a few secs]

Now scp your two bzip'd files to your personal directory on the cluster under /stage/MPCS56600/[CNETIDHERE]. This directory has been created for you to use for large files. Here's an example command:

# example cluster scp command
$ scp whatever.tar.bz2 [CNETIDHERE]@linux.cs.uchicago.edu:/stage/MPCS56600/[CNETIDHERE]/

Submit source code and transcripts that you created as usual into your lab7 repo directory as described below but please do not submit container tar files to your Subversion repo. Use your cluster directory instead.

References:

You may find the following references helpful (in addition to the links from previous labs):

General gRPC Tutorial Links

java [https://grpc.io/docs/tutorials/basic/java.html]
C++ [https://grpc.io/docs/tutorials/basic/c.html#generating-client-and-server-code]
go [https://grpc.io/docs/quickstart/go.html#install-protocol-buffers-v3]
python [https://grpc.io/docs/tutorials/basic/python.html]

gRPC Guides:  https://grpc.io/docs/guides/
gRPC Tutorials:  https://grpc.io/docs/tutorials/
C++:  https://grpc.io/docs/quickstart/cpp.html
Java:  https://grpc.io/docs/quickstart/java.html
Go:  https://grpc.io/docs/quickstart/go.html
Python:  https://grpc.io/docs/quickstart/python.html

Java RMI:

Java RMI and Object Serialization FAQ
Tutorial's Point RMI Tutorial
Oracle's RMI Getting Started Guide
JavaTpoint's RMI Guide

Windows WCF:

WCF Tutorial
Microsoft Tutorial
Code Project's Beginner's Guide to WCF

Python-related RPC mechanisms:

Python Pickle:
Python Docs Pickle
Python UsingPickle
Python Pickle DataCamp

Python JSON-RPC:
Python PyPi JSON-RPC
Python JSON-RPC Github example (not tested...)
Python JSON-RPC on jsonrpc.org
Another example

Other Examples of RPC mechanisms:

Go:
Go example
Medium's article

Haskell:
The World's Dumbest RPC Example in Haskell (untested)
Hackage example of JSON-RPC in Haskell


General Docker Tutorial Links

Docker Cheat Sheet
Learn Docker in 12 Minutes Youtube
Demystifying Docker Youtube
TutorialsPoint:  Docker Tutorial for Absolute Beginners
Docker Overview
Ubuntu package commands

Submitting:

Use the folder named "lab9" in your Subversion repository. See the syllabus for more info about submission using Subversion. Upload your Lab 9 code and any supporting materials to the repo.  Please include a README text file to explain what parts of the work you are submitting, where the different parts of the work are located and please provide a little info on how to compile and run your code.