Lab 6 Details for MPCS 56600

Each lab will consist of a small problem and details of how to proceed. Each lab is intended to give every student hands-on experience with the core concepts and technologies covered during the course.  A student may concentrate, as a team member, on one technology over another for the final project, but labs are designed to give each and every student exposure to all the technologies that come into play.  You need to submit labs to the TAs for grading--see submission instructions below.  Generally, unless otherwise specified, you will have one week to complete each assigned lab.

See the syllabus for information on grading.  Turning in lab assignments on time is required, without exception, and all late deliveries will be penalized, regardless of cause.  Submit your assignments to the subversion repository according to the directions on the syllabus page.

You may write these solutions in any programming language of your choice.  Our suggestion is now is not the time to learn a new programming language along with the concepts themselves.  So our suggestion is to use whatever programming language you  know best.

Lab 5   Due: 5:00 pm, Thursday, August 2, 2018

Problem 1:  Docker Introduction: 

BACKGROUND:

Like all programming problems, learning a new technology is not an exercise in reading but rather an exercise in thinking and typing.  This lab is designed to give you  hands-on experience in some fundamental skills involved in Docker containerization.  You will generally find the References section below helpful in addition to the required and recommended reading.  When we talk about "Docker", we are talking specifically about the Stable Community Edition of Docker, which is the version we will be using in this class.  The Stable Community Edition provides the basic container engine and built-in orchestration, networking, and security.

In this lab, you will practice creating Docker images and containers. 

WHAT YOU NEED TO DO:

There are two shell scripts that you will use to document your work with this lab (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  Contact the TAs if you're working on Windows.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab, and will ignore any output that we see that is not part of the lab requirements.

Once you run the script, you should see something like:

Script started, output file is typescript

Do NOT accidentally exit out of this shell until you are instructed to at the very bottom of this lab.

Create a working directory (perhaps something like "~/mpcs56600/lab6" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

STEP 1 - Run everything directly on your laptop:

We recommend that you run everything directly on your laptop or desktop.  In general, you will want at least 50 Gig of free disk space, and at least 8 Gig of memory on your laptop, and a minimum of 2 cores, 4 or more being ideal.  If your laptop or workstation cannot satisfy these minimum guidelines, talk to John Hadidian-Baugher and he will be able to give you guidance.  Note that trying to do this work on the Linux Cluster in the MPCS labs may prove less than ideal, as the available diskspace to students is based on a quota.  If you have the space, you may wish to install and run docker and all images and containers directly on your laptop.  Docker is available for most platforms, including MacOS, Windows, and Linux.  This is a relatively easy option and straightforward.  The only downside to this option is that your 51212 development environment is not "contained" and separate from your working laptop environment as would be the case in Option II below.  If you choose to go this route, do the following:

1.  Download and Install docker directly onto your laptop

To install docker directly on a Mac, begin here to install the Stable version.  I would suggest following the instructions on installing the Docker Application natively (aka "Docker for Mac"), as opposed to using the older Docker Toolbox.  You may find a discussion of the differences here.  To install docker directly on Windows 10, begin here to install the Stable version.  Docker for Windows requires 64bit Windows 10 Pro with Hyper-V. Take a look here for more information on requirements for Windows.  If you prefer to install Docker on Linux, you may, as Ubuntu, CentOS, RedHat, etc. are variously supported.  See here for instructions for Ubuntu (but see next sentence!) and here for instructions for RedHat (Fedora).  Further instructions for all platforms can be found here.  NOTE:  If you are installing on ubuntu,  you will find these simple steps much easier:

$ sudo apt-get update
$ sudo apt-get install docker.io
$ sudo service docker status [this will verify that docker has been installed and is running]
$ sudo adduser USERNAME docker [replace "USERNAME" with your own ubuntu username]

Once you have installed docker, run the docker "Hello World" container to verify the sanity of your installation (depending on OS and environment, you may need to manually start Docker (i.e. /Applications/Docker.app, etc.) or even reboot your Windows machine.  Here is an example of what you should see (on MacOS) when you run the Hello World exmaple (you type the command "docker run hello-world"):

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
5b0f327be733: Pull complete
Digest: sha256:1f19634d26995c320618d94e6f29c09c6589d5df3c063287a00e6de8458f8242
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
Visit:   https://cloud.docker.com/
For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

The key is to see that lovely "Hello from Docker!" output (although yours probably won't be in purple!).  Once you see that, congratulations, you're off to the races and may proceed.  Pay no attention to the "Unable to find image" warning, that's normal and par for the course, because that's the first time you ran hello-world and no image was local in your local repository, so it was downloaded from the docker hub.

STEP 2:

Create a personal docker hub ID and repository

The next step is to get real...that is, join the docker community.  Go here and sign up on the Docker Hub and create your own Docker ID.  Signing up on the hub will give you space on the docker hub where you can store your own working containers, and leverage more than 100,000 containers and images that other developers have already created (hello-world is one). 

STEP 3:

General intro to images and containers with a few examples

Make sure you have your runscript working before continuing.

For this first lab, we are going to simply run a few commands and make sure that they work, which will also give you a taste of working within Docker.  First, you will need to document your exercises, and to do this, you will use a Unix program called script.  There are two shell scripts that you will use (that utilize the script command), one if you're working on linux, and the other if you're working on MacOS.  If you're running Windows, your only option will be to either (a) start a new cmd window, perform all your actions in that window, and then, when finished, select all and then copy and paste the window output into a file and submit that file, or (b) install cygwin and run the script command in cygwin.  We will be looking to see if you have successfully run all the commands required in the lab.  Note that you may "play around" further with options and other commands, even as you work through the lab.  Therefore if your script output shows additional commands etc., that's perfectly fine (in fact it's great!).  We will simply be grading the subset of required commands that you are to follow when working through the this lab and subsequent labs, and will ignore any output that we see that is not part of the lab requirements.

First, make sure docker is running either on your laptop or in your VM (if you are using one).  If you are working in a VM, do the following commands inside your Ubuntu VirtualBox VM.

Create a working directory (perhaps something like "~/mpcs56600/lab6" and in that directory type either runscript.linux.sh or runscript.mac.sh.  That will launch a new shell (in your same window), but all  your commands and output will be recorded in a file with an extension of "*.out".  Once you are finished with Step 3 of this lab, simply type "exit" or press "^d" and you will exit the subshell and your activity will be saved in the script output.  Your script output will be saved in a file with your login id and a date-time stamp.  The filename will look something like "mark.Tue.Sep.19.17-59-26.CDT.2017.out".  Your userid and time stamp will of course be different.  This is the file you will submit for grading per the submission instructions below.

Ok, the first thing you're going to do is type "docker".  Docker will print out it's help file, which gives the various flags and commands you can pass to docker.  This is a fairly robust list of options, but it will give you an idea of the wealth of things Docker can do.  Beside each of the flags and commands, you will see a short description of what each flag and command does.  For example, beside the "exec" command, Docker prints out "Run a command in a running container."  Thus, the "exec" command can be used to run a container.  We will talk about images and containers in class.

Now type "docker exec --help".  Docker will print out the help for the "docker exec" command.  This will list the Usage pattern along with all the options and will again repeat what the command does, i.e., "Run a command in a running container."  You can pass in the --help option with any docker command.  Try getting help on the run command:  type "docker run --help". 

Now let's get to the meat of it.  type "docker images".  When you give the docker images command, it will print out all the Docker images that you have downloaded locally into your repository.  In your case, you will likely have only one image, something like this:

$ docker images
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
hello-world                                  latest              05a3bd381fc2        7 days ago          1.84kB

This shows that in your local repository you have an image called "hello-world" and its Tag is "latest" and its Image ID is 05a3bd381fc2 and the size if 1.84 kilobytes.  We will talk about tags and Image IDs in class. 

Now that we've seen our Docker images, let's look at some containers.  Type "docker ps -a".  This will list all of the instantiated containers you have created (yes, even without your  knowing it).  When I run that command, I see this:

$ docker ps -a
CONTAINER ID        IMAGE                   COMMAND          CREATED              STATUS                          PORTS      NAMES
ef60e564f68e        hello-world             "/hello"         20 seconds ago       Exited (0) 19 seconds ago                  vibrant_benz
7b06f097a37a        hello-world             "/hello"         About a minute ago   Exited (0) About a minute ago              suspicious_yonath
2c377206d2d1        hello-world             "/hello"         18 hours ago         Exited (0) 18 hours ago                    serene_edison
45406c7c504a        hello-world             "/hello"         18 hours ago         Exited (0) 18 hours ago                    youthful_feynman
17342cabc340        hello-world             "/hello"         18 hours ago         Exited (0) 18 hours ago                    thirsty_shaw

This is a listing of all the containers I have executed.  Wow, that's a lot of containers.  Every time I execute the docker run command, I get a new container!  How wonderful.  It even gives my new container an interesting name, such as youthful_feynman.  It also lists the Container ID for each container, the Image that the container is based on ("hello-world"), and when I created it, it's exit status (or if it's still running), any ports the container has bound (we'll see this later), and the container's working Name.  Note that I know that none of the containers is currently running.  I know that because they've all exited.  I can also type "docker ps" by itself.  When I do this, I'm asking Docker to print out all running containers I have.  There are none, so that command wiill not print anything out if I don't have any currently-running containers.  It prints out something like this:

$ docker ps
CONTAINER ID        IMAGE                   COMMAND          CREATED              STATUS                          PORTS      NAME

Well, great.  Let's get a real image down that can actually run for a while, why don't we?  Type the following command:  "docker run -it ubuntu:14.04 /bin/bash".  Now be careful here and notice your NEW PROMPT:

$ docker run -it ubuntu:14.04 /bin/bash
Unable to find image 'ubuntu:14.04' locally
14.04: Pulling from library/ubuntu
bae382666908: Pull complete
29ede3c02ff2: Pull complete
da4e69f33106: Pull complete
8d43e5f5d27f: Pull complete
b0de1abb17d6: Pull complete
Digest: sha256:32c87496025f5cdc5e56b5e5393a4df102b14dd4928f702b2019b18faf1ec48a
Status: Downloaded newer image for ubuntu:14.04
root@0fc3e8b45816:/#

Your prompt is a tad different.  It appears your username is now "root" and your prompt is a root #.  Congratulations.  You are IN the ubuntu container.  Which container?  In the container with the Container ID of 0fc3e8b45816.  You can verify all this by bringing up a separate xterm window and running the following command:

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
0fc3e8b45816        ubuntu:14.04        "/bin/bash"         3 minutes ago       Up 3 minutes                            determined_hawking

There's your Container ID listed 0fc3e8b45816.  It's status is "Up 3 minutes".  That means your container is running!  You can play around all you want in your new Ubuntu container.  Play around by typing a few commands at the # prompt, such as ls -la, ps -ef, echo $SHELL, df, date, etc.  Try running "apt-get update" now, which will update your ubuntu (debian) packages to the latest and greatest.  Amazing.  You just modified the contents of your container.  Now type "exit" or press "^d" and you'll be back at your original $ prompt.  Try typing "docker ps" again.  Notice nothing is currently running.  That's because you exited your container, and it stopped running the /bin/bash shell.  Type "docker ps -a" again.

$ docker ps -a
CONTAINER ID        IMAGE            COMMAND          CREATED             STATUS                          PORTS       NAMES
0fc3e8b45816        ubuntu:14.04     "/bin/bash"      10 minutes ago      Exited (0) About a minute ago               determined_hawking
ef60e564f68e        hello-world      "/hello"         2 hours ago         Exited (0) 2 hours ago                      vibrant_benz
7b06f097a37a        hello-world      "/hello"         2 hours ago         Exited (0) 2 hours ago                      suspicious_yonath
2c377206d2d1        hello-world      "/hello"         20 hours ago        Exited (0) 20 hours ago                     serene_edison
45406c7c504a        hello-world      "/hello"         20 hours ago        Exited (0) 20 hours ago                     youthful_feynman
17342cabc340        hello-world      "/hello"         20 hours ago        Exited (0) 20 hours ago                     thirsty_shaw

STEP 3:

Execute the following commands when instructed to do so.  In another terminal window, create a text file called responses.txt into which you will write your responses when asked to do so in each RESPOND section below.  Feel free to copy and paste output from any commands into your response file. 

How do we launch a container and keep it running?  Basically there are two ways:

pass the -it flags to either docker run or docker exec and launch a shell (/bin/bash):

EXECUTE:  docker run -it ubuntu:14.04 /bin/bash

Now, exit your container by either typing exit or pressing Ctrl-d:

EXECUTE:  exit [or press ^d]

RESPOND:  What did that docker run command you just executed above do?
   
Now, run the image and pass it the –id flag (daemon interactive):

EXECUTE:  docker ps

EXECUTE:  docker run -id ubuntu:14.04

EXECUTE:  docker ps

RESPOND:  What do you see and what did you learn from this?

EXECUTE:  docker exec -it f62568d02b31 /bin/bash
[of course you will use the appropriate container id (or name) from the docker ps command for YOUR container id...NOT f62568d02b31 above.  You can use docker ps [-a] to find this id]

Now, exit out of your container by pressing either ^d or typing "exit".

Ok, great, but notice every time we run a docker images, we seem to be “collecting” containers (you can see them with "docker ps -a")....wouldn’t it be nice to be able to save and work in a single container?  We can.  First, let’s stop our Ubuntu container:

EXECUTE:  docker stop f62568d02b31
[of course you will use the appropriate container id from the docker ps command for YOUR container id...
wait for it, may take a few seconds to shut down...]

Let's now erase all the ubuntu:14.04 containers we've amassed so far:

EXECUTE:  docker rm $(docker ps -a |grep 14.04|awk '{print $1}')

EXECUTE:  docker ps -a

RESPOND:  What just happened?  What did the docker rm command do?

EXECUTE:  docker run -id --name [youruserid]_ubuntu ubuntu:14.04
[supply your own usesrID above in the command]

RESPOND:  What just happened?  Is your userid image currently running in a container or not?  How do find out if it's running?

EXECUTE:  docker exec -it [youruserid]_ubuntu /bin/bash

Ok, great, now we’re in our container.  Now what?  Well, let’s update the Ubuntu operating system with the latest release:

[execute these commands INSIDE your Ubuntu container...]

EXECUTE: uname -a   

EXECUTE:  apt-get update   

RESPOND:  What just happened?
   
Now download the vi editor:

EXECUTE:  apt-get install -y vim

EXECUTE:  which vim

EXECUTE:  cd /; ls -la; whoami

RESPOND:  What did you learn from executing those commands?

Now, exit your container by either typing exit or pressing Ctrl-d:

EXECUTE:  exit [or press ^d]

You have now exited your container and are back at your OS prompt.

EXECUTE:  docker stats -a --no-stream

RESPOND:  What is all this info?  What does it mean?

EXECUTE:  docker stop [youruserid]_ubuntu

EXECUTE:  docker ps

RESPOND:  What does docker stop do (be specific about the signal)?

Now, start your container again:

docker start [youruserid]_ubuntu

EXECUTE:  docker kill [youruserid]_ubuntu

RESPOND:  What just happened?  Be specific about signals.  (run "docker kill --help" to get more info).

Now share a directory from your hard drive with your container (cd to /tmp)

On your host system (or inside you virtualbox VM, if you're running docker inside virtual box), cd to your ~/mpcs56600/lab6 directory.  Now create a new file called lab6.txt in your lab6 directory and edit its contents to be "I love docker".

EXECUTE:  docker run -it -v ~/lab6:/clab6 ubuntu:14.04 /bin/bash

Now, from within your container:

EXECUTE:  cat /clab6/lab6.txt

EXECUTE:  ls –la /clab6

RESPOND:  Discuss what just happened.  Be specific about the docker run command above and the effect of the -v flag...

Now, exit and remove the container you just created...whatever its name or id.
       
Now, suppose we’d like to copy our ~/mpcs56600/lab6/lab6.txt file into your [youruserid]_ubuntu container.  How would you do that?

EXECUTE:  docker cp ~/mpcs56600/lab6/lab6.txt [youruserid]_ubuntu:/clab6.txt

RESPOND:  Is your [youruserid]_ubuntu running?  Does it matter to the docker cp command whether the container is actually running?

EXECUTE (if your ubuntu is not running):  docker start [youruserid]_ubuntu

EXECUTE:  docker exec -it [youruserid]_ubuntu /bin/bash

Now, inside your container:

EXECUTE:  ls -la /clab6.txt

RESPOND:  What do you see?

EXECUTE:  cat /clab6.txt

RESPOND:  What do you see?

EXECUTE:  vi /clab6.txt

Now modify its contents in some way.

Now exit your container.

EXECUTE:  docker cp [youruserid]_ubuntu:/clab6.txt ./modified.txt

cat ./modified.txt

RESPOND:  What do you see?

Now make this all permanent by saving a new image based on your modified container.  That way, you can use [youruserid]_ubuntu for all sorts of new purposes:

EXECUTE:  docker images
EXECUTE:  docker commit [youruserid]_ubuntu ubuntu:[youruserid]_update
ESXECUTE: docker images

RESPOND:  What just happened?  What is the difference between the outputs of the two docker images commands above?

EXECUTE:  docker inspect ubuntu:[youruserid]_update

RESPOND:  What is all this information, in summary?

One last thing to do.  Let's practice saving a container image to a tarball so we can share our images and containers with others.

If you need to (run docker images to find out), get down busybox again:

EXECUTE:  docker run busybox echo Hello 56600 Class Peeps!

Now, let’s copy that image to a tarball (REPOSITORY and TAG are separated by colons in docker commands):

EXECUTE:  docker stop [youruserid]_ubuntu

EXECUTE:  docker save -o ubuntu_[youruserid]_update.tar ubuntu:[youruserid]_update

EXECUTE:  docker rmi ubuntu:[youruserid]_update

EXECUTE:  docker images

Now, let's load that saved image into a :

EXECUTE:  docker load  < ubuntu_[youruserid]_update.tar

Now, let’s see our newly imported image:

EXECUTE:  docker images

Now, run our new image to instantiate a container:

EXECUTE:  docker run -it ubuntu:[youruserid]_update echo Hello 56600 Class Peeps!

Finally, remove the new container you just created with docker rm.



 Make sure you SAVE YOUR SCRIPT FILE. 

Simply type "exit" and you will see something like:

Script done, output file is typescript
Saving output to file: mark.Tue.Jul.24.12-55-01.CDT.2018.out

References:

You may find the following references helpful (in addition to the links from previous labs):

General Docker Tutorial Links

Docker Cheat Sheet
Learn Docker in 12 Minutes Youtube
Demystifying Docker Youtube
TutorialsPoint:  Docker Tutorial for Absolute Beginners
Docker Overview
Ubuntu package commands

Submitting:

Use the folder named "lab6" in your Subversion repository. See the syllabus for more info about submission using Subversion. Upload your Lab 6 script file and any supporting materials to the repo.  There's no need to upload your saved image.  Since this submission is rather simple, a README text file to explain what parts of the work you are submitting, is optional.