MckoiDDB Network Guide
This document describes the network level architecture of MckoiDDB, explaining what the server software does, and gives practical examples of installing MckoiDDB on a network.
Throughout this document the following terms will mean;
- Mckoi Machine Node - The MckoiDDB software installed on a machine.
Network Machine Requirements
Before we start, we will quickly cover the software and hardware capabilities recommended to support a Mckoi Machine Node. Each machine will need some local disk space, a network port, and an operating system that supports Java 1.6. If the machine is connected to the Internet and has a public IP address, it should also have a second network interface to a private network that provides high bandwidth and low latency communication between the machines. Every machine must be able to communicate freely with every other machine.
Installing Mckoi Machine Nodes across multiple data centers where node communication is routed over the Internet is a very bad idea. MckoiDDB expects a high quality of service, low latency network for inter-machine communication, and the messages passed between machines are not encrypted.
The network.conf Configuration File
The network.conf file is used by the Machine Node software to determine which IP addresses are allowed to talk with the node and where all the other nodes in the network are located. You can store this file on a shared file system or publish it from a HTTP server so you only need to update it once whenever a machine is added or removed. The network.conf file contains three properties;
- connect_whitelist - The list of IP addresses of clients and Machine Nodes that are allowed to connect with the network.
- network_nodelist - The list of locations of all Mckoi Machine Nodes on the network.
- configcheck_timeout - The time between polls to see if the network.conf file has been updated.
The following is an example network.conf file for a network of 5 Machine Nodes. The Machine Nodes are located on IP addresses 192.168.30.0 through 192.168.30.4 on port 3900. The machines allowed to communicate with the network have IP addresses 192.168.30.0 through 192.168.30.11.
# The list of IP addresses allowed to connect to the network
connect_whitelist = 192.168.30.0, 192.168.30.1, \
192.168.30.2, 192.168.30.3, \
192.168.30.4, 192.168.30.5, \
192.168.30.6, 192.168.30.7, \
192.168.30.8, 192.168.30.9, \
192.168.30.10, 192.168.30.11
# The list of Machine Nodes on the network
network_nodelist = 192.168.30.0:3900, 192.168.30.1:3900, \
192.168.30.2:3900, 192.168.30.3:3900, \
192.168.30.4:3900
# The time, in seconds, between checks of this
# configuration file for changes. Defaults to 2 minutes.
configcheck_timeout=120
The node.conf Configuration File
The node.conf file contains a set of configuration properties which tell a Machine Machine Node where to store the data and log files in the local file system, and what the network password is. The network password is a string shared between all Machine Nodes that is given and checked at the start of all node-to-node communcation. The purpose of this password is to ensure that multiple networks can not accidentally communicate with each other. It should not be considered a security device.
The node.conf file is deployed with each Mckoi Machine Node installation. If this configuration file is changed, the Machine Node must be restarted before the changes take effect. The following is an example node.conf that stores data at /mckoiddb/ in the local file system. The network_password property is left for your choice;
# Choose a network password, network_password = # The location in the local filesystem of the directory # that stores stateful data of the roles running on the # machine. node_directory = /mckoiddb/data # The directory location in the local filesystem to store # logging information (uses the internal Java logging # format). log_directory = /mckoiddb/log # The minimum log level to report (inclusive), log_level = info
Deploying a Mckoi Machine Node
After the network.conf and node.conf files have been created you are ready to install the Machine Node on a server. To start a Machine Node, upload the MckoiDDB distribution to the server and run the following Java command from the server's operating system shell (this example would be run on the machine at 192.168.30.0);
Note: the command is entered as a single line
java -cp lib/mckoiddb.jar com.mckoi.runtime.MckoiMachineNode -nodeconfig node.conf -netconfig network.conf -host 192.168.30.0 -port 3900
The distribution also includes an init.d style Daemon script for Linux. Follow the instructions in the distribution for instructions on using the script to start up the Machine Node at system boot.
The MckoiDDB Network Architecture and Machine Roles
Once all the Machine Nodes have been installed and are running, the nodes need to be given one or more roles to perform as part of the operation of the network. Before we do this, a quick overview of the architecture will be explained.
All data in a MckoiDDB network is stored in block files. Block files are fairly large files (usually multiple megabytes) located in the block/ sub-directory of the Machine Node's data directory. When data is written into a block file, MckoiDDB will try and replicate the block file on at least three machines in the network. The reason for replicating the data is to ensure data continues to be available even when two machines fail or need to be taken down for maintenance. Additionally, replication gives options to clients on where to access data thereby providing load balancing.
For illustrative purposes, the diagram below demonstrates a set of four blocks stored over a distributed network of five devices, each block replicated three times. Any two devices in the network can fail yet the entire block file set will continue to be available.
 |
Both structural meta-data and user data are stored across these block files providing a decentralized system for reading and writing information. In MckoiDDB terminology, the service that provides access to the block files stored on a machine is called the Block Server Role. The Block Server Role is one of three roles assignable to a Machine Node. The other two roles are the Manager Server Role and the Root Server Role.
The Manager Server Role is an essential component of the network that maintains a small database about the location of blocks. The manager server resolves queries such as 'which servers contain the block files for block n '. Additionally, a manager server maintains current status information about machines in the network and is notified of any failures. A new client on the network must query the manager server to 'bootstrap' itself and become aware of the network and where data is located.
The Root Server Role manages a group of Path Instances. A Path Instance is a set of historical snapshots of some structured data model stored on the network. A Root Server is needed to manage transactional operations and updates (when a commit happens) on user data. Every update to the state of a Path Instance must be moderated through a root server to ensure consistency. For example, a Root Server may decide to veto a commit because an integrity rule would be violated by allowing it.
A minimal MckoiDDB network has three Block Servers, one Manager Server and one or more Root Server.
Using the MckoiDDB Console to set Machine Roles
The MckoiDDB Console is used to assign roles to machines in your network. The following command will start the console application (assuming network.conf is in the current directory);
java -cp lib\mckoiddb.jar com.mckoi.runtime.AdminConsole -netconfig network.conf -netpassword [the network password]
You can find more specific information about the console application at the MckoiDDB Console Reference page. In this section we'll just be covering the commands used to assign machine roles. Note that you can only run the console application from a machine that is included in the connect_whitelist property.
Assuming you have a network of 5 machines on IP addresses 192.168.30.0 through 192.168.30.4 and you have setup the network as described previously in this document, the 'show network' console command should display the following;
MckoiDDB> show network MRB RC BC Server ------------------------------------ ... 192.168.30.0:3900 ... 192.168.30.1:3900 ... 192.168.30.2:3900 ... 192.168.30.3:3900 ... 192.168.30.4:3900 5 machines in the network. 0 manager 0 root 0 block machines.
If you receive any errors, it's either because your network.conf file does not include all the machines in the network (including the machine the console application is being run on), or the Machine Node application isn't running on all the machines.
In this demonstration we will assign the Manager Server Role to the Machine Node on 192.168.30.0, a Root Server Role on 192.168.30.1, and the remaining Machine Nodes will be assigned as Block Server Roles. For this configuration enter the following commands;
start role manager on 192.168.30.0:3900 start role root on 192.168.30.1:3900 start role block on 192.168.30.2:3900 start role block on 192.168.30.3:3900 start role block on 192.168.30.4:3900
Note that you can assign multiple roles to a Machine Node if you wish. You may find in the above setup that the servers running the manager and root server are underutilized, in which case you may decide to assign those machines as Block Server Roles also.
To check the role assignments you made, the show network command should now display the following;
MckoiDDB> show network MRB RC BC Server ------------------------------------ M.. 192.168.30.0:3900 .R. 192.168.30.1:3900 ..B 192.168.30.2:3900 ..B 192.168.30.3:3900 ..B 192.168.30.4:3900 5 machines in the network. 1 manager 1 root 3 block machines.
Note the MRB column on the left of the table shows the role(s) assigned to the machine (Manager, Root, Block).
Finished
MckoiDDB is now installed on your network and ready to go. You will need to create at least one path instance before a client will be able to use the network. The Developing a Simple Database Guide is a good place to start to learn how to create a database application using the client API.