Distributing Data with Symbient™ D2™


Imagine you have a bucket you use to transport water for your village. Your village is small, so you can do with just one bucket. Over time your village begins to grow, requiring you to carry more water. Unfortunately, your bucket has a finite storage capacity, leaving you to get another bucket in order to carry more water. To help balance the load, you evenly distribute the water 50/50 between the two buckets. As your village continues to grow, you need more buckets. And, again, you distribute the water evenly between them.

Your computer is like a bucket. It has a finite amount of storage capacity. If you want to store more water (aka, data), you need more buckets (or computers). Well, that's exactly what Symbient™ D2™ was designed for.

Introducing Symbient™ D2™
Symbient™ D2™, where "D2" stands for "Distributed Data", is an application that allows you to evenly distribute and share data across multiple computers with ease. Symbient™ D2™ creates a network where nodes (computers) can join and leave at any time, providing their own resources for data storage whenever connected to the network. As nodes join the network, data stored within the network is evenly distributed among all the nodes, just like evenly distributing water among your buckets. Data is stored redundantly so that in the event any node is removed from the network without performing a "clean" disconnect, data on that node isn't lost. As nodes "cleanly" leave the network, their data is shifted to the other nodes to preserve data integrity.

Thanks to the Symbient™ D2™ architecture, you can have an unlimited number of nodes joining and leaving the network at any time. Once data is moved to a node, it is stored on the local file system. So long as you have at least one node that can store all the data of the network, you will always have access to your data even if the network itself goes down. Once the node with the data is restarted, the network is restarted, allowing for all other nodes to reconnect to the network and once again distribute and share the data.

Applications like Symbient™ D2™ are used all the time, with the largest similar type of network being that used for Torrents. In a Torrent network, users share files amongst each other where, in order to retrieve a file from a remote node, your node simply places a request-call out on the network saying which file you're interested in. The network then identifies which node(s) have the file you're looking for and takes care of retrieving that file and sending it to you. With Symbient™ D2™, if the node that hosts the file you're looking for decides to leave the network, those files will be sent to another node for storage so that they're always available as long as they're registered with the network. Of course, you can also remove a file (or other data) from the Symbient™ D2™ network at any time.

Paying it Forward
As part of our Pay-It-Forward program at Lucus Labs, we've decided to share a Lite version of Symbient™ D2™ for you to use however you so choose. Unlike it's commercial counterpart, Symbient™ D2™ Lite comes with a few limitations and restrictions. However, it still provides enough functionality to make it useful for all types of projects. As a bonus, while Symbient™ D2™ Lite doesn't support node-level or even file-level password protection like its commercial counterpart, it does support encrypting data before it's placed on the network and/or written to the file system of neighboring nodes. So, you can rest assured your data will be safe even if it's sent to and stored on a node controlled by anyone unknown.

Getting Started
To get started, choose the Symbient™ D2™ file for your operating system below. For now, we're only releasing lite versions that run on Windows and Raspberry Pi. If you need it for another operating system, simply shoot us a message using the "Let's Talk" link at the top of this page and we'll compile and send you what you need.

Downloads:
Symbient™ D2™ for Windows
Symbient™ D2™ for Raspberry Pi

Once you've downloaded and extracted the version for your operating system, simply run the executable application found in the extracted folder. When you do, you will be presented with a menu for interacting with the network. For example, press zero (0) to see ID's of the nodes that make up the network.

Note: For security and privacy reasons, users cannot see the IP addresses of other nodes. Also, networks are started with nine (9) nodes as the default. If less than 9 nodes are on the network, the primary nodes will show as duplicates until additional nodes join the network. As they do, the list of node IDs will grow and shrink accordingly.

Pressing one (1) from the menu allows you to "put" data on the network. When you press 1, the application will ask for the name you want to store the data under. This can be a file name or anything else, but it must be unique across the entire network. If you reuse a name that already exists on the network, the data at that address will be replaced with the new data.

Pressing two (2) from the menu allows you to "get" data from the network. When prompted, simply enter the name of the data you want to retrieve and, if found, the network will return it to you.

Pressing three (3) from the menu allows you to "remove" data from the network using the name you provide when prompted.

To change the IP address, port number, location where data is stored, or whether or not data should be encrypted, open up the included "settings.conf" file in a text editor of your choice. If you don't want to modify the defaults, you can always overwrite settings when launching the app from the command line. To get settings names, run the app from a command line followed by "-?", such as "SymbientD2.exe -?". This will display a usage chart that shows the options you can overwrite along with a short description of each.

Note: When joining an existing network, make sure you pass "-j true" to force your node to connect to the backbone address. Otherwise, Symbient™ D2™ will attempt to start a competing network.

Using it on Your Own
By now you're probably thinking, "what good is this if I have to use a command line menu to interact with the network?" Well, you don't. Symbient™ D2™ was built utilizing standard HTTP under the hood. That means you can interact with it using any technology that speaks HTTP. To help get you started, we will use Postman for interacting with our Symbient™ D2™ network. You can use Postman as a browser extension or download it as a standalone app. Either way, you can find it at https://postman.com.

Once you've installed and ran Postman, open a new tab in it and enter the address "http://[your IP address]:[your D2 port]/put" (such as "http://127.0.0.1:12345/put" if you're running D2™ on the same machine as Postman and you haven't changed the default port). Next, add 2 fields called "key" and "value", and set them both to whatever you'd like. Next, change the method to either "POST" or "PUT" (Symbient™ D2™ supports both methods), and click "Send". Next, in your Symbient™ D2™ window, press two (2), enter the value you used in the "key" field from before, and press enter. At this time you should see the data from the "value" field you defined in the Query Params section of Postman. See screenshot below for a reference.

Symbient D2 Example

Keep in mind that "value" in the above can be anything you want it to be so long as it can be serialized. For example, you can store blocks of JSON data using JSON.stringify({ }) or even use binary data, such as files, if you serialize them first. You can also replace "127.0.0.1" above with the IP address of any node on the network as long as you can ping it.

Here are other addresses and methods you can use to interact with your Symbient™ D2™ network.

Address Method Params
/put POST/PUT key, value
/get GET key
/removekey POST/PUT/DELETE key


That's it! You now have the ability to easily distribute and share data across multiple computers! Even though this Lite version of Symbient™ D2™ has some limitations not found in its commercial counterpart, it still provides enough functionality for you to do all kinds of cool things such as have Symbient™ IDS post messages whenever unexpected activity occurs on your network, or have Symbient™ Spine™ store messages as they traverse your Internet of Things habitat. The possibilities are endless! And the best part, it's FREE! So, enjoy!

BTW, be sure to let us know what kinds of cool things you decide to do with any of our Lite products. We're always excited to hear from you!