Introduction - GRIDMAN

It's time to say Hello.

Synopsis

My Raspberry PI passion turned into a development project that finally reached a stable release.
So it's time to write down some things, before they vanish!

Few months of coding afterhours, with few longer breaks and it turned out that a small project became a big passion and a very interesting challenge. At the end of that - i have a GRID system running on my Llibrary. Which is ULTRA easy to manage and work with - that counts. Not only that, it looks beautifull !

This blog will cover things related to my SOHO Grid Computing Library called GRIDMAN. Additionaly it will cover the Raspberry PI GRID system called PIGrid i built and any other GRIDMAN related topics.

PIGrid build

The GRIDMAN - is the grid computing library, targetted to SOHO users - so for people like you and me, at home or small office. When i first came up with that idea i was thinking about home usage only, but really it turned out it can help in small offices too!

Theory 

Now, what is a GRID COMPUTING ? Wikipedia - have a brief look, i know you are aware what's that if you're here.

First things first - i never liked Java and overwhelming opensource applications that requrie you to spend days to configure them, so i wrote my own Library that simplifies steps to setup a GRID computing to MAX 15 minutes. GRIDMAN is written in C++ and is dedicated to work with Linux platforms, so Raspberry was the best choice for a DEV/TEST environment ( i can't afford a Blade C7000 to run 24/7 at home ;).
On the other hand, i just couldn't find anything suitable - easy - small and free. So that's why i decided to make my own GRID library.

The System consists of:
- The GRID Server  binary
- The GRID Worker binary
- The GRID Commandline client
- The GRID iPhone manager ( AppStore )
- The GRID library (C++)

4 nodes version managed with GridMAN app

For GRIDMAN - you write a Task and then you submit it to the GRID. That's it. No more SLA/Planning stuff, you just don't care. When Workers are free, they'll pick up the task and return defined results after it completes. You say to yourself; oh i want to render that frame on 6 nodes. Then you split that frame into 6 parts and send 6 tasks to the grid that will render one part and return the results to your project application.
Simple.

History

Of course, the Task can be anything - one of my first tasks were:

- MD5 Distributed Cracker
- WPA Distributed Cracker
- WEP Distributed Cracker

Couple of old links: YT | YT

But there are tons of ideas really, you do the math.
Really GRIDMAN was created to run bruteforce on few scenarios, but it turned out quickly i cane make it generic. So somewhere around autumn 2013 i decided - lets make GRIDMAN a generic GRID Computing Library.

We know what GRIDMAN and GRID COMPUTING is ?
Good, let's move to the Hardware now.

Hardware Specs

Having a library is not enough, you need the hardware that will be the GRID for you. For that i decided to use a set of Raspberry PI's - cheap mini CPU's, next a PDU ( Power Distribution Unit ) - i am using a D-Link 7 port active USB HUB - it really works, and finally a way to handle the communication.
Because GRIDMAN works on TCP/IP obviously you need a LAN cable or go with WIFI.
I am using the WIFI, not the fastest for sure, but wireless - which matters when you don't have enough place for the project or kids running around.

My Raspberry GRID (PIGrid) is made of Rasppberies PI 512MB version, TP-LINK WiFi micro dongles (WN725N), 8GB SD Cards, and obviously MicroUSB cables that are hooked up to a D-Link DUB-H7 Active USB Hub.

"precious" PIGrid

I've got them talking to the GRID master via WiFi, yeah i know you might want to shout about the latency but it's not the case when it comes to DEV/TEST environment, i just like the fact i don't have to play around with the cables - thanks to the wireless approach.

PIGrid- current overnight processing

PIGrid- first test on the old PDU

Usage

I am always asked about the usage by people, so here are some things i use GRIDMAN for.
Additionaly, i'll pinpoint some benchmarks here, so that you can 'feel the difference'.

First of all i've got two GRID systems running GRIDMAN. 

The first one is obviously PIGrid - the one i use to dev and test. But not only for that. The Rasppbery PI is excellent when it comes to low-power consumption - it's pretty obvious that running a 24/7 - 6 nodes GRID system made of PC's can cost you a lot.

That's why i use my PI Grid for such daily computing, stuff like Market Trends Analysis - where i distribute the tasks to a GRID on Per-Instrument Basis. This GRID is mainly managed through the iPhone. It is fully automatic really.

The second GRID i have is a shared environment that consists of a 8 Nodes setup, in which every node is a i7 Intel CPU. This is a PUMP! GRID is stored in a remote location and used with a remote access, we do a lot of Brutefroce stuff there, lot of WPA / MD5 really - managed securely through command line.

Okay, other things i've tested but postponed ( due to the lack of free time ) : Distributed Raytracing, 3DS Max Distributed Render (via commandline interface), Distributed Rendering (with OpenGL ES on Raspberry PI) and even touched distributed Compiling once ;)

Benchmarks

Tasks in the benchmark :

1 Node MD5 lowercase bruteforce test (running on a single node)

6 Nodes MD5 lowercase bruteforce test (above split into 6 tasks and distributed to the GRID paraller)

1 Node WEP lowercase bruteforce test (running on a single node)

6 Nodes WEP lowercase bruteforce test (above split into 6 tasks and distributed to the GRID paraller)

For the single node, MD5 finished after 398seconds, WEP finished after 828 seconds.

For the GRID, MD5 finished after 193seconds, WEP finished after 186 seconds.

The benchmark was performed on PIGrid, the benefits are clearly visible.

WPA/MD5 benchmark tests on 6 PIGrid nodes

Of course, each task timing has network latency included therefore i am talking about the FULL lifecycle of a task. I hope you get the idea of the whole thing now.

Final words

Building your own GRID is as great as it sounds - you need to think in a different ways when you think of problems you want to solve. You approach them by trying to divide them into smaller things, and look for eventual performance gain.

PIGrid stacking

Building your own GRID from Raspberries, thinking of Colling it, measuring it's performance is another adventure. This is that 'look' adventure, where the visual effect matters.

I am happy that writing a  project became a hobby and is now a main area of interest.
Coding a library combined with a build of a fantastic high-tech GRID system made of Raspberries is an eye-candy experience, its affordable, it's great looking and will work for you upon-request anytime you want it.

Also, it works from your iPhone, so you can show people it is alive even when you are drinking a beer in the pub on Thursday.

Before anything happened