Phishman – Tech

After doing extensive research on single board ARM-based computers I decided to build my own cluster. The purpose of the ultra low power cluster is to develop new applications with the most recent software technologies and host few production websites. Yes, I know, I’m crazy! 😀

My research showed that previous clusters lack significant calculating power and they’re based on performance limited (hardware) boards like Raspberry Pi 2, Raspberry Pi 3 and PINE A64+. The main problems with those boards are the RAM, the limited I/O of the microSD cards (limited to 25 MB/s on Pi 2 and Pi 3), and the network speed (10/100).
Those boards are more suitable to be used as development boards. Often microSD card speeds are limited to 25MB/s (on Pi 3), which means that even if you buy UHS-II and UHS-3 cards you won’t be able to utilize them at their full speeds.

I’ve looked closely at the ODROID single boards – those have powerful CPU (Quad and Octa-core A53 and A15), fast Gigabit network (10/100/1000) , 2GB RAM USB 3.0, eMMC support.

For my build I’ve chosen the following hardware

Parts and single-boards – www.ameridroid.com

Web and Caching Nodes:
ODROID-C2:

Web Server Nodes:
ODROID-XU4:

We benefit from faster more capable CPU, eMMC (up to 400MB/s capable eMMC5.0) speeds, USB 3.0, Gigabit network and 2 GB of LPDDR3.

Database Nodes:
ASUS H110S2/CSM Mini-STX (Installing Percona With TokuDB MySQL Server)

For our cluster we will start with:

• 1 proxy node
• 2 load balancer nodes
• 2 web server nodes
• 1 database server node

Our final goal is a cluster with:

• 1 proxy node
• 2 cache nodes
• 8 nodes in web cluster
• 10 nodes in MySQL cluster and 2 nodes in NAS cluster. Total 23 nodes.

We may not be able to fit that many nodes in this printer case, because the power supplies needed to power all the nodes will require a lot of space.

Hardware Prices:
$90.75 – 2 x ODROID C2
$53.90 – 2 x 16GB eMMC Black Module C2 Linux
$133.05- 2 x ODROID-XU4Q with Passive Heatsink
$54.90 – 2 x 16GB eMMC 5.0 Module XU3/XU4 Linux (Blue Dot)
$20.15 – 2 x 5V/4A US Type Power Supply
$30.00 – 1 x Raspberry Pi B+
$39.00 – 3 x MicroSD Sandisk 16 GB UHS-I
$3 – 1 x Display flat cable 22 pin to 15 pin Pi (China)
$29.99 – 20 x 3 ft CAT6 patch cable – MAXIMM 3 Feet |20 Pack | Red| Snagless Cat6 Ethernet Cable With Cable Ties
$15 – 1 x SABRENT SSD Enclosure USB 3.0 EC-UASP
$20 – 2 x SABRENT SSD Enclosure USB 3.0 EC-UK30
$14.99 – 1 x KEEPING 5V/12A 5-Port USB Hub 60W Wall Power Charger Sation for Tablet, iPhone,Samsung Galaxy, Amazon Kindles, HTC, LG
$100 – 2 x SSD PNY 120 GB
$74.36 – 1 x NETGEAR 24-Port Gigabit Ethernet Desktop/Rackmount Switch (GS324)
$8.63 – 1 x M3 25+6mm Male Female Thread Nylon Hex Standoff Spacer Pillar 50pcs
$8.90 – 1 x HOBBYMATE 180Pcs Assorted M3 Nylon Screws Nut Standoffs Spacers for DIY
$5.89 – 1 x Orion 3d Black Carbon Fiber Vinyl Wrap – Outdoor Rated for Automotive Use – 12″ x 60″
$5.90 – 1 x Pwr+ Short 3 Ft 3 Prong AC Power Cord Cable Plug L-Type
$1.00 – 1 x Mesh organizer
$81.49 – 1 x ASUS H110S2/CSM LGA1151 DDR4 Dual GB LAN DP HDMI LVDS Mini-STX
$50.00 – 1 x Intel Pentium G4400 SkyLake 3.3 GHz LGA 1151 Boxed Processor (for BIOS update)
$50.00 – 1 x Intel Pentium G4560 Kaby Lake 3.50GHz LGA 1151
$216.98- 1 x G.SKILL 32GB (2 x 16G) Ripjaws Series DDR4 PC4-17000 2133MHz Laptop Memory Model F4-2133C15D-32GRS
$135.00- 1 x Samsung 960 EVO Series 250GB NVMe M.2 Internal SSD
$26.56 – 1 x Oneda 120W 19V 6.32A AC Power Adapter Laptop Charger for Asus
$14.99 – 1 x Cable Matters (2-Pack) 4-Outlet AC Power Cord Y-Splitter – 1.5 Feet
$7.99 – 1 x HUACAM HC05 3 Pack USB to 5.5 mm/2.1 mm 5 Volt DC Barrel Jack Power Cable
TOTAL: $1292.29

Case enclosure used is old broken Canon printer, which I liked for the project.

Building

I was installing TokuDB Storage Engine on Percona 5.7 Server based on this Percona guide, when I realized that there’s no my.cnf file.

You are still able to configure the server and install plugins from within the MySQL CLI.

Tech Specs:
OS: Debian Jessie
RAM: 32 GB
CPU 2 Cores, 4 Threads
Storage: SSD NVMe Samsung 960 Evo

More about the server:
This little MONSTER is amazing!

I have it paired with Intel G4560, 32 GB DDR4 G.Skill Ripjaws (2 x 16 GB) and M.2 Samsung Evo 960 250GB. Installed Debian Jessie server edition (with latest kernel) and did SSD read/write benchmark. It does 1.7 GB/s write and 3.8 GB/s read of 1 GB file. Insane!!!
Stock CPU fan – CPU 32C, motherboard 34C degrees. Under load 38-40C max!

I had to buy Skylake CPU (Intel G4400) to update the BIOS for the Kaby Lake (Intel G4560) support. It’s LAME! Those manufacturers must integrate USB flashing by default as a basic functionality! If I had time I would send the motherboard back or just return it. Unfortunately I had no time to ship my motherboard for BIOS update. It was in the product description that it supports 6th Gen Skylake CPU, even on ASUS’ website it was clearly stated that it needs BIOS update, so it’s not something that I didn’t know. It still is the best, smallest STX motherboard!

I would highly recommend and I would buy AGAIN this little monster board! I hope it stays sharp and fast!
I’ve been waiting for an STX form factor for years. It’s dream come true! 🙂

Thanks crazy awesome ASUS! 🙂

Here’s how to do it:

1. Installing Percona Server on Debian and Ubuntu (Official)
2. To avoid “ERROR 1123 (HY000): Can’t initialize function ‘TokuDB’; Plugin initialization function failed”, you need to disable Kernel Transparent Hugepages for TokuDB, if not disabled.
   Here’s how to check:
   
   Check Transparent Hugepages Status

   cat /sys/kernel/mm/transparent_hugepage/enabled

   1	cat /sys/kernel/mm/transparent_hugepage/enabled

   
   It should return something like:
   
   [always] madvise never

   1	[always] madvise never

   
   If it’s not set to never, you can do:
   
   Change Transparent Hugepages in The Kernel Config

   if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi

   1 2 3

   if test -f /sys/kernel/mm/transparent_hugepage/enabled; then    echo never > /sys/kernel/mm/transparent_hugepage/enabled fi

   
   Reboot.
   Test again after boot:
   
   Check Transparent Hugepages in Linux Kernel

   cat /sys/kernel/mm/transparent_hugepage/enabled

   1	cat /sys/kernel/mm/transparent_hugepage/enabled

   
   Now it should be:
   
   always madvise [never]

   1	always madvise [never]

   
    
3. Dependencies
   You need libjemalloc library installed. Quick and simple installation:
   
   Install libjemalloc And Unmet Dependencies

   apt-get install libjemalloc-dev apt-get build-dep libjemalloc-dev

   1 2	apt-get install libjemalloc-dev apt-get build-dep libjemalloc-dev

   
    
4. Install The TokuDB Engine
   
   Install The TokuDB Engine and Plugins

   mysql -uroot -p # Install the TokuDB Engine mysql> install plugin tokudb soname 'ha_tokudb.so'; # Install the Other TokuDB Plugins mysql> install plugin tokudb_trx soname 'ha_tokudb.so'; mysql> install plugin tokudb_locks soname 'ha_tokudb.so'; mysql> install plugin tokudb_lock_waits soname 'ha_tokudb.so'; mysql> install plugin tokudb_file_map soname 'ha_tokudb.so'; mysql> install plugin tokudb_fractal_tree_info soname 'ha_tokudb.so'; mysql> install plugin tokudb_fractal_tree_block_map soname 'ha_tokudb.so';

   1 2 3 4 5 6 7 8 9 10 11 12

   mysql -uroot -p   # Install the TokuDB Engine mysql> install plugin tokudb soname 'ha_tokudb.so';   # Install the Other TokuDB Plugins mysql> install plugin tokudb_trx soname 'ha_tokudb.so'; mysql> install plugin tokudb_locks soname 'ha_tokudb.so'; mysql> install plugin tokudb_lock_waits soname 'ha_tokudb.so'; mysql> install plugin tokudb_file_map soname 'ha_tokudb.so'; mysql> install plugin tokudb_fractal_tree_info soname 'ha_tokudb.so'; mysql> install plugin tokudb_fractal_tree_block_map soname 'ha_tokudb.so';

   
   Test to check, if the TokuDB storage engine was successfully installed do:
   
   View Installed MySQL Engines

   mysql>SHOW ENGINES;

   1	mysql>SHOW ENGINES;

   
    

That’s it. Enjoy! 🙂

Feel free to shoot me a message in the comments.

I decided to go with the Dreamhost‘s VPS server after being on shared for 6 years. I moved all of my domains to 300MB VPS which was crashing every 20-30 minutes, then I decided to give it a shot and upgrade to a 400MB VPS, that easily reduced the reboots and after that I had reboots just every 3-4 hours.

I had some troubles at the beginning because my websites were generating quite of traffic and I decided to do some calculations of what traffic load can the Dreamhost VPS handle. Here are the results:

VPS Setup:
RAM: 400MB
Total running domains: 12

1 month load/traffic results on the VPS:

Total Hits: 965922/mo
Total Unique Visitors (IPs): 34079/mo
Total Bandwidth Used: 56.3GB/mo
Server IOPS Used – Avg: 9403; Max: 18421; Min: 1157;

Fair enough! The following numbers are what can a Dreamhost VPS handle per 100MB of VPS RAM resource:

Total Hits (Approx): 250000 hits/mo
Total Unique Visitors (IPs): 8519/mo
Total Bandwidth: 14.07GB/mo

NOTE: This is the traffic that was utilizing the maximum of my VPS, no  resources were left unused! Sometimes the server was rebooting on spikes by itself in order to keep the host safe and stable.
At some point I had to disable all cron jobs on the VPS because it was barely handling the load and it was rebooting constantly. After awhile I decided to shut down some of my domains and sub-domains that
didn’t bring any money, but doing great traffic and I was able to lower the server load and resume all the cron jobs. After shutting down some of the websites, now my account does 8.5GB Bandwidth per month which is 7 times less than what I was utilizing before with all the domains.At some point after keeping an eye on my access.log I found no reason to keep traffic from Russia and China, because I was getting only brute force attacks and website scans! Dreamhost lacks ability to install iptables and I had to ban them at configuration level. Which reduced my bandwidth to 56GB/mo (saved few gigs and server load)!

Now I know when, how and why I should upgrade my VPS and I know how important is to protect it against Brute Force Attacks, DDoS attacks, Network Scans, Web Scrappers, Bots and Search Engine Robots.
I was surprised what this tiny server can handle! After all it’s a tiny beast!
Epic Story!

Thanks! To my friend Mr. Gulik guiding me thru the beginner path of  the young sysadmin!

Screenshot with the benchmarks:

I recently moved to my brand new DreamHost VPS just to find out how much better is than my shared hosting. I get a bit over 180000 hits for 24 hours, but it happened that my server was rebooting due to heavy load and excessive memory usage. Right now I’m using the VPS with 400MB of ram. It was rebooting like every 20-30 minutes, but it seemed that not all of the traffic was legit. Large amount was web page scrappers, regular search engine bots, hacker attacks (brute force, sql injections), etc and my server was benching against the heavy load. I tried installing iptables, but due to the ancient VM technology that dreamhost is using is not possible 🙁

So i had two options, to get my server back from the dead without paying tons of money for feeding all the automated traffic and dreamhost:

1. iptables alternative (e.g. shorewall)
2. htaccess ip blocking

For the faster and immediately efficient option I chose the second one, so I compiled a list with all the httpd.ip-filter.conf

In order to block China and Russia I’d recommend to create separate file that is included in your httpd.conf. Apache (if apache is used) is C language based program and once it loads the long IPs list in the memory will react super fast to any calls from IPs in the list. Unless you don’t have access to edit the httpd.conf I wouldn’t recommend adding this line in .htaccess file.

The list contains 8864 Russian networks and 5073 Chineese.

1. Download httpd.ip-filter.conf

2. Save the file in the folder where httpd.conf is and name it the way you want. I used httpd.ip-filter.conf

3. Open the httpd.conf and include the line ” include /path/to/the/httpd.ip-filter.conf ” (without the quotes).

4. Save the conf and restart the apache

If you don’t get any errors you should be all set!

After doing so, my server is now relieved! I get far less traffic and my VPS don’t die from excessive memory that is not used by legit traffic.

Finally I found some time to post my new custom build. I’m building my workstation for 3D rendering, graphic and video design, gaming and high resource intensive math and cryptography calculations and benchmarks. I’m excited to perform some of my planned OpenCL and ImageMagick tests…

Build Details: I was looking for motherboard with a many PCI-E slots, because I need to run 6 x Sapphire 7970 Vapor-X 3GB version. I found excellent deal for used P6T7 WS on eBay with set of memory, memory fan, CPU and CPU fan.

ORIGINAL CONFIG (with the parts from eBay)
Motherboard: P6T7 WS SuperComputer 7 x PCI-E x16 (4-way CrossFireX x16 or 7 x PCI-E x8).
CPU: Intel® Core™ i7-920 Processor (8M Cache, 2.66 GHz, 4.80 GT/s Intel® QPI)
Memory: CORSAIR DOMINATOR 8GB (4 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 + Memory fan
SSD: OCZ Vertex 120GB
CPU FAN: ZALMAN CNPS9900MAX-B 135mm Long life bearing CPU Cooler
GPU: 6 x SAPPHIRE Vapor-X VXSR Radeon HD 7970 GHz Edition 3GB 384-bit GDDR5
PSU: 2 x ZALMAN 1250 Platinum 1250W ATX12V v2.3 SLI Certified CrossFire Certified 80 PLUS PLATINUM Certified Modular Active

LATEST CONFIG
Motherboard: P6T7 WS SuperComputer 7 x PCI-E 2.0 x16 (4-way CrossFireX x16 or 7 x PCI-E x8).
CPU: Intel® Xeon® Processor X5690 12M Cache, 3.46 GHz, 6.40 GT/s Intel® QPI
Memory Kit 1: HyperX Savage – DDR3 – 32 GB: 4 x 8 GB – DIMM 240-pin – unbuffered
Memory Kit 2: HyperX Savage – DDR3 – 16 GB: 2 x 8 GB – DIMM 240-pin – unbuffered
Total Memory: 48GB
SSD: SAMSUNG 950 PRO M.2 512GB SSD PCI-Express 3.0 x4 Internal Solid State Drive
SSD PCI-E Adapter: Angelbird Wings PX1 PCIe x4 M.2 Adapter and 512GB NVMe M.2 SSD Kit
CPU FAN: ZALMAN CNPS9900MAX-B 135mm Long life bearing CPU Cooler
GPU: 2 x SAPPHIRE Vapor-X VXSR Radeon HD 7970 GHz Edition 3GB 384-bit GDDR5
PSU: EVGA SuperNOVA 1600 P2 1600 Watt 80 Plus Platinum ATX Fully Modular Power Supply

This is going to be no case build (no case for now) until I decide how much space I’ll need with all the 6 GPUs and dual PSU, and then I’m planning to build custom rack.
I’m very creative and I love to generate new ideas every single minute. So here we go, yeah I’m getting excited too! Let’s build some funny legs…
For now I’ve got some basic parts to build custom motherboard legs/stands to put some clearance between the motherboard and the table: 4 x Push-on F-connectors for coaxial cable (Ideal RG-6) for motherboard feet. Machine screws 6 x M3 25.4mm length metric (US standard #6-32 x 1″) Flat Head Phillips, 8 x Machine Screw Nuts M3 (US standard #6-32). Shepherd Vynil Bumpers 10mm (3/8″).

UPGRADES:

Stage 1: removed/upgraded hardware:
CPU: Intel® Core™ i7-920 Processor (8M Cache, 2.66 GHz, 4.80 GT/s Intel® QPI) – Upgrade to Intel Xeon W3690 (…which died later. I killed this basta’d – OC overheat!)
PSU: 2 x ZALMAN 1250 Platinum 1250W ATX12V v2.3 SLI Certified CrossFire Certified 80 PLUS PLATINUM Certified Modular Active – Died, switched to EVGA SuperNOVA 1600 P2

Stage 2: removed/upgraded hardware
SSD: OCZ Vertex 120GB – Upgrade to Smasung EVO 860 256GB SATA 3
Memory: Added additional Corsair Dominator kit 2 x 4GB 1600 (total 6 x 2GB)
One of the old Dominator modules died. It was replaced!

Stage 3: removed/upgraded hardware:
Motherboard: Got a brand spanking new same (barely used) motherboard P6T7 WS (free)
Memory: CORSAIR DOMINATOR 8GB (4 x 2GB) 240-Pin DDR3 SDRAM DDR3 1600 + Memory fan – Upgrade to 48 GB HyperX Savage
SSD: Smasung EVO 860 256GB SATA 3 – upgraded to SAMSUNG 950 PRO M.2 512GB SSD PCI-Express 3.0 x4 Internal Solid State Drive
SSD PCI-E Adapter: Angelbird Wings PX1 PCIe x4 M.2 Adapter and 512GB NVMe M.2 SSD Kit
CPU: Intel Xeon W3690 – CPU died, switched to Intel Xeon X5690
PCI-E USB 3.1: StarTech.com USB 3.1 PCIE Card – 5 Port – 1x USB-C – 2x USB-A – 1x 2 Port IDC – Internal USB Header Expansion – USB C PCIe Card (PEXUSB312EIC)
Upgraded Case Fans: (old case fans started rattling)
4 x Cougar CFD14HBR 140mm CPU Fan Cooling (Red LED)
2 x Cougar CFD12HBR 120mm CPU Fan Cooling (Red LED)
HDD:
Drive #1 – HGST HDN724030ALE640 (2794 GB)
Drive #2 – WDC WD30EZRX-00D8PB0 (2794 GB)
Drive #3 – WDC WD40EFRX-68WT0N0 (3726 GB)
Drive #4 – HP SAS EF0300FARMU (279 GB)
Case: CoolerMaster Storm Trooper Black

March 2020 PSU UPDATE: Two PSUs EVGA SuperNOVA 1600 P2 Platinums have died. Right now the system is running on Corsair RM850x Gold.

Stage 4:
P9X79 WS is arriving in few days (5/25/2020).
The plan is to replace the motherboard and the CPU!
Considering those two CPUs Intel Xeon E5-2690 V2 and Intel Xeon E5-2697 V2
To be continued…

Stage 1 Photos:
I had no full tower case that could fit this extended motherboard, so it was running on a table for a while.
I came up with stand-offs (legs) made of push-on f-connectors.

Stage 3:
Love the red colors!