HTML5 -- Possiblity

I have been hearing about html5 for a while now.
Finally took a peak at it 3 days ago and it was awesome.

HTML5 is a flash killer!
Now you can integrate sound, video, image manipulation, SVG right into your web page.



Here are some pit holes when I was testing canvas:

• canvas.getContext('2d').putImagedata is slow when using large offset
• creating 1000 canvas consume 500-800mbs, better to create one huge canvas for all your buffers.
  

I always dislike flash, so I'll take the plunge into HTML5.
I'm going to be creating a game with HTML5.
It a real time strategy game know as Z, it's around 400 mbs. But I have figure a way to shrink it down to under 3 mb.

Plan:

• Decode the games rules, stats for unit and terrain 0%
• Extract tiles, sprites, and audio. 60%
• Extract original levels 5%
• Loss-less Compress of resources for web usage.30%
• JavaScript Engine to run the game
• AI for the single player
• hack in to google/facebook for multiplayer

Beside HTML5 awesomeness, IE9 were faster then firefox 3.
go here to test out your browser with html5 http://ie.microsoft.com/testdrive/

DOS, DDOS, BOTNET -- IT ARENT A PROBLEM

The follow are aim at hackers that aren't very smart (average hacker), the smarter hacker normally don't play around with you.

A friend of mine own a money making on line forum, and a group of hackers want to blackmail him for benefits. At first they start with a single ip to DOS the server. The server went down and my friend ask me for help.

I install a simple iptable to block attack from a single source and swap apache for light httpd. The server stay up for another few days until they start using DDOS and BOTNET.

When they start using botnet I made the program below, since the introduction of the program the server never crash again. At the peak of the attack there was 8000 ips and 26000 connection and the cpu was using 1-2%.


The basic of DDOS is any one of the following.
Resource Starvation: CPU 100%, RAM 100% or BandWidth 100%.

What to do when your under attack?

1. Go in and turn off your web service.
   You can't do anything if your server hang :D.
2. Run the following command, it change the priority of the process being attack, so your server wont hang.
   echo "www-data hard priority 10" >> /etc/security/limits.conf
   For more detail vist here http://kissconcept.blogspot.com/2010/09/set-default-priority-for-all-processes.html
3. Install the little script below, name Anti-Dos :D.

What do anti-dos do?
It take access data from your web service, analysis the data and determine the bad guys.
It use iptables to block people, pretty sweet. The best part is that it unbad the bad guys if they stop attacking after x duration.

What web service can it run with?

• lighttpd
• for nginx and apache it require a bit of change.

Iptables:
The following script create 100 bucket in iptables, and each bucket can hold 100 ips.
If some one can extend the amount of ips in the bucket give me call :D

Put the code in a php file

<?php 

`iptables -F`; 

for($index=0; $index <100; $index++){ 

    `iptables -A INPUT -p tcp --dport 80 -m recent --update --seconds 1800 --name ROBOT$index -j DROP`;
}

`chmod 777 /proc/net/ipt_recent/*` 

Configure Lighttpd:

•  Find a place to put the script and logs,
  
  • antidos : /home/www-data/antidos.php
  • log: /home/www-data/log/
  • error: home/www-data/error
  • Secret:^Z --> single letter
• Edit your antidos.php
  define ( 'LOGGING', '/home/www-data/log/');
• Edit /etc/lighttpd/lighttpd.conf
  
  • accesslog.filename = "| /usr/bin/php /home/www-data/antidos.php 2> /home/www-data/error"
  • accesslog.format = "%h^Z%t^Z%H^Z%m^Z%U^Z%X^Z%b^Z%{Referer}i^Z%{User-Agent}i"
    in vim ^Z is for 26 character code, you can type it by using ctrl+z in vim.

That's it :D, restart your lighttpd and wait for 2-5 min.
Your cpu usage should drop to 0%.

If it don't check for activity in the log folder and the error file.
you can use tail -f /home/www-data/log/access to follow the log

<?php

define ( 'SEPARATOR', 26 );
define ( 'ACCESSED_EXPIRE', 600);
define ( 'ACCESSED_POINT', 600); // Point to start banning people
define ( 'IPTABLES_NAME', 'ROBOT');
define ( 'IPTABLES_BUCKET', 100); // Including 0
define ( 'IPTABLES_PATH', '/proc/net/ipt_recent/');
define ( 'LOGGING', '/home/www-data/log/');

class AntiDos {
    /**
     * Contain, time and point for the ip
     */
    private $_logPoint          = array();
    private $_logTime           = array();
    private $_logIpNumber       = array();

    private $_firstLogPoint     = null;
    private $_firstLogTime      = null;
    private $_firstLogIpNumber  = null;

    private $_banned = array();

    /**
     * Contain the point use to ban people
     */
    private $_point = array();

    /**
     * Time Of last Accessed
     */
    private $_lastAccessed = array();

    private $_extension = array();
    private $_extensionPattern;
    private $_time  = null;
    private $_count = 0;
    private $_ban_path = 'ban';
    private $_points_path = 'points';
    private $_access_path = 'access';

    public function __construct(){
        $this->_extension = array(
            'Image' => "(?(JPG|GIF|BMP|JPEG|PNG))?",
            'Css' => "(?CSS)?",
            'Javascript' => "(?JS)?",
            'Other' => "(?.)?",
        );

        $this->_extensionPattern = implode('',$this->_extension);
        $this->_lastAccessed = array(
            'Image'=>array(),
            'Css'=>array(),
            'Javascript'=>array(),
            'Other'=>array(),
        );
        $this->_ban_path = LOGGING.'ban';
        $this->_points_path = LOGGING.'points';
        $this->_access_path = LOGGING.'access';
    }

    /**
     * Convert the string time from lighttpd to Unix time.
     * @param string $time
     */
    private function _strtotime($time){
        $time = explode('[',$time);
        $time = explode(' ',$time[1]);
        list($date,$hour,$minute,$second) = explode(':',$time[0]);
        list($day,$month,$year) = explode('/',$date);
        return strtotime("$day $month $year $hour:$minute:$second");
    }

    private function _nextLog(){
        if(!isset($this->_firstLogTime)){
            if(!empty($this->_logTime)){
                $this->_firstLogTime = array_shift($this->_logTime);
                $this->_firstLogPoint = array_shift($this->_logPoint);
                $this->_firstLogIpNumber = array_shift($this->_logIpNumber);
            } else {
                return false;
            }
        }
        return $this->_firstLogTime < $this->_time;
    }

    private function _cleanUp(){
        while($this->_nextLog()){
            $this->_point[$this->_firstLogIpNumber] -= $this->_firstLogPoint;

            if($this->_point[$this->_firstLogIpNumber] <= 0){
                unset($this->_point[$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Image'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Css'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Javascript'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Other'][$this->_firstLogIpNumber]);
            }

            unset($this->_firstLogTime);
            unset($this->_firstLogPoint);
            unset($this->_firstLogIpNumber);
        }
    }

    private function _getHumanFactor($ipNumber){
        $human = 0.0000000001;
        $robot = 1;

        if($this->_lastAccessed['Image'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Css'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Javascript'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Other'][$ipNumber] > $this->_time){
            $human += 0.2;
        } else {
            $robot *= 2;
        }

        return $human / $robot;
    }

    private function _checkAgent($agent){
        if(stristr($agent,'.NET CLR')){
            $agent = 4;
        } else {
            $agent = 1;
        }
        return $agent;
    }

    private function _stringToIpNumber($string){
        return ip2long(array_pop(explode(':',$string)));
    }

    private function _checkReferred($referred){
        if($referred != '-'){
            $referred = 0.5;
        } else {
            $referred = 2;
        }
        return $referred;
    }

    private function _parse($log){
        if(!$log){
            return;
        }
        $this->_cleanUp();

        list ( $ipAddress, $time, $http, $method, $url, $status, $byte, $referred, $agent ) = (explode ( chr ( SEPARATOR ), $log ));
        // accesslog.format = "%h %t %H %m %U %X %b %{Referer}i\" \"%{User-Agent}i\""
        // accesslog.format = "%h^Z%t^Z%H^Z%m^Z%U^Z%X^Z%b^Z%{Referer}i\"^Z\"%{User-Agent}i\""
        // ^Z is ctrl+z in vim
        $this->_time = $time = $this->_strtotime($time);
        $ipNumber = $this->_stringToIpNumber($ipAddress);

        if(!array_key_exists($ipNumber, $this->_point)){
            $this->_point[$ipNumber] = 0;
            $this->_lastAccessed['Image'][$ipNumber] = 0;
            $this->_lastAccessed['Css'][$ipNumber] = 0;
            $this->_lastAccessed['Javascript'][$ipNumber] = 0;
            $this->_lastAccessed['Other'][$ipNumber] = 0;
        }

        preg_match ( "/{$this->_extensionPattern}$/i", $url, $extensionMatch );
        foreach($extensionMatch as $key => $value){
            if($value && !is_numeric($key)){
                $this->_lastAccessed[$key][$ipNumber] = $time + ACCESSED_EXPIRE;
            }
        }


        $referred = $this->_checkReferred($referred);
        $agent = $this->_checkAgent($agent);
        $human = $this->_getHumanFactor($ipNumber);


        $point = 1 / $human * $referred * $agent;
        $this->_logTime[] = $time + ACCESSED_POINT;
        $this->_logPoint[] = $point;
        $this->_logIpNumber[] = $ipNumber;

        $this->_point[$ipNumber] += $point;
        $this->ban($ipNumber);
    }

    public function ban($ipNumber){
        $ipAddress = long2ip($ipNumber);
        if($this->_point[$ipNumber] > ACCESSED_POINT){
            $this->_count++;
            $index = $this->_count % IPTABLES_BUCKET;
            $iptables = IPTABLES_PATH.IPTABLES_NAME.$index;
            `echo '+$ipAddress' > $iptables 2> /dev/null`;
            `echo '$ipAddress {$this->_point[$ipNumber]}' >> {$this->_ban_path}`;
        }
        `echo '$ipAddress {$this->_point[$ipNumber]}' >> {$this->_points_path}`;
    }

    public function run(){
        do{
            $log = trim(fgets(STDIN));
            `echo '{$log}' >> {$this->_access_path}`;
            $this->_parse($log);
        } while($log);
    }
}
// Check for memory leaks
$first = memory_get_usage();
//echo "Memory :" . memory_get_usage() . PHP_EOL;
$antiDos = new AntiDos();
//echo "Memory :" . (memory_get_usage() - $first) . PHP_EOL;
$antiDos->run();
//echo "Memory :" . (memory_get_usage() - $first) . PHP_EOL;

Tetris Version 2 Part 1

To read my tetris version one click here

Tetris Version 2, is a quantum leap compare to version one. 

• Reduce code; No more position management for each blocks
• Run faster; Taking advantage of groups and HTML ability.

The position are now handle using CSS and HTML. We are now using css properties float, allow DOM object to self align to the top left or right corner.

With out float

With Float, the object inside auto align. Instead of position each block, we can now just put 10 block in and they auto align.

For performance increase, instead of creating and destroying dom object, all object we use are now created before hand and reuse. With grouping and css float, we have our biggest gain come when the row are filled with blocks. In the previous version the blocks have to be remove and all the blocks on top have to move down. The new approach we just move the row to the top (O1).

Link List in JavaScript

I'm making tetris version 2, and it have a link list in it. Figure it a good idea to share :D.

Using the list list,
The following use inheritance in JavaScript and make a sub class Candy.

Candy = function(flavor){
  this.super = LinkListItem;
  this.super();
  this.flavor = flavor;
  this.toString = function(){
    return this.flavor + ' Candy' ;
  }
}

box = new LinkList();
box.append(new Candy('Apple'));
box.append(new Candy('Banana'));
box.append(new Candy('Orange'));
alert('I have '+box+' in my box');

Source Code

/**
 * A simple like list
 */
 
LinkList = function(){
  startItem = null;
  endItem = null;
  /**
   * Add an item to the end of the list
   */
  this.append = function (item){
    item.backward = this.endItem;
    item.forward = null;
    item.parent = this;
    if(this.endItem){
      this.endItem.forward = item
    }
    if(!this.startItem){
      this.startItem = item;
    }
    this.endItem = item;
  }
  /**
   * Add the item to the front of the list
   */
  this.prepend = function (item){
    item.forward = this.startItem;
    item.backward = null;
    item.parent = this;
    if(this.startItem){
      this.startItem.backward = item
    }
    this.startItem = item;
  }
  
  /**
   * Remove the item from the list
   */
  this.remove = function(item){
    if(item.backward){
      item.backward.forward = item.forward;
    }
    
    if(item.forward){
      item.forward.backward = item.backward;
    }
    
    if(this.startItem == item){
      this.startItem = item.forward;
    }

    if(this.endItem == item){
      this.endItem = item.backward;
    }
    item.parent = null;
  }
  
  this.toString = function(){
    var item = this.startItem;
    var items = new Array();
    while(item != null){
      items.push(item);
      item = item.forward;
    }
    if(!items.length){
      return "Empty";
    }
    return items.join(', ');
  }
}

/**
 * A link list item
 */
LinkListItem = function(){
  this.parent = null;
  this.forward = null;
  this.backward = null;

  this.remove = function(){
    if(this.parent){
      this.parent.remove(this);
    }
  }
  
  this.moveToFront = function(){
    if(this.parent){
      var parent = this.parent;
      parent.remove(this);
      parent.prepend(this);
    }
  }
  
  this.moveToBack = function(){
    if(this.parent){
      var parent = this.parent;
      parent.remove(this);
      parent.append(this);
    }
  }
}

/**
 * Testing Script
 *
Candy = function(flavor){
  this.inherit = LinkListItem;
  this.inherit();
  this.flavor = flavor;
  this.toString = function(){
    return this.flavor + ' Candy' ;
  }
}

var box = new LinkList();
box.append(new Candy('Apple'));
box.append(new Candy('Banana'));
box.append(new Candy('Orange'));
alert("What Do I have in my box? " + box);
/***/

UGotFile Stabilizer HOW TO

This is for the future people at UGotFile. I'm going to be talking about the stabilizer, what it do, and how to improve or fix error.

UGotFile stabilizer is a tool to automatically evenly distribute files between hard drive with in the server and on the network. The stabilizer achieve it task via http, a not very secure channel,but in our case it ideal, http don't cost CPU power or extra ram and it aren't cap at 100 Mbps per connection unlike SSH.

Stabilizer are split into three part

1. Gather accurate information about the server and auto index the server into the database.
   This part is pretty well done, it unlikely to cause any problem.
2. Volume that are 5% bigger then the average volume are selected to mark some of their files for export.
3. Volume that are smaller then the average volume are selected to download the files mark to be exported. Volume that are next to the exporting volume have higher priority.

Problems in paradise.
After the take over of UGotFile, all the hardware change, the special equipment we require cost a lot of money. We came up with some new ideas that increase the speed from each server from 200Mbps to beyond 1000Mbps. This is one step of the plan, to use each volume separates. This might shock some people but removing raid increase performance with concurrent connection. Some will ask how about the data integrity, well we can always check for a hard dive that is about to die and move their data away.

The problem is before the take over of ugotfile, we where doing some upgrade and it's half done. Then we made another upgrade. The volume have three different file structural and the database also have three tables. All three set still works nicely.

1. All files are store in one volumes
2. file are split using mod by the number of volumes.
3. Each volumes have it own file structure, the volumes can be transfer between server and automatically work.

TODO: Remove initial, and second upgrade, be warned there are some system that is still using them.

if some one where to delete the old tables, one of the problem that will occurred is the server will run out of space with in a few month and crash.

TODO: Make some tool to clean up files that aren't register in the database for third upgrade. 

How to run the stabilizer:

/etc/init.d/admin.uGotFile.com.Stabilizers

Please do not run it from any where else, that script prevent some conflict and automatically launch the program if it crash. There is an inbuild timer for the stabilizer to stop every so often, just use /etc/init.d/admin.uGotFile.com.Stabilizers it better.

Problem that you may encounter 

The stabilizers are running and nothing is happening. There are multiple system running ugotfile.com and one of them is the automatic deletion of old files. The problem is that the system don't notify the stabilizers.

Quick Fix: Delete everything from the stabilizer database.

Solution 1: Fix the store procedure to clean up the stabilizer database.

Solution 2: Make a foreign key with cascade delete from table "file__volume" to table "stabilizer"

After doing the quick fix or applying the solution and it still do nothing.

Quick Fix: Run "find * -size 0c -exec rm -f '{}' \;" in each volume to delete files that are zero size in length, and replicate "/home/web/admin.ugotfile.com/AdminUGotFile/models/Stabilizers.php" from server 13 code over to the other server. Just add some checking to the code. 

Solution 1: Check http upload and ftp upload. For error they should both delete zero size files. FTP upload should be the problem.

Solution 2: Modified the stabilizer to ignore zero size files.

Again after all your hard work and it look like it's doing nothing. Then you have been fool by your on own negligence. You have let the volume fill up with out running the stabilizer and there are now corrupted files with zero size length, but registered XXX byte in the database. The problem will go away after 2 month.
Quick Fix: NONE
Solution 1: Use "find * -size 0c -exec rm -f '{}' \;" on the volume, mark the volume as "dying", run the stabilizer. Mart the volume as "enable" and truncate the stabilizer table. Before all of this you need to replicate /home/web/admin.ugotfile.com/AdminUGotFile/models/Stabilizers.php from server 13.
Solution 2: Change the display for the admin control to use the stabilizer statistics, it way more accurate.


If the solution are apply all the above will not happen again.

Good Luck Bob,

Cheers.