DOS, DDOS, BOTNET -- IT ARENT A PROBLEM

The follow are aim at hackers that aren't very smart (average hacker), the smarter hacker normally don't play around with you.

A friend of mine own a money making on line forum, and a group of hackers want to blackmail him for benefits. At first they start with a single ip to DOS the server. The server went down and my friend ask me for help.

I install a simple iptable to block attack from a single source and swap apache for light httpd. The server stay up for another few days until they start using DDOS and BOTNET.

When they start using botnet I made the program below, since the introduction of the program the server never crash again. At the peak of the attack there was 8000 ips and 26000 connection and the cpu was using 1-2%.


The basic of DDOS is any one of the following.
Resource Starvation: CPU 100%, RAM 100% or BandWidth 100%.

What to do when your under attack?

1. Go in and turn off your web service.
   You can't do anything if your server hang :D.
2. Run the following command, it change the priority of the process being attack, so your server wont hang.
   echo "www-data hard priority 10" >> /etc/security/limits.conf
   For more detail vist here http://kissconcept.blogspot.com/2010/09/set-default-priority-for-all-processes.html
3. Install the little script below, name Anti-Dos :D.

What do anti-dos do?
It take access data from your web service, analysis the data and determine the bad guys.
It use iptables to block people, pretty sweet. The best part is that it unbad the bad guys if they stop attacking after x duration.

What web service can it run with?

• lighttpd
• for nginx and apache it require a bit of change.

Iptables:
The following script create 100 bucket in iptables, and each bucket can hold 100 ips.
If some one can extend the amount of ips in the bucket give me call :D

Put the code in a php file

<?php 

`iptables -F`; 

for($index=0; $index <100; $index++){ 

    `iptables -A INPUT -p tcp --dport 80 -m recent --update --seconds 1800 --name ROBOT$index -j DROP`;
}

`chmod 777 /proc/net/ipt_recent/*` 

Configure Lighttpd:

•  Find a place to put the script and logs,
  
  • antidos : /home/www-data/antidos.php
  • log: /home/www-data/log/
  • error: home/www-data/error
  • Secret:^Z --> single letter
• Edit your antidos.php
  define ( 'LOGGING', '/home/www-data/log/');
• Edit /etc/lighttpd/lighttpd.conf
  
  • accesslog.filename = "| /usr/bin/php /home/www-data/antidos.php 2> /home/www-data/error"
  • accesslog.format = "%h^Z%t^Z%H^Z%m^Z%U^Z%X^Z%b^Z%{Referer}i^Z%{User-Agent}i"
    in vim ^Z is for 26 character code, you can type it by using ctrl+z in vim.

That's it :D, restart your lighttpd and wait for 2-5 min.
Your cpu usage should drop to 0%.

If it don't check for activity in the log folder and the error file.
you can use tail -f /home/www-data/log/access to follow the log

<?php

define ( 'SEPARATOR', 26 );
define ( 'ACCESSED_EXPIRE', 600);
define ( 'ACCESSED_POINT', 600); // Point to start banning people
define ( 'IPTABLES_NAME', 'ROBOT');
define ( 'IPTABLES_BUCKET', 100); // Including 0
define ( 'IPTABLES_PATH', '/proc/net/ipt_recent/');
define ( 'LOGGING', '/home/www-data/log/');

class AntiDos {
    /**
     * Contain, time and point for the ip
     */
    private $_logPoint          = array();
    private $_logTime           = array();
    private $_logIpNumber       = array();

    private $_firstLogPoint     = null;
    private $_firstLogTime      = null;
    private $_firstLogIpNumber  = null;

    private $_banned = array();

    /**
     * Contain the point use to ban people
     */
    private $_point = array();

    /**
     * Time Of last Accessed
     */
    private $_lastAccessed = array();

    private $_extension = array();
    private $_extensionPattern;
    private $_time  = null;
    private $_count = 0;
    private $_ban_path = 'ban';
    private $_points_path = 'points';
    private $_access_path = 'access';

    public function __construct(){
        $this->_extension = array(
            'Image' => "(?(JPG|GIF|BMP|JPEG|PNG))?",
            'Css' => "(?CSS)?",
            'Javascript' => "(?JS)?",
            'Other' => "(?.)?",
        );

        $this->_extensionPattern = implode('',$this->_extension);
        $this->_lastAccessed = array(
            'Image'=>array(),
            'Css'=>array(),
            'Javascript'=>array(),
            'Other'=>array(),
        );
        $this->_ban_path = LOGGING.'ban';
        $this->_points_path = LOGGING.'points';
        $this->_access_path = LOGGING.'access';
    }

    /**
     * Convert the string time from lighttpd to Unix time.
     * @param string $time
     */
    private function _strtotime($time){
        $time = explode('[',$time);
        $time = explode(' ',$time[1]);
        list($date,$hour,$minute,$second) = explode(':',$time[0]);
        list($day,$month,$year) = explode('/',$date);
        return strtotime("$day $month $year $hour:$minute:$second");
    }

    private function _nextLog(){
        if(!isset($this->_firstLogTime)){
            if(!empty($this->_logTime)){
                $this->_firstLogTime = array_shift($this->_logTime);
                $this->_firstLogPoint = array_shift($this->_logPoint);
                $this->_firstLogIpNumber = array_shift($this->_logIpNumber);
            } else {
                return false;
            }
        }
        return $this->_firstLogTime < $this->_time;
    }

    private function _cleanUp(){
        while($this->_nextLog()){
            $this->_point[$this->_firstLogIpNumber] -= $this->_firstLogPoint;

            if($this->_point[$this->_firstLogIpNumber] <= 0){
                unset($this->_point[$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Image'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Css'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Javascript'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Other'][$this->_firstLogIpNumber]);
            }

            unset($this->_firstLogTime);
            unset($this->_firstLogPoint);
            unset($this->_firstLogIpNumber);
        }
    }

    private function _getHumanFactor($ipNumber){
        $human = 0.0000000001;
        $robot = 1;

        if($this->_lastAccessed['Image'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Css'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Javascript'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Other'][$ipNumber] > $this->_time){
            $human += 0.2;
        } else {
            $robot *= 2;
        }

        return $human / $robot;
    }

    private function _checkAgent($agent){
        if(stristr($agent,'.NET CLR')){
            $agent = 4;
        } else {
            $agent = 1;
        }
        return $agent;
    }

    private function _stringToIpNumber($string){
        return ip2long(array_pop(explode(':',$string)));
    }

    private function _checkReferred($referred){
        if($referred != '-'){
            $referred = 0.5;
        } else {
            $referred = 2;
        }
        return $referred;
    }

    private function _parse($log){
        if(!$log){
            return;
        }
        $this->_cleanUp();

        list ( $ipAddress, $time, $http, $method, $url, $status, $byte, $referred, $agent ) = (explode ( chr ( SEPARATOR ), $log ));
        // accesslog.format = "%h %t %H %m %U %X %b %{Referer}i\" \"%{User-Agent}i\""
        // accesslog.format = "%h^Z%t^Z%H^Z%m^Z%U^Z%X^Z%b^Z%{Referer}i\"^Z\"%{User-Agent}i\""
        // ^Z is ctrl+z in vim
        $this->_time = $time = $this->_strtotime($time);
        $ipNumber = $this->_stringToIpNumber($ipAddress);

        if(!array_key_exists($ipNumber, $this->_point)){
            $this->_point[$ipNumber] = 0;
            $this->_lastAccessed['Image'][$ipNumber] = 0;
            $this->_lastAccessed['Css'][$ipNumber] = 0;
            $this->_lastAccessed['Javascript'][$ipNumber] = 0;
            $this->_lastAccessed['Other'][$ipNumber] = 0;
        }

        preg_match ( "/{$this->_extensionPattern}$/i", $url, $extensionMatch );
        foreach($extensionMatch as $key => $value){
            if($value && !is_numeric($key)){
                $this->_lastAccessed[$key][$ipNumber] = $time + ACCESSED_EXPIRE;
            }
        }


        $referred = $this->_checkReferred($referred);
        $agent = $this->_checkAgent($agent);
        $human = $this->_getHumanFactor($ipNumber);


        $point = 1 / $human * $referred * $agent;
        $this->_logTime[] = $time + ACCESSED_POINT;
        $this->_logPoint[] = $point;
        $this->_logIpNumber[] = $ipNumber;

        $this->_point[$ipNumber] += $point;
        $this->ban($ipNumber);
    }

    public function ban($ipNumber){
        $ipAddress = long2ip($ipNumber);
        if($this->_point[$ipNumber] > ACCESSED_POINT){
            $this->_count++;
            $index = $this->_count % IPTABLES_BUCKET;
            $iptables = IPTABLES_PATH.IPTABLES_NAME.$index;
            `echo '+$ipAddress' > $iptables 2> /dev/null`;
            `echo '$ipAddress {$this->_point[$ipNumber]}' >> {$this->_ban_path}`;
        }
        `echo '$ipAddress {$this->_point[$ipNumber]}' >> {$this->_points_path}`;
    }

    public function run(){
        do{
            $log = trim(fgets(STDIN));
            `echo '{$log}' >> {$this->_access_path}`;
            $this->_parse($log);
        } while($log);
    }
}
// Check for memory leaks
$first = memory_get_usage();
//echo "Memory :" . memory_get_usage() . PHP_EOL;
$antiDos = new AntiDos();
//echo "Memory :" . (memory_get_usage() - $first) . PHP_EOL;
$antiDos->run();
//echo "Memory :" . (memory_get_usage() - $first) . PHP_EOL;

Tetris Version 2 Part 1

To read my tetris version one click here

Tetris Version 2, is a quantum leap compare to version one. 

• Reduce code; No more position management for each blocks
• Run faster; Taking advantage of groups and HTML ability.

The position are now handle using CSS and HTML. We are now using css properties float, allow DOM object to self align to the top left or right corner.

With out float

With Float, the object inside auto align. Instead of position each block, we can now just put 10 block in and they auto align.

For performance increase, instead of creating and destroying dom object, all object we use are now created before hand and reuse. With grouping and css float, we have our biggest gain come when the row are filled with blocks. In the previous version the blocks have to be remove and all the blocks on top have to move down. The new approach we just move the row to the top (O1).

Link List in JavaScript

I'm making tetris version 2, and it have a link list in it. Figure it a good idea to share :D.

Using the list list,
The following use inheritance in JavaScript and make a sub class Candy.

Candy = function(flavor){
  this.super = LinkListItem;
  this.super();
  this.flavor = flavor;
  this.toString = function(){
    return this.flavor + ' Candy' ;
  }
}

box = new LinkList();
box.append(new Candy('Apple'));
box.append(new Candy('Banana'));
box.append(new Candy('Orange'));
alert('I have '+box+' in my box');

Source Code

/**
 * A simple like list
 */
 
LinkList = function(){
  startItem = null;
  endItem = null;
  /**
   * Add an item to the end of the list
   */
  this.append = function (item){
    item.backward = this.endItem;
    item.forward = null;
    item.parent = this;
    if(this.endItem){
      this.endItem.forward = item
    }
    if(!this.startItem){
      this.startItem = item;
    }
    this.endItem = item;
  }
  /**
   * Add the item to the front of the list
   */
  this.prepend = function (item){
    item.forward = this.startItem;
    item.backward = null;
    item.parent = this;
    if(this.startItem){
      this.startItem.backward = item
    }
    this.startItem = item;
  }
  
  /**
   * Remove the item from the list
   */
  this.remove = function(item){
    if(item.backward){
      item.backward.forward = item.forward;
    }
    
    if(item.forward){
      item.forward.backward = item.backward;
    }
    
    if(this.startItem == item){
      this.startItem = item.forward;
    }

    if(this.endItem == item){
      this.endItem = item.backward;
    }
    item.parent = null;
  }
  
  this.toString = function(){
    var item = this.startItem;
    var items = new Array();
    while(item != null){
      items.push(item);
      item = item.forward;
    }
    if(!items.length){
      return "Empty";
    }
    return items.join(', ');
  }
}

/**
 * A link list item
 */
LinkListItem = function(){
  this.parent = null;
  this.forward = null;
  this.backward = null;

  this.remove = function(){
    if(this.parent){
      this.parent.remove(this);
    }
  }
  
  this.moveToFront = function(){
    if(this.parent){
      var parent = this.parent;
      parent.remove(this);
      parent.prepend(this);
    }
  }
  
  this.moveToBack = function(){
    if(this.parent){
      var parent = this.parent;
      parent.remove(this);
      parent.append(this);
    }
  }
}

/**
 * Testing Script
 *
Candy = function(flavor){
  this.inherit = LinkListItem;
  this.inherit();
  this.flavor = flavor;
  this.toString = function(){
    return this.flavor + ' Candy' ;
  }
}

var box = new LinkList();
box.append(new Candy('Apple'));
box.append(new Candy('Banana'));
box.append(new Candy('Orange'));
alert("What Do I have in my box? " + box);
/***/

UGotFile Stabilizer HOW TO

This is for the future people at UGotFile. I'm going to be talking about the stabilizer, what it do, and how to improve or fix error.

UGotFile stabilizer is a tool to automatically evenly distribute files between hard drive with in the server and on the network. The stabilizer achieve it task via http, a not very secure channel,but in our case it ideal, http don't cost CPU power or extra ram and it aren't cap at 100 Mbps per connection unlike SSH.

Stabilizer are split into three part

1. Gather accurate information about the server and auto index the server into the database.
   This part is pretty well done, it unlikely to cause any problem.
2. Volume that are 5% bigger then the average volume are selected to mark some of their files for export.
3. Volume that are smaller then the average volume are selected to download the files mark to be exported. Volume that are next to the exporting volume have higher priority.

Problems in paradise.
After the take over of UGotFile, all the hardware change, the special equipment we require cost a lot of money. We came up with some new ideas that increase the speed from each server from 200Mbps to beyond 1000Mbps. This is one step of the plan, to use each volume separates. This might shock some people but removing raid increase performance with concurrent connection. Some will ask how about the data integrity, well we can always check for a hard dive that is about to die and move their data away.

The problem is before the take over of ugotfile, we where doing some upgrade and it's half done. Then we made another upgrade. The volume have three different file structural and the database also have three tables. All three set still works nicely.

1. All files are store in one volumes
2. file are split using mod by the number of volumes.
3. Each volumes have it own file structure, the volumes can be transfer between server and automatically work.

TODO: Remove initial, and second upgrade, be warned there are some system that is still using them.

if some one where to delete the old tables, one of the problem that will occurred is the server will run out of space with in a few month and crash.

TODO: Make some tool to clean up files that aren't register in the database for third upgrade. 

How to run the stabilizer:

/etc/init.d/admin.uGotFile.com.Stabilizers

Please do not run it from any where else, that script prevent some conflict and automatically launch the program if it crash. There is an inbuild timer for the stabilizer to stop every so often, just use /etc/init.d/admin.uGotFile.com.Stabilizers it better.

Problem that you may encounter 

The stabilizers are running and nothing is happening. There are multiple system running ugotfile.com and one of them is the automatic deletion of old files. The problem is that the system don't notify the stabilizers.

Quick Fix: Delete everything from the stabilizer database.

Solution 1: Fix the store procedure to clean up the stabilizer database.

Solution 2: Make a foreign key with cascade delete from table "file__volume" to table "stabilizer"

After doing the quick fix or applying the solution and it still do nothing.

Quick Fix: Run "find * -size 0c -exec rm -f '{}' \;" in each volume to delete files that are zero size in length, and replicate "/home/web/admin.ugotfile.com/AdminUGotFile/models/Stabilizers.php" from server 13 code over to the other server. Just add some checking to the code. 

Solution 1: Check http upload and ftp upload. For error they should both delete zero size files. FTP upload should be the problem.

Solution 2: Modified the stabilizer to ignore zero size files.

Again after all your hard work and it look like it's doing nothing. Then you have been fool by your on own negligence. You have let the volume fill up with out running the stabilizer and there are now corrupted files with zero size length, but registered XXX byte in the database. The problem will go away after 2 month.
Quick Fix: NONE
Solution 1: Use "find * -size 0c -exec rm -f '{}' \;" on the volume, mark the volume as "dying", run the stabilizer. Mart the volume as "enable" and truncate the stabilizer table. Before all of this you need to replicate /home/web/admin.ugotfile.com/AdminUGotFile/models/Stabilizers.php from server 13.
Solution 2: Change the display for the admin control to use the stabilizer statistics, it way more accurate.


If the solution are apply all the above will not happen again.

Good Luck Bob,

Cheers.

Captcha - TPG - Part 1

Today, I was planning to change my internet to unlimited with TPG. To my surprise it will take 3 weeks to completed, it was disappointing. The cool thing is that TPG captcha look really cool, so we are going to crack it.
For a live demo go here.
https://cyberstore.tpg.com.au/register/homephone/images/generateimage.php

Feature of this captcha:

• Random rotation
• Random character and numbers
• Random Font
• Fix background distortion
• Random circle background distortion

This is a good captcha, but it not a great captcha, to be great it need a bit more work

• Random font size
• Tilting of image vertical and horisontal
• Random width for circle background distortion

Luckly for us they didn't have any of the three missing feature.
The first step is to remove the background like below.

 The image was clean using python with python image library. What happen is that each pixcel are compare with it surrounding, the ones with not so much surrounding are remove.

The next step is to identify the character, it a bit tricky, the approach I'm using is assist learning for the programs. The program is going to get a set of captcha and link the same character together. Since the captch don't know what the character are, we have to let it know watch each picture mean. This will involve assigning 26 character and 10 digit.

Stay tune for part 2, I'll go into more detail for identify the characters, and some cool method to hack chaptcha that are distorted via tilting.