Thursday 9 December 2010

Ubuntu 10.10, 64Bit, Blue Tooth.

If your blue tooth is not working, here is a solution for you :D, fingers cross and hope it work. I have been searching the web ever since I got my brand new blue tooth keyboard over a year ago. Today I lucky stumble up on www.bluez.org and they just release a new bluez that claim to fix regression. Unfortunately ubuntu don't have it in the repository yet so I have to build it. Excited, I try it out and it works for me, hopefully it work for you too.

The solution is not perfect, but I have restarted my computer a few time and my blue tooth keyboard still worked nicely.

We are going to compile the latest bluez 4.81, with out the regression that have dread the ubuntu blue tooth since version 9.4(that when I got my beautiful keyboard)

Installing some dependency
sudo aptitude install build-essential bison zlib1g-dev zlib1g

Visit http://www.gtk.org/download-linux.html for the latest glib
wget http://ftp.gnome.org/pub/gnome/sources/glib/2.26/glib-2.26.1.tar.gz
cd /glib-2.26.1/
./configure
make 
sudo make install

Visit http://www.bluez.org for the latest drivers
wget http://www.kernel.org/pub/linux/bluetooth/bluez-4.81.tar.gz
tar -xvf bluez*.tar.gz
cd bluez-4.81
./configure
make
sudo make install
Restart your computer and it should work beautifully.

At this point I was feeling pretty happy so I play around abit and screw it up :D
If you are still experiencing problem try the following:
Check hciconfig in console
hciconfig 
hci0: Type: BR/EDR  Bus: USB
 BD Address: 00:1F:81:00:02:50  ACL MTU: 1021:4  SCO MTU: 180:1
 UP RUNNING PSCAN 
 RX bytes:80028 acl:4395 sco:0 events:52 errors:0
 TX bytes:617 acl:12 sco:0 commands:28 errors:0

If you see "UP RUNNING PSCAN", then you can start adding new devices and start using your blue tooth.

Otherwise do the following:
Bring down the interface
sudo hciconfig hci0 down

Bring up the interface
sudo hciconfig hci0 up
Bring the interface up can cause some problems.
If the interface keep timing out pull the blue tooth out and plug it back in.
If hciconfig complain about rfkill do the following
sudo rfkill unblock bluetooth
Go back and check

Hopefully ubuntu team integrate the new bluez fix as soon as possible.

Good Luck.

Wednesday 8 December 2010

Forex MetaTrader 5 HCC to CVS

Meta Trader 5, store history data in HCC files.
The quickest ways to get all the history data is to run a Strategy Tester on the entire history.

History files are store at MetaTrader 5\Bases\MetaQuotes-Demo\history\.

Don't exactly know what's the entire structural, but know enough to extract what we want, the history data.

Data Structural


4 byte, seperator , little endian encoding 18385028, hex(84 88 18 01)
4 byte, time, int divisible by 60
8 byte, double open
8 byte, double high
8 byte, double low
8 byte, double close
1 byte, char | small int spread
1 byte, char | small int tick volume

The data repeat over and over for each day.

I made one in c++, it ugle :D,
Just switch to ubuntu again, lost all my IDE :(
SRC: http://example.citium.net/forex/HccReader.cpp
BIN: http://example.citium.net/forex/HccReader

Friday 12 November 2010

Reset MySql Password

I was having trouble following instruct on some website with using mysqladmin, 
some how it didn't work, luckily I remember the method below.

Reseting your mysql password

1 .You need to run mysql in safe-mode
safe_mysqld --user=mysql --skip-grant-tables --skip-networking &

2. Access mysql:
mysql

3. Get a list of user from the database
use mysql;
select * from user;

4. Pick a user to change their password.
In my case it was a fresh install so I change all the user password
update user set password=password('newPassword');

5. Flush the privileges
FLUSH PRIVILEGES

6. Restart mysql
killall mysqld
service mysql start

Thursday 11 November 2010

UGotFile Undermaintance

It pretty heart breaking to see "Under Network Maintenance. Please come back later" covering up two years of your hard works at ugotfile.com. This is a really big issue, there were never any problem that can't be solve with the server still running. All other server is redundancy for the main server so, it's impossible to failed.
I'm totally surprise, to redeem this horrible incident I'm going to host another copy of ugotfile.com at citium.net. It's unfortunate but there aren't going to be any files at citium.net. The process is going to take some time, the database is 5GB, going to takes ages to upload.

Here is some picture of me being happy at the data center.

Cheers

Wednesday 27 October 2010

HTML5 -- Possiblity

I have been hearing about html5 for a while now.
Finally took a peak at it 3 days ago and it was awesome.

HTML5 is a flash killer!
Now you can integrate sound, video, image manipulation, SVG right into your web page.



Here are some pit holes when I was testing canvas:
  • canvas.getContext('2d').putImagedata is slow when using large offset
  • creating 1000 canvas consume 500-800mbs, better to create one huge canvas for all your buffers.
I always dislike flash, so I'll take the plunge into HTML5.
I'm going to be creating a game with HTML5.
It a real time strategy game know as Z, it's around 400 mbs. But I have figure a way to shrink it down to under 3 mb.

Plan:
  • Decode the games rules, stats for unit and terrain 0%
  • Extract tiles, sprites, and audio. 60%
  • Extract original levels 5%
  • Loss-less Compress of resources for web usage.30%
  • JavaScript Engine to run the game
  • AI for the single player
  • hack in to google/facebook for multiplayer

Beside HTML5 awesomeness, IE9 were faster then firefox 3.
go here to test out your browser with html5 http://ie.microsoft.com/testdrive/

Sunday 24 October 2010

DOS, DDOS, BOTNET -- IT ARENT A PROBLEM

The follow are aim at hackers that aren't very smart (average hacker), the smarter hacker normally don't play around with you.

A friend of mine own a money making on line forum, and a group of hackers want to blackmail him for benefits. At first they start with a single ip to DOS the server. The server went down and my friend ask me for help.

I install a simple iptable to block attack from a single source and swap apache for light httpd. The server stay up for another few days until they start using DDOS and BOTNET.

When they start using botnet I made the program below, since the introduction of the program the server never crash again. At the peak of the attack there was 8000 ips and 26000 connection and the cpu was using 1-2%.


The basic of DDOS is any one of the following.
Resource Starvation: CPU 100%, RAM 100% or BandWidth 100%.

What to do when your under attack?
  1. Go in and turn off your web service.
    You can't do anything if your server hang :D.
  2. Run the following command, it change the priority of the process being attack, so your server wont hang.
    echo "
    www-data hard priority 10" >>
    /etc/security/limits.conf
    For more detail vist here 
    http://kissconcept.blogspot.com/2010/09/set-default-priority-for-all-processes.html
  3. Install the little script below, name Anti-Dos :D.


What do anti-dos do?
It take access data from your web service, analysis the data and determine the bad guys.
It use iptables to block people, pretty sweet. The best part is that it unbad the bad guys if they stop attacking after x duration.

What web service can it run with?
  • lighttpd
  • for nginx and apache it require a bit of change.

Iptables:
The following script create 100 bucket in iptables, and each bucket can hold 100 ips.
If some one can extend the amount of ips in the bucket give me call :D

Put the code in a php file
<?php 
 
`iptables -F`; 
 
for($index=0; $index <100; $index++){ 
    `iptables -A INPUT -p tcp --dport 80 -m recent --update --seconds 1800 --name ROBOT$index -j DROP`;
}
 
`chmod 777 /proc/net/ipt_recent/*` 
 


Configure Lighttpd:
  •  Find a place to put the script and logs,
    • antidos : /home/www-data/antidos.php
    • log: /home/www-data/log/
    • error: home/www-data/error
    • Secret:^Z --> single letter
  • Edit your antidos.php
    define ( 'LOGGING', '/home/www-data/log/');
  • Edit /etc/lighttpd/lighttpd.conf
    • accesslog.filename = "| /usr/bin/php /home/www-data/antidos.php 2> /home/www-data/error"
    • accesslog.format = "%h^Z%t^Z%H^Z%m^Z%U^Z%X^Z%b^Z%{Referer}i^Z%{User-Agent}i"
      in vim ^Z is for 26 character code, you can type it by using ctrl+z in vim.
That's it :D, restart your lighttpd and wait for 2-5 min.
Your cpu usage should drop to 0%.

If it don't check for activity in the log folder and the error file.
you can use tail -f /home/www-data/log/access to follow the log

<?php

define ( 'SEPARATOR', 26 );
define ( 'ACCESSED_EXPIRE', 600);
define ( 'ACCESSED_POINT', 600); // Point to start banning people
define ( 'IPTABLES_NAME', 'ROBOT');
define ( 'IPTABLES_BUCKET', 100); // Including 0
define ( 'IPTABLES_PATH', '/proc/net/ipt_recent/');
define ( 'LOGGING', '/home/www-data/log/');

class AntiDos {
    /**
     * Contain, time and point for the ip
     */
    private $_logPoint          = array();
    private $_logTime           = array();
    private $_logIpNumber       = array();

    private $_firstLogPoint     = null;
    private $_firstLogTime      = null;
    private $_firstLogIpNumber  = null;

    private $_banned = array();

    /**
     * Contain the point use to ban people
     */
    private $_point = array();

    /**
     * Time Of last Accessed
     */
    private $_lastAccessed = array();

    private $_extension = array();
    private $_extensionPattern;
    private $_time  = null;
    private $_count = 0;
    private $_ban_path = 'ban';
    private $_points_path = 'points';
    private $_access_path = 'access';

    public function __construct(){
        $this->_extension = array(
            'Image' => "(?(JPG|GIF|BMP|JPEG|PNG))?",
            'Css' => "(?CSS)?",
            'Javascript' => "(?JS)?",
            'Other' => "(?.)?",
        );

        $this->_extensionPattern = implode('',$this->_extension);
        $this->_lastAccessed = array(
            'Image'=>array(),
            'Css'=>array(),
            'Javascript'=>array(),
            'Other'=>array(),
        );
        $this->_ban_path = LOGGING.'ban';
        $this->_points_path = LOGGING.'points';
        $this->_access_path = LOGGING.'access';
    }

    /**
     * Convert the string time from lighttpd to Unix time.
     * @param string $time
     */
    private function _strtotime($time){
        $time = explode('[',$time);
        $time = explode(' ',$time[1]);
        list($date,$hour,$minute,$second) = explode(':',$time[0]);
        list($day,$month,$year) = explode('/',$date);
        return strtotime("$day $month $year $hour:$minute:$second");
    }

    private function _nextLog(){
        if(!isset($this->_firstLogTime)){
            if(!empty($this->_logTime)){
                $this->_firstLogTime = array_shift($this->_logTime);
                $this->_firstLogPoint = array_shift($this->_logPoint);
                $this->_firstLogIpNumber = array_shift($this->_logIpNumber);
            } else {
                return false;
            }
        }
        return $this->_firstLogTime < $this->_time;
    }

    private function _cleanUp(){
        while($this->_nextLog()){
            $this->_point[$this->_firstLogIpNumber] -= $this->_firstLogPoint;

            if($this->_point[$this->_firstLogIpNumber] <= 0){
                unset($this->_point[$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Image'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Css'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Javascript'][$this->_firstLogIpNumber]);
                unset($this->_lastAccessed['Other'][$this->_firstLogIpNumber]);
            }

            unset($this->_firstLogTime);
            unset($this->_firstLogPoint);
            unset($this->_firstLogIpNumber);
        }
    }

    private function _getHumanFactor($ipNumber){
        $human = 0.0000000001;
        $robot = 1;

        if($this->_lastAccessed['Image'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Css'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Javascript'][$ipNumber] > $this->_time){
            $human += 0.4;
        } else {
            $robot *= 2;
        }

        if($this->_lastAccessed['Other'][$ipNumber] > $this->_time){
            $human += 0.2;
        } else {
            $robot *= 2;
        }

        return $human / $robot;
    }

    private function _checkAgent($agent){
        if(stristr($agent,'.NET CLR')){
            $agent = 4;
        } else {
            $agent = 1;
        }
        return $agent;
    }

    private function _stringToIpNumber($string){
        return ip2long(array_pop(explode(':',$string)));
    }

    private function _checkReferred($referred){
        if($referred != '-'){
            $referred = 0.5;
        } else {
            $referred = 2;
        }
        return $referred;
    }

    private function _parse($log){
        if(!$log){
            return;
        }
        $this->_cleanUp();

        list ( $ipAddress, $time, $http, $method, $url, $status, $byte, $referred, $agent ) = (explode ( chr ( SEPARATOR ), $log ));
        // accesslog.format = "%h %t %H %m %U %X %b %{Referer}i\" \"%{User-Agent}i\""
        // accesslog.format = "%h^Z%t^Z%H^Z%m^Z%U^Z%X^Z%b^Z%{Referer}i\"^Z\"%{User-Agent}i\""
        // ^Z is ctrl+z in vim
        $this->_time = $time = $this->_strtotime($time);
        $ipNumber = $this->_stringToIpNumber($ipAddress);

        if(!array_key_exists($ipNumber, $this->_point)){
            $this->_point[$ipNumber] = 0;
            $this->_lastAccessed['Image'][$ipNumber] = 0;
            $this->_lastAccessed['Css'][$ipNumber] = 0;
            $this->_lastAccessed['Javascript'][$ipNumber] = 0;
            $this->_lastAccessed['Other'][$ipNumber] = 0;
        }

        preg_match ( "/{$this->_extensionPattern}$/i", $url, $extensionMatch );
        foreach($extensionMatch as $key => $value){
            if($value && !is_numeric($key)){
                $this->_lastAccessed[$key][$ipNumber] = $time + ACCESSED_EXPIRE;
            }
        }


        $referred = $this->_checkReferred($referred);
        $agent = $this->_checkAgent($agent);
        $human = $this->_getHumanFactor($ipNumber);


        $point = 1 / $human * $referred * $agent;
        $this->_logTime[] = $time + ACCESSED_POINT;
        $this->_logPoint[] = $point;
        $this->_logIpNumber[] = $ipNumber;

        $this->_point[$ipNumber] += $point;
        $this->ban($ipNumber);
    }

    public function ban($ipNumber){
        $ipAddress = long2ip($ipNumber);
        if($this->_point[$ipNumber] > ACCESSED_POINT){
            $this->_count++;
            $index = $this->_count % IPTABLES_BUCKET;
            $iptables = IPTABLES_PATH.IPTABLES_NAME.$index;
            `echo '+$ipAddress' > $iptables 2> /dev/null`;
            `echo '$ipAddress {$this->_point[$ipNumber]}' >> {$this->_ban_path}`;
        }
        `echo '$ipAddress {$this->_point[$ipNumber]}' >> {$this->_points_path}`;
    }

    public function run(){
        do{
            $log = trim(fgets(STDIN));
            `echo '{$log}' >> {$this->_access_path}`;
            $this->_parse($log);
        } while($log);
    }
}
// Check for memory leaks
$first = memory_get_usage();
//echo "Memory :" . memory_get_usage() . PHP_EOL;
$antiDos = new AntiDos();
//echo "Memory :" . (memory_get_usage() - $first) . PHP_EOL;
$antiDos->run();
//echo "Memory :" . (memory_get_usage() - $first) . PHP_EOL;



Thursday 14 October 2010

Tetris Version 2 Part 1

To read my tetris version one click here


Tetris Version 2, is a quantum leap compare to version one. 
  • Reduce code; No more position management for each blocks
  • Run faster; Taking advantage of groups and HTML ability.
The position are now handle using CSS and HTML. We are now using css properties float, allow DOM object to self align to the top left or right corner.

With out float


With Float, the object inside auto align. Instead of position each block, we can now just put 10 block in and they auto align.

For performance increase, instead of creating and destroying dom object, all object we use are now created before hand and reuse. With grouping and css float, we have our biggest gain come when the row are filled with blocks. In the previous version the blocks have to be remove and all the blocks on top have to move down. The new approach we just move the row to the top (O1).