Monday, December 30, 2013

How to fix hadoop 2.2.0 with 64bit os ploblem

If you show a message 

"WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform."

you need to build hadoop source

1. download hadoop source

2. edit pom.xml
    https://issues.apache.org/jira/browse/HADOOP-10110

3. build hadoop
    mvn package -Pnative -DskipTests

4. copy libhadoop.so.1.0.0
    cp hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so.1.0.0 /usr/local/hadoop/lib/native/.

Thursday, December 26, 2013

MySQL Replication Master Server Change without stopping

How to change master server without stop when  we have Server Old-Master, Slave, New-Master
I'm not an expert of database administration. If there are better way, share me please.

I. Database no-stop server change

1. New-Master Server Setting 
Setup New-Master as slave, also master

1) edit my.cnf
server-id=newid
log_bin=mysql-bin
#This option should be enabled for passing replicated data from old-master  to slave
log_save_updates=true 

2) Backup data from Old-Master and restore to New-Master by using mysqldump or other utilities.
mysqldump -h... -u... -p... --hex-blob --routines --triggers --all-databases | gzip > backup.sql

3) start New-Master as slave
mysql-shell) stop slave;
mysql-shell) CHANGE MASTER TO MASTER_HOST = 'Old-Master-Host', MASTER_USER = 'old-master-user', MASTER_PASSWORD = 'old-master-password', MASTER_LOG_FILE = 'old-master-logfile', MASTER_LOG_POS = old-master-pos;
mysql-shell) start slave;
4) check status
mysql-shell) show slave status\G;

5) stop slave

6) Backup data from Old-Master and restore to New-Master again (for changes)
choose alternative methods.

6-1) dump all data
mysqldump -h... -u... -p... --hex-blob --routines --triggers --all-databases --insert-ignore > backup.sql
6-2) dump partial data with mysqlbinlog
mysqlbinlog binlog.lastlog > backup-partial.sql
mysql -uroot -p -e "source backup-partial.sql
* if there are duplicate key problem
sed -e "s/^INSERT INTO/INSERT IGNORE INTO/" < backup-partial.sql > convert-insertinto.sql

7) start slave

8) Create new master replication users for slave and grant them.
create user 'replicator'@'%' identified by 'password';
grant replication slave on *.* to 'replicator'@'%';
9) flush logs


2. Slave Servers
change master to New-Master dynamically, also change my.cnf
mysql-shell) stop slave;
mysql-shell) CHANGE MASTER TO MASTER_HOST = 'New-Master-Host', MASTER_USER = 'New-master-user', MASTER_PASSWORD = 'New-master-password', MASTER_LOG_FILE = 'New-master-logfile', MASTER_LOG_POS = New-master-pos; 
mysql-shell) start slave;

3. Change your application setting. 

4. Stop Old-Master server


II. Database stop, easy to change

First, Setup New-Master server (I-1)
Second, shutdown old-master Server
Third, change new-master address to old-master server
Finally, Bring up New-Master Server

Monday, December 2, 2013

Gitlab Install, Slackware 13.37 or 14.1

Gitlab install, Slackware 13.37, 14.1
Install Redis
Download and compile from source
wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
make

copy files
cd src
sudo cp redis-server /usr/local/bin/
sudo cp redis-cli /usr/local/bin/
edit initial script
editor /etc/rc.d/rc.local

or make initial script

Git (Under 1.7.10)
Download and compile from source
cd /tmp
curl --progress https://git-core.googlecode.com/files/git-1.8.4.1.tar.gz | tar xz
cd git-1.8.4.1/
make prefix=/usr/local all

Install into /usr/local/bin
sudo make prefix=/usr/local install
When editing config/gitlab.yml (Step 6), change the git bin_path to /usr/local/bin/git

Ruby
mkdir /tmp/ruby && cd /tmp/ruby
curl --progress ftp://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p353.tar.gz | tar xz
cd ruby-2.0.0-p353
./configure --prefix=/usr/local/
make && make install

Check installed path
login agin
which ruby
ruby -v

gem install bundler --no-ri --no-rdoc

System User
useradd --system --shell /bin/bash --comment 'GitLab' --create-home --home /home/git/ git
check shadow file
cat /etc/shadow |grep git
git:*:::::::
if it shows git:!::::::: change to git:*:::::::

Git-Shell (v 1.7.9)
su git
git clone https://github.com/gitlabhq/gitlab-shell.git
cd gitlab-shell
git checkout v1.7.9
cp config.yml.example config.yml
edit config.yml
./bin/install

Database (MySQL/MariaDB)
CREATE USER 'gitlab'@'localhost' IDENTIFIED BY 'supersecret';
CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
GRANT SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON `gitlabhq_production`.* TO 'gitlab'@'localhost'

Gitlab (v6.3)
su - git
Clone GitLab repository
git clone https://github.com/gitlabhq/gitlabhq.git gitlab
Go to gitlab directory
cd /home/git/gitlab
Checkout to stable release
git checkout 6-3-stable
Setup files
cp config/gitlab.yml.example config/gitlab.yml
chown -R git log/
chown -R git tmp/
chmod -R u+rwX  log/
chmod -R u+rwX  tmp/
mkdir tmp/pids/
mkdir tmp/sockets/
chmod -R u+rwX  tmp/pids/
chmod -R u+rwX  tmp/sockets/
mkdir public/uploads
chmod -R u+rwX  public/uploads

cp config/unicorn.rb.example config/unicorn.rb

git config --global user.name "GitLab"
git config --global user.email "gitlab@your_domain_name"
git config --global core.autocrlf input

cp database.yml.mysql database.yml
chmod o-rwx config/database.yml
gem install charlock_holmes --version '0.6.9.4'
bundle install --deployment --without development test postgres puma aws
bundle exec rake gitlab:setup RAILS_ENV=production