Monday, December 30, 2013

How to fix hadoop 2.2.0 with 64bit os ploblem

If you show a message 

"WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform."

you need to build hadoop source

1. download hadoop source

2. edit pom.xml
    https://issues.apache.org/jira/browse/HADOOP-10110

3. build hadoop
    mvn package -Pnative -DskipTests

4. copy libhadoop.so.1.0.0
    cp hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/libhadoop.so.1.0.0 /usr/local/hadoop/lib/native/.

Thursday, December 26, 2013

MySQL Replication Master Server Change without stopping

How to change master server without stop when  we have Server Old-Master, Slave, New-Master
I'm not an expert of database administration. If there are better way, share me please.

I. Database no-stop server change

1. New-Master Server Setting 
Setup New-Master as slave, also master

1) edit my.cnf
server-id=newid
log_bin=mysql-bin
#This option should be enabled for passing replicated data from old-master  to slave
log_save_updates=true 

2) Backup data from Old-Master and restore to New-Master by using mysqldump or other utilities.
mysqldump -h... -u... -p... --hex-blob --routines --triggers --all-databases | gzip > backup.sql

3) start New-Master as slave
mysql-shell) stop slave;
mysql-shell) CHANGE MASTER TO MASTER_HOST = 'Old-Master-Host', MASTER_USER = 'old-master-user', MASTER_PASSWORD = 'old-master-password', MASTER_LOG_FILE = 'old-master-logfile', MASTER_LOG_POS = old-master-pos;
mysql-shell) start slave;
4) check status
mysql-shell) show slave status\G;

5) stop slave

6) Backup data from Old-Master and restore to New-Master again (for changes)
choose alternative methods.

6-1) dump all data
mysqldump -h... -u... -p... --hex-blob --routines --triggers --all-databases --insert-ignore > backup.sql
6-2) dump partial data with mysqlbinlog
mysqlbinlog binlog.lastlog > backup-partial.sql
mysql -uroot -p -e "source backup-partial.sql
* if there are duplicate key problem
sed -e "s/^INSERT INTO/INSERT IGNORE INTO/" < backup-partial.sql > convert-insertinto.sql

7) start slave

8) Create new master replication users for slave and grant them.
create user 'replicator'@'%' identified by 'password';
grant replication slave on *.* to 'replicator'@'%';
9) flush logs


2. Slave Servers
change master to New-Master dynamically, also change my.cnf
mysql-shell) stop slave;
mysql-shell) CHANGE MASTER TO MASTER_HOST = 'New-Master-Host', MASTER_USER = 'New-master-user', MASTER_PASSWORD = 'New-master-password', MASTER_LOG_FILE = 'New-master-logfile', MASTER_LOG_POS = New-master-pos; 
mysql-shell) start slave;

3. Change your application setting. 

4. Stop Old-Master server


II. Database stop, easy to change

First, Setup New-Master server (I-1)
Second, shutdown old-master Server
Third, change new-master address to old-master server
Finally, Bring up New-Master Server

Monday, December 2, 2013

Gitlab Install, Slackware 13.37 or 14.1

Gitlab install, Slackware 13.37, 14.1
Install Redis
Download and compile from source
wget http://download.redis.io/redis-stable.tar.gz
tar xvzf redis-stable.tar.gz
cd redis-stable
make

copy files
cd src
sudo cp redis-server /usr/local/bin/
sudo cp redis-cli /usr/local/bin/
edit initial script
editor /etc/rc.d/rc.local

or make initial script

Git (Under 1.7.10)
Download and compile from source
cd /tmp
curl --progress https://git-core.googlecode.com/files/git-1.8.4.1.tar.gz | tar xz
cd git-1.8.4.1/
make prefix=/usr/local all

Install into /usr/local/bin
sudo make prefix=/usr/local install
When editing config/gitlab.yml (Step 6), change the git bin_path to /usr/local/bin/git

Ruby
mkdir /tmp/ruby && cd /tmp/ruby
curl --progress ftp://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p353.tar.gz | tar xz
cd ruby-2.0.0-p353
./configure --prefix=/usr/local/
make && make install

Check installed path
login agin
which ruby
ruby -v

gem install bundler --no-ri --no-rdoc

System User
useradd --system --shell /bin/bash --comment 'GitLab' --create-home --home /home/git/ git
check shadow file
cat /etc/shadow |grep git
git:*:::::::
if it shows git:!::::::: change to git:*:::::::

Git-Shell (v 1.7.9)
su git
git clone https://github.com/gitlabhq/gitlab-shell.git
cd gitlab-shell
git checkout v1.7.9
cp config.yml.example config.yml
edit config.yml
./bin/install

Database (MySQL/MariaDB)
CREATE USER 'gitlab'@'localhost' IDENTIFIED BY 'supersecret';
CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
GRANT SELECT, LOCK TABLES, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON `gitlabhq_production`.* TO 'gitlab'@'localhost'

Gitlab (v6.3)
su - git
Clone GitLab repository
git clone https://github.com/gitlabhq/gitlabhq.git gitlab
Go to gitlab directory
cd /home/git/gitlab
Checkout to stable release
git checkout 6-3-stable
Setup files
cp config/gitlab.yml.example config/gitlab.yml
chown -R git log/
chown -R git tmp/
chmod -R u+rwX  log/
chmod -R u+rwX  tmp/
mkdir tmp/pids/
mkdir tmp/sockets/
chmod -R u+rwX  tmp/pids/
chmod -R u+rwX  tmp/sockets/
mkdir public/uploads
chmod -R u+rwX  public/uploads

cp config/unicorn.rb.example config/unicorn.rb

git config --global user.name "GitLab"
git config --global user.email "gitlab@your_domain_name"
git config --global core.autocrlf input

cp database.yml.mysql database.yml
chmod o-rwx config/database.yml
gem install charlock_holmes --version '0.6.9.4'
bundle install --deployment --without development test postgres puma aws
bundle exec rake gitlab:setup RAILS_ENV=production

Tuesday, November 26, 2013

Gitlab, Redmine Install with Maria DB in subdirectory, Ubuntu 12.04LTS

Gitlab, Redmine Install with Maria DB in subdirectory, Ubuntu 12.04LTS


1. Maria DB


install packages
sudo apt-get install libmariadbclient-dev libmariadbclient18 mariadb-client-5.5 mariadb-client-core-5.5 mariadb-common mariadb-server mariadb-server-5.5 mariadb-server-core-5.5       

if there are dependency problem, you may specify the version of maria db
sudo apt-get install libmysqlclient18=5.5.30-mariadb1~precise mysql-common=5.5.30-mariadb1~precise
sudo apt-get install libmysqlclient18=5.5.33a+maria-1~precise mysql-common=5.5.33a+maria-1~precise

2. Gitlab - Subdirectory

first, follow gitlab.org instruction.

Modify 3 files in gitlab home directory
# 1) In your application.rb file: config.relative_url_root = "/gitlab"
# 2) In your gitlab.yml file: relative_url_root: /gitlab
# 3) In your unicorn.rb: ENV['RAILS_RELATIVE_URL_ROOT'] = "/gitlab"

modify gitlab-shell/config.yml
gitlab_url: "http://localhost/gitlab/"

nginx conf
upstream gitlab {
  server unix:/home/git/gitlab/tmp/sockets/gitlab.socket;
}


location /gitlab {

            proxy_read_timeout 300; # https://github.com/gitlabhq/gitlabhq/issues/694
            proxy_connect_timeout 300; # https://github.com/gitlabhq/gitlabhq/issues/694
            proxy_redirect     off;

            proxy_set_header   X-Forwarded-Proto $scheme;
            proxy_set_header   Host              $http_host;
            proxy_set_header   X-Real-IP         $remote_addr;

            proxy_pass http://gitlab;
}

3. Redmine - Subdirectory

install dependent packages
sudo install imagemagick libmagickwand-dev
follow Redmine install instruction

install unicorn
gem install unicorn

downloads config file
curl -o config/unicorn.rb https://raw.github.com/defunkt/unicorn/master/examples/unicorn.conf.rb

modify config

config/unicorn.rb
working_directory /home/redmine
listen "/home/redmine/tmp/sockets/.unicorn.sock", :backlog => 64
pid "/home/redmine/tmp/pids/unicorn.pid"

config/environment.rb
Redmine::Utils::relative_url_root = "/redmine"

nginx conf
upstream redmine {
  server unix:/home/redmine/tmp/sockets/redmine.socket;
}

location /redmine {
            proxy_read_timeout 300;
            proxy_connect_timeout 300;
            proxy_redirect     off;

            proxy_set_header   X-Forwarded-Proto $scheme;
            proxy_set_header   Host              $http_host;
            proxy_set_header   X-Real-IP         $remote_addr;
            proxy_pass http://redmine;
}