Linux: Follow and Download using wget

Your Linux box is incomplete if you do not have wget installed. It is a simple CLI based application used to download files from network.  It support HTTP, HTTPS and FTP protocols as well as retrieval through HTTP proxies.

 

 

Download latest version of PHPmyAdmin from SourceForge.

 

You can just simply use the latest link of SourceForge software and wget will automatically follow it. You also need to specify the ‘content-disposition’ flag. This option is useful for some file-downloading CGI programs that use “Content-Disposition” header to describe what the name of a downloaded file should be:

$ wget --content-disposition http://sourceforge.net/projects/phpmyadmin/files/latest/download

Download TokuMX  with cookies and redirection from Tokutek download site.

 

To download from this page, I was required to have cookies from download.php=tokumx-1.0-3-linux-x86_64.tgz page and use the same cookies to download from download.php?df=1. You can navigate to this page if you want to try that with a web browser, http://www.tokutek.com/download.php?download_file=tokumx-1.0.3-linux-x86_64.tgz.

$ wget --keep-session-cookies \
--save-cookies cookies.txt \
http://www.tokutek.com/download.php?download_file=tokumx-1.0.3-linux-x86_64.tgz && \
wget --load-cookies cookies.txt \
--content-disposition http://www.tokutek.com/download.php?df=1 && \
rm -f download.php* cookies.txt

 

Download Certain Files from Web Directory

 

You can download certain files (*.gz) under same directory using wget with -r flag. You need to specify –no-directories and –no-host-directories to output the files in the current directory while –level is the depth of the directory it will go:

$ wget -r --accept="*.gz" \
--no-directories \
--no-host-directories \
--level 1 \
ftp://ftp.fu-berlin.de/pub/misc/movies/database/

 

I will keep updating this page to share my collection of wget commands.

 

Linux: Run Command in Many Servers Simultaneously

Server administrators usually have many servers to manage. There will be some time when we need to run the same command in every server we have. In my case, I need to update MySQL version on all 6 database servers we have to the latest version via yum. It just need to run simple command but I need to repeat the process 6 times for each server. Furthermore, we will keep doing this in the future over and over again.

To achieve this, I am going to use Webmin cluster. So all servers must have been installed with Webmin and we need to integrate all of them and joining the Webmin cluster. In this tutorial, I will only use 2 servers as example of setup.

Variables as below:

OS: CentOS 6.2 64bit
Webmin (master)/MySQL server #1: 192.168.0.160
Webmin (node1)/MySQL server #2: 192.168.0.163
Webmin username: root
Webmin password: Gn&Pe42#e

1. Download and install Webmin for all servers. You need to repeat step number 1 to 3 on all servers:

$ cd /usr/local/src
$ wget http://cdnetworks-kr-2.dl.sourceforge.net/project/webadmin/webmin/1.580/webmin-1.580-1.noarch.rpm
$ rpm -Uhv  webmin-1.580-1.noarch.rpm

2. Configure firewall to open port 10000 for Webmin communication. Add following line into /etc/sysconfig/iptables before any REJECT line (-j REJECT):

-A INPUT -p tcp -m tcp --dport 10000 -j ACCEPT

Save the file and restart firewall:

$ service iptables restart

3. Start Webmin:

$ service webmin start

4. In master node, access the Webmin via http://192.168.0.160:10000 and login as root. Go to Webmin > Webmin Server Index > and click “Register a new server“. Enter the server information as screenshot below:

And click “Save“.

5. The server should now register in the Webmin but not joining the cluster yet. Go to Webmin > Cluster > Cluster Webmin Servers > and click Add Server for “this server” and “192.168.0.163“.

6. If the joining is completed, you should see all servers in the managed list like below:

7. Now we can start to run any command on all servers. What I need to do now is go to Webmin > Cluster > Cluster Shell Commands > and enter the command that I need to run and which server you want it to run. I will choose “<All hosts>” and click “Run Command Now”  as screenshot below:

The output will then appeared from both of the servers. In my real case, I need to repeat step 4,5,6 to add another 4 servers and finally just execute the command as step 7. This simple setup is just a great way for your server administration in the future. Cheers!

Linux: Remove Files/Folder More Than Certain Time

One of our server encounter backup problem because there are too many files need to be backup, near 200 million total of files. What I need to do is to remove some files in some folders, and let it run automatically everyday to remove the unwanted files. In our web server, we have one temporary folder that use to have lots of temporary files. The folder is located under /home/mywebsite/temp_upload/ .

I started by checking the inodes (the number of files) in this folder:

$ du -sk /home/mywebsite/temp_upload/
10543660 /home/mywebsite/temp_upload/

As you can see, I have 10 million files inside this directory. Our developer has forgot to remove the unused files so I need to create a cron job to remove files which older than 3 months (90 days) in this directory. To remove, the command should be as below:

$ find /home/mywebsite/temp_upload/ -type f -mtime +90 | xargs rm -Rf

It takes some time to complete and once done, the inodes has dropped to 2077:

du -sk /home/mywebsite/temp_upload/
2077 /home/mywebsite/temp_upload

To automate this, just add the command into crontab and schedule to run on weekly basis (at 6 AM every Sunday):

$ crontab -e

Add following line:

0 6 * * 0 /bin/find /home/mywebsite/temp_upload/ -type f -mtime +90 | xargs rm -Rf

Restart crond to apply the cron changes:

$ service crond restart

Warning: Make sure you run the command during low peak hours. This process might overloading your server, as what happened to me due to wrong time zone 🙂

Now you should automate the files removal and you can focus on other things!

Linux: Install and Configure PostgreSQL with pgAdmin

PostgreSQL is a powerful, open source object-relational database system. It has more than 15 years of active development and a proven architecture that has earned it a strong reputation for reliability, data integrity, and correctness. People will usually comparing PostgreSQL with MySQL, and there are several key factors which let us choose PostgreSQL over MySQL:

  • Single storage engine – no hassle to choose which storage engine like MySQL
  • PostgreSQL aims for SQL standards compliance (the current standard is ANSI-SQL:2008)
  • High level of data integrity demanded by a serious transactional database application, the current generation PostgreSQL performs better than MySQL
  • Open source and run on GPL license – no licensing cost

I will be using standard CentOS 6 with and yum installer. Variable as below:

OS: Centos 6.2 64bit
Server IP: 192.168.0.1
Client IP: 192.168.0.100
Root user: postgres
Password: Bf44Ks#1

1. Install PostgreSQL using yum:

$ yum install -y postgresql*

2. Initialize the server database:

$ service postgresql initdb

3. By default, PostgreSQL will be installed under /var/lib/pgsql/data. Open the configuration file at /var/lib/pgsql/data/postgresql.conf using text editor and enable following line:

listen_addresses = '*'

4. Open client authentication configuration file at /var/lib/pgsql/data/pg_hba.conf to configure how user can access PostgreSQL via network or locally. User postgres is the super-user and I need to allow this user to be access from my Windows 7 laptop using pgAdminIII client. I need to add following line as below:

host    all    postgres    192.168.0.0/24       md5

Explanation: host = access via network, all = can access all database, postgres = user that I want to use to access, 192.168.0.0/24 =  network address which the user used to access this server, md5 = authentication method

5. Start and enable the PostgreSQL service:

$ service postgresql start
$ chkconfig postgresql on

6. Now we need to reset the user postgres password so we can use md5 authentication when connecting through client later. We need to change to user postgres environment by ‘su’ command and then access the PostgreSQL database:

$ su - postgres
-bash-4.1$ psql
psql (8.4.9)
Type "help" for help.
 
postgres=#

Then run following command:

postgres=# ALTER USER postgres WITH ENCRYPTED PASSWORD 'Bf44Ks#1';
postgres=# \q

After quit from psql, reload the server:

$ psql_ctl reload

7. Add following line into /etc/sysconfig/iptables to allow port 5432 for PostgreSQL communication using text editor. Make sure you put the line before any REJECT (-j REJECT) rules:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 5432 -j ACCEPT

Dont forget to restart IPtables:

$ service iptables restart

8. Now lets download and install pgAdminIII, a client which run on Windows for PostgreSQL database server at http://www.postgresql.org/ftp/pgadmin3/release/v1.8.4/win32/. Once downloaded, unzip the compress file and run the installer. Follow the wizard by accepting default value and complete the installation.

9. Lets connect to the server and run the pgAdminIII client. It usually located under Windows > All Programs > pgAdmin III 1.8 > pgAdmin III. Enter required information as screenshot below:

Done! You may now manage your PostgreSQL server remotely!

Upgrade DELL Open Manage Server Administrator (OMSA)

Some of our servers are running on Dell which include Open Manage Server Administrator (OMSA) to manage the physical server  remotely. OMSA can be access via port 1311 in HTTPS by using web browser. Most of the time, I only use OMSA to manage and monitor our physical disk which run on RAID.

The version that I currently use is 6.3.0, which is the old one (because I have not update anything on OMSA since first usage). Following steps show on how to update Dell OMSA with the easiest way. Before upgrade, lets take a look on how OMSA’s current look:

Variable as below:

OS: Red Hat Enterprise Linux Server release 5.6 (Tikanga)
Server IP: 192.168.100.30
OMSA URL: https://192.168.100.30:1311

1. We will use yum to update current OMSA installation. Lets add the Dell Open Manage repository into it:

$ cd /usr/local/src
$ wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash

2. The safest way is to stop and remove the old version:

/opt/dell/srvadmin/sbin/srvadmin-services.sh stop
$ yum remove -y srvadmin*

3. Lets install new OMSA:

$ yum install -y srvadmin-all

4. Log out from SSH and relogin back to reset OMSA path.

5. Start OMSA service to load new update:

$ /opt/dell/srvadmin/sbin/srvadmin-services.sh start

And check whether it runs on the correct port:

$ netstat -tulpn | grep 1311
tcp     0       0 0.0.0.0:1311       0.0.0.0:*       LISTEN       29960/dsm_om_connsv

Done! Now lets see how our OMSA new look:

It seems Dell has hired great web developer for this new OMSA interface (version 6.5.0). As for me, it is not really matter as long as all required functionalities are still there. Cheers!

Solaris: Setting Up Package Manager and Installer

Oracle Solaris 11 is one of the  most popular UNIX-based operating system which widely download to be used by enterprise user, after FreeBSD. If you have FreeBSD basic knowledge, you should able to play around with Solaris as well.

If FreeBSD we have ports, the package manager and application installer, in Solaris, we can use pkgutil. It is rather simple to use and similar to apt-get and yum on how to use it. We will use OpenCSW, which aims to produce an easy to use open source software distribution installable on top of Solaris and Solaris-based systems, developed by a transparent and democratic community.

My Oracle Solaris version:

$ cat /etc/release
Oracle Solaris 11 11/11 X86
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Assembled 18 October 2011

1. Run following command as root to install the pkgutil application:

$ pkgadd -d http://get.opencsw.org/now

2. We need to set some environment for CSW. Open ~/.profile using text editor:

$ nano ~/.profile

And append :/opt/csw/bin at the end of PATH line and add another line for XFILESEARCHPATH so it will be shown like below:

export PATH=/usr/bin:/usr/sbin:/opt/csw/bin
export XFILESEARCHPATH=/opt/csw/lib/X11/%T/%N%C:/usr/openwin/lib/X11/%T/%N%C

To check whether it is correctly inserted, use following command:

$ echo $PATH
/usr/bin:/usr/sbin:/opt/csw/bin
$ echo $XFILESEARCHPATH
/opt/csw/lib/X11/%T/%N%C:/usr/openwin/lib/X11/%T/%N%C

3. Now we should able to use pkgutil. Lets update our catalog:

$ pkgutil -U

Note: If you unable to execute the command with error similar to ‘command not found’, you need to relogin your session or type su – command

4. It will download the list from this mirror: http://mirror.opencsw.org/opencsw/unstable/i386/5.11/ . To find available application in the catalog, we can use following command:

$ pkgutil -a mysql
libmysqlclient15   CSWlibmysqlclient15 5.0.92,REV=2011.10.05     1.4 MB
libmysqlclient_r15 CSWlibmysqlclient-r15 5.0.92,REV=2011.10.05   1.4 MB
mysql4             CSWmysql4 4.1.22,REV=2008.01.20              13.0 MB
mysql4bench        CSWmysql4bench 4.1.22,REV=2006.11.28        487.4 KB
mysql4client       CSWmysql4client 4.1.22,REV=2007.12.01         2.4 MB
mysql4devel        CSWmysql4devel 4.1.22,REV=2008.01.20          2.1 MB
mysql4rt           CSWmysql4rt 4.1.22,REV=2008.01.20             3.6 MB
mysql4test         CSWmysql4test 4.1.22,REV=2006.11.28           1.3 MB
mysql5             CSWmysql5 5.0.92,REV=2011.10.05              11.0 MB
mysql5client       CSWmysql5client 5.0.92,REV=2011.10.05         3.0 MB
mysql5devel_stub   CSWmysql5devel 5.0.92,REV=2011.10.05          7.9 KB
mysql5rt_stub      CSWmysql5rt 5.0.92,REV=2011.10.05             7.9 KB
mysql_dev          CSWmysql-dev 5.0.92,REV=2011.10.05          115.8 KB
php4_mysql         CSWphp4mysql 4.4.9,REV=2009.05.28            17.3 KB
php5_mysql         CSWphp5-mysql 5.3.8,REV=2011.09.04           22.6 KB
php5_mysqli        CSWphp5-mysqli 5.3.8,REV=2011.09.04          49.8 KB
php5_pdomysql      CSWphp5-pdomysql 5.3.8,REV=2011.09.04        17.3 KB
php5mysql_stub     CSWphp5mysql 5.3.8,REV=2011.09.04             2.5 KB
php5mysqli_stub    CSWphp5mysqli 5.3.8,REV=2011.09.04            2.5 KB
php5pdomysql_stub  CSWphp5pdomysql 5.3.8,REV=2011.09.04          2.5 KB
pm_dbdmysql        CSWpmdbdmysql 4.013,REV=2010.02.11          169.8 KB
py_mysql           CSWpy-mysql 1.2.3,REV=2011.03.13             41.0 KB
pymysql            CSWpymysql 1.2.3,REV=2011.03.13               1.8 KB
rb18_mysql_2_8_1   CSWrb18-mysql-2-8-1 2.8.1,REV=2011.08.14    217.0 KB

5. Now lets install application using pkgutil:

$ pkgutil -i -y mysql5

To update application:

$ pkgutil -u mysql5

To remove application:

$ pkgutil -r mysql5

To update all applications:

$ pkgutil -u

From here you may start to manage your applications and let your Solaris working!