<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
<META http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<META NAME="GENERATOR" CONTENT="lfparser_2.13">
<META NAME="LFCATEGORY" CONTENT="System Administration">
<!-- this is used be a number of tools:
=LF=AUTHOR: Atif Ghaffar
=LF=CAT___: System Administration
=LF=TITLE_: Realtime data mirroring under Linux
=LF=NUMBER: 199
-->
<TITLE>lf199, System Administration: Realtime data mirroring under Linux</TITLE>
<!-- stylesheet added by lfparser: -->
<style type="text/css">
<!--
td.top {font-family: Arial,Geneva,Verdana,Helvetica,sans-serif; }
pre { font-familiy:monospace,Courier }
p.cl { color:#EE9500 }
a.nodec { text-decoration:none }
p.trans { font-size:8pt; text-align:right }
p.clbox { width:50%; alignment:center; background-color:#FFD700; border-style:none; border-width:medium; border-color:#FFD700; padding:0.5cm ; text-align:center }
p.foot { background-color:#AAAAAA; color:#FFFFFF; border-style:none; border-width:medium; border-color:#AAAAAA; padding:0.5cm ; margin-top:0.1cm; margin-right:1cm; margin-left:1cm; text-align:center }
-->
</style>
</HEAD>
<BODY bgcolor="#ffffff" text="#000000">
<!-- this is generated html code. NEVER use this file for your
translation work. Instead get the file with the same article number
and .meta.shtml in its name. Translate this meta file and then
use lfparser program to generate the final article -->
<!-- lfparser can be obtained from http://main.linuxfocus.org/~guido/dev/lfparser.html -->
<!-- 2pdaIgnoreStart -->
<!-- start navegation bar -->
<!-- top navegation bar -->
<TABLE cellspacing="0" cellpadding="0" border="0" align="center" width="90%">
<TR bgcolor="#2e2292">
<TD class="top"><TABLE cellspacing="0" cellpadding="0" border="0" width=
"100%">
<TR><TD width="144"><IMG src="../../common/images/logolftop.gif"
alt="[LinuxFocus-icon]" width="350" height="45" align="left"
border="0"></TD>
<TD class="top">
<TABLE width="100%">
<TR align="right">
<TD class="top"><A class="nodec" href="../index.shtml"><FONT color=
"#DDDDDD">Home</FONT></A> | <A class=
"nodec" href="../map.html"><FONT color=
"#DDDDDD">Map</FONT></A> | <A class=
"nodec" href="../indice.html"><FONT color=
"#DDDDDD">Index</FONT></A> | <A class="nodec" href="../Search/index.html"><FONT color=
"#DDDDDD">Search</FONT></A> </TD>
</TR>
<TR align="right">
<TD class="top">
<HR width="100%" noshade size="1">
</TD>
</TR>
</TABLE>
</TD>
</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<!-- end top navegation bar -->
<!-- blue bar -->
<TABLE cellspacing="0" cellpadding="0" border="0" align="center"
width="90%">
<TR bgcolor="#00ffff">
<TD><IMG src="../../common/images/transpix.gif" width="1" height=
"2" alt=""></TD>
</TR>
</TABLE>
<!-- end blue bar -->
<!-- bottom navegation bar -->
<TABLE cellspacing="0" cellpadding="0" border="0" align="center"
width="94%">
<TR bgcolor="#000000">
<TD>
<TABLE cellspacing="0" cellpadding="1" border="0" width=
"100%">
<TR align="center">
<TD class="top"><A class="nodec" href="../News/index.html"><FONT color=
"#FFFFFF">News</FONT></A> </TD>
<TD><FONT color="#FFFFFF">|</FONT> </TD>
<TD><A class="nodec" href="../Archives/index.html"><FONT color=
"#FFFFFF">Archives</FONT></A> </TD>
<TD><FONT color="#FFFFFF">|</FONT> </TD>
<TD><A class="nodec" href="../Links/index.html"><FONT color=
"#FFFFFF">Links</FONT></A> </TD>
<TD><FONT color="#FFFFFF">|</FONT> </TD>
<TD><A class="nodec" href="../aboutus.html"><FONT color=
"#FFFFFF">About LF</FONT></A> </TD>
<TD> </TD>
</TR>
</TABLE>
</TD>
</TR>
</TABLE>
<!-- end bottom navegation bar -->
<!-- stop navegation bar -->
<!-- SSI_INFO -->
<!-- tr_staticssi include virtual -->
<!-- tr_staticssi exec cmd -->
<!-- addedByLfdynahead ver 1.1 --><TABLE ALIGN="right" border=0><TR><TD ALIGN="right"><FONT SIZE="-1" FACE="Arial,Helvetica">This article is available in: <A href="../../English/March2001/article199.shtml">English</a> <A href="../../Castellano/March2001/article199.shtml">Castellano</a> <A href="../../Deutsch/March2001/article199.shtml">Deutsch</a> <A href="../../Francais/March2001/article199.shtml">Francais</a> <A href="../../Nederlands/March2001/article199.shtml">Nederlands</a> <A href="../../Turkce/March2001/article199.shtml">Turkce</a> </FONT></TD></TR></TABLE><br>
<!-- 2pdaIgnoreStop -->
<!-- SHORT BIO ABOUT THE AUTHOR -->
<TABLE ALIGN=LEFT BORDER=0 hspace=4 vspace=4 WIDTH="30%" >
<TR>
<TD>
<!-- 2pdaIgnoreStart -->
<!-- PALM DOC -->
<TABLE BORDER=0 hspace=4 vspace=4> <TR> <TD>
<font size=1> <img src="../../common/images/2doc.gif" width=34 align=left border=0 height=22 alt="convert to palm"><a href="http://cgi.linuxfocus.org/cgi-bin/2ztxt">Convert to GutenPalm</a><br>or <a href="http://cgi.linuxfocus.org/cgi-bin/2pda">to PalmDoc</a></font>
</TD> </TR> </TABLE>
<!-- END PALM DOC -->
<!-- 2pdaIgnoreStop -->
<br>
<IMG SRC="../../common/images/Atif-Ghaffar.jpg" ALT="[Photo of the Author]" WIDTH=124 HEIGHT=115>
<BR>by <A HREF="mailto:atif@developer.ch">Atif Ghaffar</A>
<BR><BR>
<I>About the author:</I><BR>
<p>Atif is a chameleon. He changes his roles, from System
Administrator, to programmer, to teacher, to project manager, to
whatever is required to get the job done.<br>
Atif thinks that he owes a lot to Linux, the open-source community
and projects for being his teacher.<br>
More about him can be found at his <a href=
"http://atifghaffar.com">homepage</a></p>
<BR><i>Content</i>:
<UL>
<LI><A HREF="#lfindex0">Why replicate instead of sharing?</A></LI>
<LI><A HREF="#lfindex1">Keeping it simple</A></LI>
<LI><A HREF="#lfindex2">Keeping it smart</A></LI>
<LI><A HREF="#lfindex3">What is FAM?</A></LI>
<LI><A HREF="#lfindex4">What is IMON?</A></LI>
<LI><A HREF="#lfindex5">Installing FAM and IMON</A></LI>
<LI><A HREF="#lfindex6">Installing SGI::FAM Perl module</A></LI>
<LI><A HREF="#lfindex7">Replicating with fam_mirror</A></LI>
<LI><A HREF="#lfindex8">Resources</A></LI>
<LI><A HREF="#lfindex9">Known Problems</A></LI>
<LI><A HREF="http://cgi.linuxfocus.org/cgi-bin/lftalkback?anum=199&lang=en">Talkback form for this article</A></LI>
</UL>
</TD></TR></TABLE>
<!-- HEAD OF THE ARTICLE -->
<br>
<H2>Realtime data mirroring under Linux</H2>
<IMG SRC="../../common/images/tuxmirror.gif" ALT="[Illustration]" HSPACE=10 WIDTH=149 HEIGHT=246>
<!-- ABSTRACT OF THE ARTICLE -->
<P><i>Abstract</i>:
<P>
In this article we will explore real-time data replication under Linux
without using expensive SANs (Storage Area Network, e.g GFS) or other Network Block devices.<br>
We will use <a href="http://oss.sgi.com/projects/fam">FAM and IMON</a> for our replication system.<br>
FAM (File Alteration Monitor) and IMON (Inode Monitor) have been developed by SGI originally for IRIX. <br><br>
The folks at SGI have been very cool to port it to Linux and make it open source.
<br><br>
When cost is not an issue, I would go for a <a
href="http://www.globalfilesystem.org">GFS (Global File System)</a> and SAN based solution, but when cost is a factor, and data sharing is necessary I am not left with a lot of choices.<br>
I have a few choices to pick from. In this article we will discuss them and see what are the advantages/disadvantages.<br><br></P>
<HR size="2" noshade align="right"><BR>
<!-- BODY OF THE ARTICLE -->
<A NAME="lfindex0"> </A>
<H2>Why replicate instead of sharing?</H2>
<p>
Aren't fileservers supposed to make data available to clients?<br>
Yes they are.<br><br>
If we use a file server that shares files over NFS or SMB etc, then we have a <b>bottle neck</b> and a <b>Single Point Of Failure</b>.<br><br>
If we share data over GFS with a shared storage (SAN or MultiChannel SCSI), we have the Storage Box as the <b>Single Point Of Failure</b> and its very expensive to set up a system with that configuration.<br><br>
We can use NBD (Network Block Devices) to set up a network mirror, but I am not very comfortable with that. NBDs have their limitations, are difficult to setup and manage and are just too much bother, when all you need is to replicate a few webserver data across a few webservers.<br><br>
</p>
<A NAME="lfindex1"> </A>
<H2>Keeping it simple</H2>
<P>
Ok, lets try replicating.
<br>
<b>Here is one scenario</b><br>
You have 2 webservers, one main server and the other as the backup.<br>
You make all changes on the master machine and rsync the changes to the second machine.<br>
Simple?<br>
But how to automate it? Your users will FTP to master machine multiple times a day. What will happen if there is a failure on the master server and the back server takes over?<br>
Easy. I have the answer to that. They will not see the changes they made,
and will be pretty angry. :)<br>
Well you can run "rsync -av --delete source destination" from CRON every 5 seconds, but then your machine will not be really useful for anything else, would it?<br>
<br><br>
<b>Here is another scenario</b><br>
You have one FTP server to upload the data and <br>
six webserver that respond in a round robin fashion.<br>
So the data on each machine should be the same. You can get away with NFS for sometime if you are lucky, but you wont for long.<br><br>
Now, what should be done?<br>
I think the answer is "copy the data to the webservers only if there is a
change to the files", and if there is no change to the data, don't do anything.<br><br>
This is exactly what we will do using "fam".
<A NAME="lfindex2"> </A>
<H2>Keeping it smart</H2>
<P>
So how do we know there is a change on the files?<br>
Here is one answer that I would expect from a M$ Windows developer.<br>
<b>We can search the directory we are monitoring every few seconds and compare its timestamps and size with the version we had in cache.</b><br>
Yeah right<br>
<br>
<b>Polling</b>: looking for files timestamps/size and comparing with the older version is expensive.<br>
Imagine if your box is running "ls -lR /somedirectory" every 5 seconds on your webserver :)<br>
<br>
The elegant way would be for the file to tell us when it has changed, so we can take an action upon it.<br>
This is exactly what "IMON" will do for us.
</p>
<A NAME="lfindex3"> </A>
<H2>What is FAM?</H2>
<P>
<i>source: <a href="http://oss.sgi.com/projects/fam/faq.html">http://oss.sgi.com/projects/fam/faq.html</a></i><br>
fam, the File Alteration Monitor, provides an API which applications can use
to be notified when specific files or directories are changed.
<br>
FAM comes in two parts: <B>fam,</B> the daemon which listens for requests
and delivers notification, and <B>libfam,</B> a library which client
applications can use to communicate with FAM.
<br>
If the monitored files are mounted from a remote host, the local fam will
attempt to contact fam on the remote host, and will pass the requests on to
the remote fam.
<br>fam can also notify its clients when a file starts and stops execution.
(The IRIX Interactive Desktop uses this to change a program's icon while it's
running, for example.)
<br>fam was originally written for IRIX in 1989 by Bruce Karsh, and was
rewritten in 1995 by Bob Miller. This open-source release of fam builds and
runs on both Linux and IRIX, and is the same fam that will be included with
IRIX 6.5.8.
<A NAME="lfindex4"> </A>
<H2>What is IMON?</H2>
<P><i>source: <a href="http://oss.sgi.com/projects/fam/faq.html">http://oss.sgi.com/projects/fam/faq.html</a></i><br>
imon, the Inode Monitor, is the part of the kernel that tells fam when
files have changed. When applications tell fam they're interested in files or
directories, fam passes that interest on to imon. When file operations are
performed on files monitored by imon, the kernel tells imon; imon tells fam,
and fam notifies the applications which are interested in the files.
<br>imon was originally written for the IRIX kernel in 1989 by Wiltse Carpenter;
the Linux port was done by Roger Chickering. The Linux implementation in the
imon kernel patch is similar to the IRIX implementation in most ways, but it
hooks into the kernel filesystem code differently.
</p>
<A NAME="lfindex5"> </A>
<H2>Installing FAM and IMON</H2>
<p>
FAM and IMON are both available from SGI's website. See Resources below.<br>
IMON is a patch that you can apply to your kernel. This will add possibility for your kernel to monitor Inodes.<br>
To patch the kernel, cd to your kernel sources directory.<br>
and apply the patch<br>
cd /usr/src/linux<br>
patch -pi < patchfile
<br><br>
then run make config or make menuconfig and select
when you are asked for <br>
<b>Inode Monitor (imon) support (EXPERIMENTAL)</b><br>
in the FileSystems section
<br>
compile the kernel as usual and reboot (sorry).
<br>
Compiling FAM itself is pretty simple.
<br>
cd to the fam sources directory and run
<br>
./configure && make all install
<br>
Voila its installed.
<br>
<br>
Next we will install a Perl module called SGI::FAM, so we can write our event handler in perl.
</p>
<A NAME="lfindex6"> </A>
<H2>Installing SGI::FAM Perl module</H2>
<p>
You didn't really think, I would ask you to code C/C++. Did you?<br>
Well I don't know about you, but I am too lazy and impatient, so I will write my replication handler in Perl<br>
<br>
Download and install SGI::FAM by <a href="mailto::jglick@sig.bsh.com">Jesse N. Glick</a><br>
To install these modules, simply run the CPAN module
<br>
perl -MCPAN -e shell
<br>install SGI::FAM<br>
this should install SGI::FAM and all prerequisite modules.
<A NAME="lfindex7"> </A>
<H2>Replicating with fam_mirror</H2>
<p>
<b>fam_mirror</b> is a script that I wrote to automate the replication.<br>
you can <a href="../../common/src/article199/fam_mirror.html">view</a>
or <a href="../../common/src/article199/fam_mirror">download</a> it here.<br>
You can edit it and <br>
change $replicaHosts to meet your hosts,
<br> change $rsh with whatever command you can run from one machine to another <br>
and the same with $rsync.
<br><br>
So back to <b>scenario 1</b><br>
2 machines running as webservers (web1, web2). 1 of them as master (web1) and the other as slave (web2).
<br>
Primary FTP server is (web1). <br>
web2 does not run FTP service at all. (otherwise users may try to write to files even when the system is in backup mode)<br><br>
The web document root on both machines is <b>/var/www</b><br>
setup rsh or ssh on both machines. web2 should allow web1 to run remote commands without a password. I usually add my ssh_key to the authorized_keys of replica Hosts.
<br>
rsync all data from web1 to web2<br>
rsync -avz /var/www/ web2:/var/www/
<br>
Edit fam_mirror and change @replicaHosts to <br>
<b>@replicaHosts=qw(web2)</b><br>
run <b>fam_mirror</b> on web1.<br>
fam_mirror /var/www &
<br>
and then make changes to files on web1. All changes will also be written to web2.
<br><br>
Now to <b>scenario 2</b> (A farm of webservers)<br>
Hosts "linuxweb1", "linuxweb2", "linuxweb3" and "linuxweb4" runs as webservers<br>
Host "linuxftp1" runs as ftp server (main fileserver)<br>
web hosts do not allow FTP to users.<br>
install fam, imon, SGI::FAM and fam_mirror on host "linuxftp1"<br>
Setup rsh or ssh between the machines.<br>
hosts linuxweb[1-4] should allow linuxftp1 to run remote commands without prompting for a password.<br>
Edit fam_mirror and set @replicaHosts to <br>
<b>@replicaHosts=qw(linuxweb1 linuxweb2 linuxweb3 linuxweb4);</b><br>
Change $rsh and $rsync if neccessary.
Assuming that web document root is /var/www on all machines.<br>
run on linuxftp1<br>
INIT_MIRROR=1 fam_mirror /var/www &
<br><br>Now all changes on linuxftp1 should be visible on linuxweb[1-4]<br>
</p>
</p>
<A NAME="lfindex8"> </A>
<H2>Resources</H2>
<p><ul>
<li><a href="http://oss.sgi.com/projects/fam">SGI's fam site</a>
<li><a href="http://search.cpan.org/search?dist=SGI-FAM">SGI::FAM Perl Module</a>
<li><a href="http://www.nas.nasa.gov/Groups/WWW/subpages/topology.html">Realtime Data Mirroring (WEBD and SLIM) for IRIX</a>
<li><a href="../../common/src/article199/fam_mirror">fam_mirror</a>
</ul>
</p>
<A NAME="lfindex9"> </A>
<H2>Known Problems</H2>
I found that the solution that I have presented here has a little
problem: Its actually not working with large directories. (directory with
4-5 thousand
sub directories). The kernel is complaining about kmalloc etc.
<br>
I am trying to get this sorted out. Once I have this sorted out, then I
will add
the information in the article.
<br>
Let me know if you are already aware of a solution to this problem.
<!-- vim: set sw=2 ts=2 et tw=74: -->
<!-- 2pdaIgnoreStart -->
<A NAME="talkback"> </a>
<h2>Talkback form for this article</h2>
Every article has its own talkback page. On this page you can submit a comment or look at comments from other readers:
<center>
<table border="0" CELLSPACING="2" CELLPADDING="1">
<tr BGCOLOR="#C2C2C2"><td align=center>
<table border="3" CELLSPACING="2" CELLPADDING="1">
<tr BGCOLOR="#C2C2C2"><td align=center>
<A href="http://cgi.linuxfocus.org/cgi-bin/lftalkback?anum=199&lang=en"><b> talkback page </b></a>
</td></tr></table>
</td></tr></table>
</center>
<HR size="2" noshade>
<!-- ARTICLE FOOT -->
<CENTER><TABLE WIDTH="95%">
<TR><TD ALIGN=CENTER BGCOLOR="#9999AA">
<A HREF="../../common/lfteam.html">Webpages maintained by the LinuxFocus Editor team</A>
<BR><FONT COLOR="#FFFFFF">© Atif Ghaffar, <a href="../../common/copy.html">FDL</a> <BR><a href="http://www.linuxfocus.org">LinuxFocus.org</a></FONT>
<BR><a href="http://cgi.linuxfocus.org/cgi-bin/lfcomment?lang=en&article=article199.shtml" target="_TOP">Click here to report a fault or send a comment to LinuxFocus</A><BR></TD>
<!-- OLD FORMAT, NO TRANSLATION INFO -->
</TR></TABLE></CENTER>
<p><font size=1>2001-04-27, generated by lfparser version 2.13</font></p>
<!-- 2pdaIgnoreStop -->
</BODY>
</HTML>