Archive 1.x:Local Drupal Sandbox

It is useful to have sandboxes for local development, for which multiples may be active at any given time.

It is useful to use Drupal's multi-site capabilities since that makes it possible to host multiple sites under a single install of the LAMP stack.

The combination of these suggest it is worthwhile to work out general procedures for local and remote multi-site Drupal installs, and the procedures for easily and effectively moving the data between the installations.

Much trial-and-error was required in developing these steps, and researching (mostly poor) online documentation. Only the recipes for the final adopted steps are reproduced below, along with Tips as encountered.

Summary of Process

 * 1) Assuming the standard starting place is a single intended production site on a remote server, start there.  How basic Drupal is first set up is not covered further here (see the standard Drupal install guide with the reference to [http://drupal.org/node/327636 remote servers as needed)
 * 2) Create a local sandbox instance; the case below is based on Windows
 * 3) Configure the local instance for multiple sites
 * 4) Based on the local multi-site configuration, re-configure the remote instance for directories, themes and modules (of course, if this information is known in advance, it is better to begin initial set-up with this configuration)
 * 5) Clean up remote databases and export
 * 6) Make global search-and-replace changes to the text DB exports to deal with remote and local configuration differences (hopefully kept as minor as possible)
 * 7) Import remote databases into the local instance
 * 8) Make local changes to the files, etc
 * 9) Make changes, updates, content additions to the local sandbox instance
 * 10) Repeat Steps # 5 to # 9 and reverse the process to re-import back to the remote instance
 * 11) Roundtrip as necessary
 * 12) As new lessons and tips are learned, improve and update this guide.

Tips
'''
 * Keep written records of all logins and passwords across the multiple systems and applications
 * '''Keep notes about new issues and fixes encountered and add them to this guide

Setting Up a Local Sandbox (Windows)
There are a number of LAMP-like packages for Windows ("WAMP") with a comparison on Wikipedia. Because of some initial reviews and the amount of existing Drupal documentation, we chose the XAMPP package.

Here are the basic steps followed (culled from too many various Drupal notes ! ):
 * 1) Install XAMPP per instructions
 * 2) Install Drupal per instructions (see the Windows local install guide)
 * 3) Make sure the directory structure is set-up properly (see below)
 * 4) Then do multi-site installs (see below)

Tips

 * The local Apache needs access to port 80; there can be conflicts with Skype for which there are ways to resolve and there is the netstat utility for checking port assignments from the command prompt ('netstat -ona')
 * Put all of your individual Drupal site configurations under this subdirectory: [XAMPP root]\htdocs\drupal\sites
 * Install Apache and MySQL as services, but ignore Mercury and FileZilla
 * Besides port conflicts, you may see issues with starting the Apache service because of logging issues or conflicts (due to some of the later multi-site stuff). If you attempt to start Apache, and do not see it running, and the port issues are fixed, then you may need to look at the Apache logs at:  [XAMPP root]\apache\logs\error.log
 * XAMPP may be applicable to other OSs

Other Useful References

 * Setting up a local test server using XAMPP: http://drupal.org/node/161975

Setting Up a New (Multi-) Site
For both the local and remote servers, multi-sites is handled through two mechanisms:
 * 1) Drupal's own multi-site capabilities, which are based on a "cascade-access" model of querying directories, and that begin under the Drupal /sites subdirectory
 * 2) The use of "virtual hosts" or vhosts within Apache

General Steps
Irrespective of local or remote, Windows or Linux, here are the general steps for setting up a multi-site Drupal install. Note we address specific Linux and Windows considerations in the next sections.

For each new site in a multi-site: new-site default-settings.php settings.php files icons images temp logs modules themes tmp image
 * 1) Create a new MySQL database (via root or admin access using something like PHPMyAdmin). At this point, merely create the new DB from intro page:
 * 2) * e.g., new-site
 * 3) On the local or remote platform, create a new folder underneath the Drupal root directory folder ( [Drupal root]/drupal/sites/new-site) folder by copying the default folder and then renaming it with the name of the new site:
 * 4) * e.g., new-site
 * 5) Create the following sub-folders (directories structure) underneath this new-site directory as follows:

 Note: you may want to copy over an existing site structure. With a small reference template, this works well. However, try to avoid copying the actual logs; rather, simply create the empty files.  Note this specific directory structure is related to SD's specific purposes and modules. However, the higher levels of the structure are common to any multi-site installation. Actually, the best method is to create a template for the above as a shell with structure and permissions as indicated. Then, the template only need be copied and then re-named. Here is a second structure, now embedded in the overall context: sites all modules themes default blah-blah new-site blah-blah new-and-different-site default-settings.php settings.php files icons images temp logs modules themes tmp image  Make sure that all new directories and files are given "write" permissions (remove Read-only for Windows properties or chmod 777 for Linux systems) Edit the database and base URL in the settings.php file in the folder you have just renamed. $db_url = 'mysql://username:password@localhost/new-site'; $base_url = 'http://new-site'; // NO trailing slash!  Create a new Virtual Host in the appropriate Apache file (see different local or remote, Windows or Linux definitions below) Restart the Apache server to load the virtual hosts. With XAMPP, you can do this with the Stop and then Start buttons in the service console Point your web browser at new folder and install Drupal</li></ol> http://new-site/install.php  Create your first user and Administer the File System paths to match step three above</li></ol> sites/new-sites/files sites/new-sites/etc.
 * In this basic installation, core install modules are listed for the individual site
 * Other modules under /drupal/modules are NOT recognized

Windows Considerations
A number of steps are specific to a Windows installation using XAMPP.

NameVirtualHost *:80 <VirtualHost *:80> DocumentRoot [XAMPP root]/htdocs/drupal ServerName new-site CustomLog "[XAMPP root]/htdocs/drupal/sites/new-site/logs/new-site.access.log" combined ErrorLog "[XAMPP root]/htdocs/drupal/sites/new-site/logs/new-site.error.log" <Directory "[XAMPP root]/htdocs/drupal/sites/new-site"> Options Indexes FollowSymLinks AllowOverride All Order allow,deny Allow from all </Directory> </VirtualHost>  Create a new host in the Windows hosts file with same name as new folder. In Windows XP, this folder is found at: C:\WINDOWS\SYSTEM32\DRIVERS\ETC\HOSTS. For the new site, add this line:</li></ol> 127.0.0.1 new-site
 * 1) Create a new site (see #3 above) under the htdocs (similar to www or public-html in Linux) directory: [XAMPP root]\htdocs\drupal\sites
 * 2) Add a new virtual host to the standard vhosts directory used by XAMPP.  That directory is found at:  [XAMPP root]\apache\conf\extra\httpd-vhosts.conf
 * 3) Here is the sample vhosts file to add to the end of that listing:
 * Note there is no trailing slash on the DocumentRoot
 * Also note that DocumentRoot will be the SAME for every new site you create (that is because Drupal manages the site re-directions internally)
 * Note we are unquoting the #NameVirtualHost (if it was previously quoted) and are assigning it to listen to port 80 (*:80)
 * Notice the use of forward slashes, even though standard Windows file paths use backward slashes
 * Note that [XAMPP root] should include the full path (with drive)

Linux Considerations
There are slightly different considerations for Linux virtual hosts. /usr/share/drupal/sites/new-site.com  Create a file directory structure as in #3 above and also make all files and directories readable (chmod 777); you will need to change these after site configuration</li> Create a new virtual hosts (vhosts) listing for the new site. In our instance, we put this file into the following subdirectory, /etc/apache2/sites-enabled/new-sites.conf, and notice the name new-sites.conf. .conf files are Apache configuration files. We give each new site its own file, named the same as the site but without a tld extension, just to make it easier to inspect and modify. Here is what we put into this file:</li></ol> NameVirtualHost *:80 <VirtualHost *.80> DocumentRoot /usr/share/drupal ServerName new-site.com ServerAlias www.new-site.com ServerAdmin webmaster@new-site.com CustomLog /usr/share/drupal/sites/new-site.com/logs/new-site.access.log combined ErrorLog /usr/share/drupal/sites/new-site.com/logs/new-site.error.log <Directory /usr/share/drupal/sites/new-site.com> Options Indexes FollowSymLinks MultiViews AllowOverride All Order allow,deny allow from all </Directory> </VirtualHost>  Stop and then re-start Apache for this change to take effect. On our system we go to the usr/bin directory and issue these commands [update may be: /etc/init.d/apache2 restart]:</li></ol> apache2ctl stop apache2ctl start  Then test site access and keep your fingers crossed!</li></ol>
 * 1) First, name your site with its exact tld extension (not sure why this works or is required, but it appears to be so.  Thus, for our new-sites reference for Windows, it becomes:

Tips
/etc/apache2/sites-enabled/ /etc/apache2/conf.d/apache.conf /etc/apache2/apache2.conf and .htaccess in multiple locations. Generally, try to keep to the specific ones recommended
 * This material assumes use of Apache2.2; Apache documentation is notoriously bad and obscure, but it may be found [xxx here]
 * Depending on your version of Linux, the actual file locations noted above may differ (the ones above pertain to Ubuntu and Debian); if in doubt, look for the directory where vhosts.conf might be listed (or similar) or inspect the bottom of the apache2.conf file to see what directories Apache looks to for its configurations
 * Note there are many ways to change Apache configurations. On our system, other locations for configuration files can include:
 * Try to keep the directory and DB names closely aligned for each new site (so as to be readily identifiable and uniform)
 * Keep subdirectory names very close, but slightly different for the local and remote hosts. This will enable easy search-and-replace (see below) while not getting confused in calling up the local or remote versions of the same site.
 * In written material, you might see reference to var/www on Linux machines. This is generally a symbolic redirect directory. But, again, as noted, make sure you understand where your Apache and Drupal sites directories are actually located on your specific installation
 * On Linux machines, use 'sites-available' v. 'sites-enabled'

Other Useful References

 * http://www.givegoodweb.com/post/99/drupal-multi-site-dedicated-server
 * http://kutucape.com/tag/drupal
 * http://www.howtoforge.com/multisite_drupal_installation_ubuntu
 * http://drupal.org/node/134439

Transferring Sites between Local (Sandbox) and Remote
We can now put all of these pieces together and describe the steps necessary for moving sites between local and remote (or vice versa). This assumes you have at least one working site, and both local and remote sites have already been initially installed.

Configuring for Transfers
Prior to beginning a transfer, some prep steps are helpful:
 * 1) Cleaning up tables from within PHPMyAdmin; Drupal tends to accumulate all prior changes which can lead to much cruft in the database.  Certain queries or basic table inspections can be helpful here.  Check search engines with variants of the 'drupal mysql clean up tables queries' search query to learn more
 * 2) Disable any undesired modules or themes, then delete them from the file system
 * 3) Make sure installed modules and themes are similar between both installations
 * 4) Run cron on the local system
 * 5) Clear cache (Performance) under the Admin panel

Preparing the Database
'/new-site' --> '/new-site.com' by this manner we are making local file references conform to the remote file naming convention (or vice versa). The references need to refer exactly to the site directory names on the target and destination installations. Again, note that the direction of this global search and replace may be the opposite for a transfer in the other direction. Important: also note the use of the '/' slash, which only makes changes for the file directory references  Save the file as something different. In this case, where we are moving from local to remote, consider naming the file new-site_remote.sql</li></ol>
 * 1) From within PHPMyAdmin for the site with the database you wish to transfer, login with your username and password and select the database of interest.  Then, chose the export option with all defaults left on (export as SQL) and chose the 'Save as File' option and then 'Go'.  Pick a location to save the export (which is in readable text) somewhere you can easily find it
 * 2) After the export is complete, exit PHPMyAdmin
 * 3) Open the file (new-site.sql) with a capable text editor. Depending on the site being transferred, make this global change (it could be reverse direction if the remote Linux is being transferred to the local Windows):

The Actual Transfer
mysql -pyourpassword new-site<new-site_remote.sql  Shut PHPMyAdmin and invoke the now updated Drupal site (e.g., http://new-site.com)</li> Log in (prefereably as the super admin; see above) and run update.php. If you are already logged in, you can invoke http://new-site.com/update.php. Follow the instructions on the screen. Depending, you may see a site without styles or your theme when completed with the update. Don't panic! If this is the case, simply go into the /admin/build/themes option and turn your theme back on </li> Make other minor file adjustments and set file system paths within the Drupal install </li> Check modules and whether they are turned on and configured properly</li> Check permissions for all modules and users</li> Check the Status Reports to see if any unaddressed errors remain</li>  Run cron manually (Admin --> Status Report)</li> Change footer if necessary (it resides outside of the Drupal MySQL code base).</li></ol>
 * 1) Create DB shell using PHPMyAdmin
 * 2) Install Drupal (registers the DB to Drupal) to the target instance if you have not already done so
 * 3) Delete the DB from the system using PHPMyAdmin
 * 4) Copy the replacement database from the target to the destination.  Consider saving the transferred text (*.sql) file to a destination directory such as 'backups' (because these transferred files also act as backups as well)
 * 5) Copy the full DB to the shell.  Depending on the size of the database, you can either do this from within PHPMyAdmin or not:
 * 6) * For smaller files, use the Import option from within PHPMyAdmin by first selecting the destination database (new-site), which should be empty, then Import and follow the dialog to pick the new-site_remote.sql file option from the subdirectory to which it was copied during transfer
 * 7) * For larger files, you may have to do the import directly with MySQL. So, go to your backup directory, and invoke this command:
 * You may also need to make some adjustments to various settings. Here are a few to pay attention to:
 * Your theme configuration, such as:
 * new-site --> new-site.com
 * image and favicon directories
 * footer information (make sure and set any image links correctly!)
 * Update and provide links under File Settings
 * Update and provide links under Images --> Image Import

Reverse Transfers

 * Simply reverse the target and destinations in the above and repeat the same steps.

Other Useful References

 * Backing up and moving MySQL files: http://satellite.esn.org/wiki/index.php?title=Back-up and the earlier version, http://drupal.org/node/75545

Other Tips
$update_free_access = TRUE; the flag should be set back to 'FALSE' when the update is done to reinstitute correct security. Also, if you do not make this change, you will get error warnings within Drupal until you do!
 * As best practice, for both themes and modules, try to adhere to these guidelines:
 * For installed modules and themes that come with the basic Drupal distribution, keep them and them alone in the theme and module directories at the first level under Drupal
 * Place all shared themes and modules that are not part of the original Drupal distribution under the 'all' sites directory (e.g., [Drupal root]/sites/all)
 * Only place site-specific modules or themes under their respective site directory (e.g., [Drupal root]/sites/new-site/themes or [Drupal root]/sites/new-site/modules)
 * When first installing a new site, try to have the site's "super admin" or main owner do the actual install. That is because the first user to configure a Drupal site is given user ID 1, which has later update rights (see further this article)
 * However, when later updates are required, by changing this normal value of 'FALSE' to 'TRUE' as below in the site's settings.php file, the current logged in admin can do the update without being a super admin:
 * We suggest to offline when not in active development (via /admin/settings/site-maintenance), or set up some IP-based access restriction within the Apache virtual host file
 * When you want to bring the site back online, simply type in 'user' after the slash for your site (http://new-site.com/user) to get the login which will enable you to see the site or turn the site back on.

Other References

 * More multi-sites on Windows: http://drupal.org/node/32715
 * http://mdrahman.wordpress.com/2008/07/30/how-to-install-drupal-on-xampp-in-windows-xp/ basic initial setup
 * http://drupal.org/node/307956
 * Junction symbolic linking utility for Windows: (not sure needed)
 * junction.exe C:\3a-XAMPP\htdocs\drupal\sites C:\3a-XAMPP\htdocs\drupal