Magento upgrade from 1.4, 1.5, 1.6, 1.7, or 1.8 to 1.9

Created on 2011-Jul-09

NOTE: this upgrade process works the same way for Magento version 1.4, 1.5, 1.6, 1.7, 1.8, and 1.9

As many magento users will know, Magento is no fun to work with-- mainly due to poor community support (the developers help paying customers; i.e. Magento Enterprise ... Only payed support there... Great for business but bad for the average or beginning user...)

Anyhow...

So here's how we do it.

MAKE A BACKUP!

You can skip this step all together, however, please make a backup of everything prior to beginning the upgrade.

Create a test environment somewhere on your server (preferably away from your production installation)

I usually create a subdomain on another domain and setup files there.

cd /home/DEVUSER/public_html/DEVSUBDOMAIN
# This following command will sync all files from /home/PRODUCTIONUSER/public_html/ (SOURCE) to /home/DEVUSER/public_html/DEVSUBDOMAIN/ (DESTINATION) 
# the --delete-before switch tells it to delete any files on the DESTINATION that are NOT present on the SOURCE.
# 'z'  tells it to compress the files in transfer so to use less bandwidth (in case you're doing external locations).
# 'P' combines --progress and --partial ; which means it will show a progress bar and will resume partial transfers.
rsync --avzP --delete-before /home/PRODUCTIONUSER/public_html/ /home/DEVUSER/public_html/DEVSUBDOMAIN/

Again, just so you don't mess this up, the rsync command works like this:

rsync -avzP --delete-before [SOURCE] [DESTINATION]

BONUS: if you're updating an already existing DEV location, do the following instead, so to NOT overwrite app/etc/local.xml

rsync -avzP --exclude=app/etc/local.xml --exclude=media/import --delete-before [SOURCE] [DESTINATION]

Fix modman linked files

cd .modman
modman repair

Create a new database for this development domain--via whatever method you have available (usually through your web server control panel).
Clean Magento Logs to minimize database backup size

Under Admin, go to System --> Configuration --> (sidebar menu) Advanced --> System --> (main column) Log Cleaning and set "Enable Log Cleaning" to "Yes". I also set the "Save Log, Days" to "15"

NOTE: if this is your first time enabling log cleaning, you will then want to run the following in SSH at your base magento installation--this will take a while if you haven't ever cleaned your logs:
```
php -f shell/log.php clean –days 15
```
Or... If you don't care about saving any logs and want to do this as fast as possible, then do this in phpMyAdmin:
```
truncate dataflow_batch_export;
truncate dataflow_batch_import;
truncate log_customer;
truncate log_quote;
truncate log_summary;
truncate log_summary_type;
truncate log_url;
truncate log_url_info;
truncate log_visitor;
truncate log_visitor_info;
truncate log_visitor_online;
truncate report_event;
# truncate report_viewed_product_index 
# truncate report_compared_product_index
```

Make a backup of the current database in SSH via:

mysqldump -u user -p --add-drop-table DB_NAME > DB-NAME.sql

Dump the backup you just made into your new database
```
mysql -u user -p DB_NAME < DB-NAME.sql
```
Change the database settings to point to new database
```
vi app/etc/local.xml
```
Change your magento URLs to In phpmyadmin go to table
```
core_config_data
```
and change both to: http://YOURDEVDOMAIN.com/ (do NOT put 'https' for any, as this will create problems)
```
web/unsecure/base_url
```
and
```
web/secure/base_url
```
Also, change your web/cookie/cookie_domain from www.YOURPROUDCTIONDOMAIN.com to www.YOURDEVDOMAIN.com
(If you don't, you won't be able to add products to your cart while testing)
```
web/cookie/cookie_domain
```
Side note: as an alternative, you can do a full search and replace of the backup to change the urls like so
```
sed -i 's|www.PRODUCTIONDOMAIN.com|www.YOURDEVDOMAIN.com|g' ./DB-NAME.sql
```

Turn on system.log logging, set from '0' to '1'

system/log/enabled
dev/log/active

Turn on FULL PHP error reporting in index.php

/* change to or add the following--this will be scattered throughout the file */
error_reporting(E_ALL);
$_SERVER['MAGE_IS_DEVELOPER_MODE'] = true;
ini_set('display_errors', 1);

REMEMBER to change all settings back to normal, after the upgrade is done and upgraded files and database are transferred to the production environment.

Upgrade Magento to 1.9

1. Download the latest version of Magento

Since Magento 1.9.2.0, you can no longer directly download the files. Hence you'll have to use https://www.magentocommerce.com/download (and manually SFTP to your server OR use the github repo

git clone https://github.com/OpenMage/magento-mirror.git

Or an older version:

wget http://www.magentocommerce.com/downloads/assets/1.8.1.0/magento-1.8.1.0.tar.gz
wget http://www.magentocommerce.com/downloads/assets/1.7.0.2/magento-1.7.0.2.tar.gz   
wget http://www.magentocommerce.com/downloads/assets/1.6.2.0/magento-1.6.2.0.tar.gz  
wget http://www.magentocommerce.com/downloads/assets/1.5.1.0/magento-1.5.1.0.tar.gz  

tar xvfz magento-1.8.1.0.tar.gz
tar xvfz magento-1.7.0.2.tar.gz
tar xvfz magento-1.6.2.0.tar.gz 
tar xvfz magento-1.5.1.0.tar.gz

Direct download of magento patches

wget http://www.magentocommerce.com/downloads/assets/ce_patches/NAME-OF-PATCH

# Example: SUPEE-4829 - This patch fixes an issue in which product images become larger when a shopper selects a swatch on a search result page. - Added Nov 26, 2014
# Note: This patch is applicable only to Magento Community Edition 1.9.1

wget http://www.magentocommerce.com/downloads/assets/ce_patches/PATCH_SUPEE-4829_EE_1.14.1.0_v1.sh

2. Disallow access to your site if doing the upgrade on a production environment (this is why you should do this on a test environment)

mv .htaccess .htaccess-bkp
cp magento/.htaccess .
cp .htaccess .htaccess-new

In your new .htaccess change the Order Allow to (at the bottom of file):

#Order allow,deny     
#Allow from all 
Order deny,allow 
Deny from all 
Allow from YOUR IP ADDRESS

Also, up the memory usage (especially if your database backup is over 500MB--hopefully you're running magento on a dedicated server.)

php_value memory_limit 1024M
php_value max_execution_time 45000

3. Flush all caches and rebuild compiler

Using a bash script to clear all caches and restart all related web server services (requires root/su access):

cd magento/installation
touch clear_cache.sh
chmod 700 clear_cache.sh
vi clear_cache.sh

## PASTE THE FOLLOWING and SAVE : ##

#!/bin/bash
read -p "Continue to clear all magento cache? (y/n)" -n 1 -r
echo -e "\n"  # move to new line
if [[ $REPLY =~ ^[Yy]$ ]]
then
    echo -e "Clearing file cache \n"
    rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/*
    rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*
    echo -e  "Restarting PHP-FPM, Apache, and Memcached \n"
    service php-fpm70 restart
    service php-fpm71 restart
    service php-fpm56 restart
    service httpd restart
    service memcached restart
    echo -e "\n***  BAM! All caches cleared *** \n"
fi

Now to clear all caching we simply do:

./clear_cache.sh

Here is the manual way of doing the above:


rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/*
rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*
service httpd restart
service memcached restart

or

rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/* ; rm -rf downloader/pearlib/cache/* downloader/pearlib/download/* ; service php-fpm70 restart ; service php-fpm71 restart ; service php-fpm56 restart; service httpd restart ; service memcached restart

Enable/Disable compiler and clear compiled files

Make sure /includes is writable

chmod 777 includes
vi includes/config.php

Uncomment lines:

define('COMPILER_INCLUDE_PATH', dirname(__FILE__).DIRECTORY_SEPARATOR.'src');
define('COMPILER_COLLECT_PATH', dirname(__FILE__).DIRECTORY_SEPARATOR.'stat');

php -f shell/compiler.php -- state
php -f shell/compiler.php -- disable
php -f shell/compiler.php -- clear
php -f shell/compiler.php -- compile
php -f shell/compiler.php -- enable

4. Remove the 'downloader' and 'app/design/frontend/base' directory

rm -rf downloader

DON'T DO THE FOLLOWING if you have 3rd Party extensions that use these paths!

rm -rf downloader
rm -rf app/design/frontend/base
rm -rf skin/frontend/base

This is the safest bet so mage/pear doesn't upgrade the wrong installation; trust me this happened to me a couple of times, EVEN after doing ' mage-setup'.

As for the 'base' directory, we remove that because deprecated files exist in there that may cause problems. E.g. One page checkout would not work for anon users, but would work for registered users--and this happened even when all the files were overwritten with magento 1.5.1; meaning magento was picking up deprecated files.

You may also want to clear out redundant theme files in app/design/frontend/default/YOURTHEME/ AND app/design/frontend/default/default/

The following are the only custom template files I use--it's better to use the least amount possible to avoid future upgrade headaches

app/design/frontend/default/YOURTHEME/layout/catalog.xml
app/design/frontend/default/YOURTHEME/template/callouts/left_col.phtml
app/design/frontend/default/YOURTHEME/template/catalog/product/view.phtml
app/design/frontend/default/YOURTHEME/template/checkout/success.phtml
app/design/frontend/default/YOURTHEME/template/page/1column.phtml
app/design/frontend/default/YOURTHEME/template/page/2columns-left.phtml
app/design/frontend/default/YOURTHEME/template/page/2columns-right.phtml
app/design/frontend/default/YOURTHEME/template/page/3columns.phtml
app/design/frontend/default/YOURTHEME/template/page/html/footer.phtml
app/design/frontend/default/YOURTHEME/template/page/html/head.phtml
app/design/frontend/default/YOURTHEME/template/page/html/header.phtml

Same goes for app/design/frontend/default/default/. You want to remove any redundant theme files in there as well. Basically, you'll ONLY want to keep any installed modules you may be using; get rid of the rest.

e.g. I only have the following in app/design/frontend/default/default/:

app/design/frontend/default/default/etc/widget.xml # I probably don't even need this
app/design/frontend/default/default/layout/vertnav.xml
app/design/frontend/default/default/template/vertnav/left.phtml

5. Copy 'downloader' and 'app/design/frontend/base' folder from magento-1.8.1.0

cp -a magento/downloader .
cp -a magento/app/design/frontend/base/ app/design/frontend/

6. Install/get 'mage'.

cp magento/mage .
chmod 755 ./mage

We do this because the 'pear' method doesn't work anymore--it only does an upgrade up to magento 1.4.2; it won't work upgrading any higher than that.

7. Do the upgrade

** IMPORTANT: make sure your Magento Connect Manager 'Preferred State' is set to 'Stable' **

./mage config-set preferred_state stable

Or, you can set it here:

https://www.YOURWEBSITE.com/downloader/index.php?A=settings

Do the copy 3 times. Yes, you read that correctly. For whatever reason, doing

yes | cp -Rf magento/* .

sometimes does NOT copy all files on it's first try; and you get no error or indication that this happened. I had this happen to me at least during 2 different upgrades.

yes | cp -Rf magento/* .
yes | cp -Rf magento/* . 
yes | cp -Rf magento/* .

./mage mage-setup .
./mage sync --force
./mage install http://connect20.magentocommerce.com/community Mage_All_Latest --force
rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/*
rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*  
chmod 755 mage 
php shell/indexer.php reindexall 
#upgrade everything including third party modules
./mage upgrade-all --force

8. Set the proper file permissions

mkdir -p var/package; mkdir -p var/cache; mkdir -p media/; mkdir -p var/locks; mkdir -p var/report; mkdir -p var/export;
find . -type f -print0 | xargs -0 chmod 644; l
find . -type d -print0 | xargs -0 chmod 755;
  
chmod 755 mage 
chmod o+w var var/.htaccess app/etc
chmod -R o+w media
chmod -R 777 var/package var/locks var/report var/export downloader

# or do the above on a single line
mkdir -p var/package; mkdir -p media/; mkdir -p var/cache; mkdir -p var/locks; mkdir -p var/report; mkdir -p var/session; mkdir -p var/log; mkdir -p var/export; find . -type f -print0 | xargs -0 chmod 644; find . -type d -print0 | xargs -0 chmod 755; chmod 755 mage; chmod -R 777 var/cache; chmod -R 777 var/log; chmod -R 777 var/session; chmod a+x cron.sh; chmod o+w var var/.htaccess app/etc; chmod -R o+w media; chmod -R 777 var/package var/locks var/report var/export downloader;

# replace USER with whatever is the username of the account you're working under
chown -R USER:USER *

yes | cp -Rf magento/* . ;yes | cp -Rf magento/* . ;yes | cp -Rf magento/* .

9. Visit the home page of your upgraded site. This will trigger the magento upgrade process.

This can take anywhere from 2-30minutes, so be patient. You'll also likely get a browser error before the upgrade is done--don't worry about it, it's normal, and keep waiting...

Run

top

and watch mysql process until it stops using up high cpu usage--once it does, you'll know the database upgrade is done.

top

If everything goes well, you should NOT see any errors. If you do see the troubleshoot secion below.

Tidy up the upgrade

Apply Magento 1.9.2.0 patch SUPEE-4829

This patch fixes an issue in which product images become larger when a shopper selects a swatch on a search result page. - Added Nov 26, 2014 Note: This patch is applicable only to Magento Community Edition 1.9.1

wget http://www.magentocommerce.com/downloads/assets/ce_patches/PATCH_SUPEE-4829_EE_1.14.1.0_v1.sh
sh PATCH_SUPEE-4829_EE_1.14.1.0_v1.sh

As of Magento 1.8.1 you're going to get a google checkout error in the /admin section. Here is the official Magento fix:

Delete all files except config.xml from the following directory:

rm -rf app/code/core/Mage/GoogleCheckout/etc/adminhtml.xml && rm -rf app/code/core/Mage/GoogleCheckout/etc/system.xml && rm -rf app/code/core/Mage/GoogleCheckout/etc/wsdl.xml && rm -rf app/code/core/Mage/GoogleCheckout/etc/wsi.xml

rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/* && rm -rf downloader/pearlib/cache/* downloader/pearlib/download/* && service httpd restart && service memcached restart

11. Login to the magento admin section and mage sure the version number reflects the version you just upgraded to: https://YOURSITE.com/admin

12. Change back your .htaccess to original value

cp .htaccess-new .htaccess

13. You'll probably want to use a file comparison program to see the difference in template files

download the latest version from magento connect if possible
if not possible and you made your own template, then the best thing to do is to DELETE all the unecessary custom template files from your custom template directory, and only keep the crucial ones.
Following use something like Beyond Compare and file compare between the /base/default app/design/frontend/base/default and /app/design/frontend/default/CUSTOM

14. Fix magento URLs

In the admin interface admin/system_config/edit/section/catalog/ set the following under Search Engine Optimization:

Autogenerated Site Map: enable
Popular Search Terms: enable
Product URL Suffix: blank
Category URL Suffix: blank
Use Categories Path for Product URLs: yes # or no depends on who you ask
Create Permanent Redirect for old URLs if Url key changed: yes
Page Title Separator: -
Use Canonical Link Meta Tag For Categories : yes
Use Canonical Link Meta Tag For Products: yes

Now re-index your url rewrite data in admin/process/list/: Catalog URL Rewrites

Or in SSH:

 php shell/indexer.php reindexall

Troubleshooting

Now...

If something went wrong (and it probably did) you have several options.

If a non-release version is showing then mage installer grabbed the release candidate stuff instead.

The fix: Go back to Do the Upgrade and restart from there.

The site just hangs after upgrade (keeps loading for minutes on end):

Be patient, use

top

in SSH and wait for mysql high CPU usage to go down

PHP Fatal error: Call to a member function toHtml()

edit app/design/frontend/default/YOURTHEME/layout/page.xml

or edit app/design/frontend/default/default/layout/page.xml

Change from

to:

Cannot login to magento

edit app/code/core/Mage/Core/Model/Session/Abstract/Varien.php and comment out the following:

/* 
$cookieParams = array( 
\'lifetime\' => $cookie->getLifetime(), 
\'path\' => $cookie->getPath(), 
\'domain\' => $cookie->getConfigDomain(), 
\'secure\' => $cookie->isSecure(), 
\'httponly\' => $cookie->getHttponly() 
);
if (!$cookieParams[\'httponly\']) { 
unset($cookieParams[\'httponly\']); 
if (!$cookieParams[\'secure\']) { 
unset($cookieParams[\'secure\']); 
if (!$cookieParams[\'domain\']) { 
unset($cookieParams[\'domain\']); 
} 
} 
}

if (isset($cookieParams[\'domain\'])) { 
$cookieParams[\'domain\'] = $cookie->getDomain(); 
}

call_user_func_array(\'session_set_cookie_params\', $cookieParams);

if (!empty($sessionName)) { 
$this->setSessionName($sessionName); 
} 
*/

Cannot add or update a child row: a foreign key constraint fails: googleshopping_setup

The only remedy I was able to find for my case when upgrading from magento 1.3.2 to 1.6.0 was i needed to do upgrade to magento 1.6.1-rc1.

./mage upgrade-all --force

Cannot add products to cart at DEV/TESTING URL/Domain

This is caused by a mismatch of the development URL with what is set in your database under 'core_config_data' for 'web/cookie/cookie_domain'. Use phpMyAdmin and change the domain name to whatever you are using under your DEV url

Go to table:

core_config_data

Change the value to your dev url in:

web/cookie/cookie_domain

Blank screen or PHP Fatal error: Class 'Magento_Db_Adapter_Pdo_Mysql' not found at /admin

One cause of this that I ran into was the server's PHP or memcached caching. Solution: flush all magento caches and restart the apache and memcached.

rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/*
rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*
service httpd restart
service memcached restart

or

rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/* && rm -rf downloader/pearlib/cache/* downloader/pearlib/download/* && service httpd restart && service memcached restart

Another strange reason was due to 'skin/adminhtml/default/default' not containing all files.

yes | cp -Rf magento/* .
yes | cp -Rf magento/* . 
yes | cp -Rf magento/* .
# yes 3x times; see above.

Fresh start: in case something went terribly wrong

Go into phpmyadmin and drop all the tables so your database is empty (you'll get errors in phpmyadmin about foreign keys; that's normal, just repeat till all tables are gone) (i prefer this over the SSH method, cause it's too easy to drop the wrong database by accident)

However, there's a good chance phpMyAdmin may run out of resources when doing this, so you may want to checkout this post instead: Drop all tables in MySQL via SSH

Now re-dump your database backup

mysql -u user -p DB_NAME < DB-NAME.sql

and copy over a fresh version of magento over your current install

yes | cp -Rf magento/* .
yes | cp -Rf magento/* . 
yes | cp -Rf magento/* .
# yes 3x times; see above.
  

rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/*
rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*
service httpd restart
service memcached restart

or

rm -rf var/cache/* var/session/* var/locks/* var/full_page_cache/* tmp/* && rm -rf downloader/pearlib/cache/* downloader/pearlib/download/* && service httpd restart && service memcached restart

Visit your site again...

Bonus: Install Google Base (shopping) vis SSH

./mage install http://connect20.magentocommerce.com/community Mage_GoogleShopping --force

To uninstall an extension:

./mage uninstall http://connect20.magentocommerce.com/community Mage_GoogleShopping --force

If you get a permissions error trying to run './mage'

-bash: ./mage: Permission denied

easy fix:

chmod 750 ./mage

Look here for a complete listing of required Google Base attributes for different product categories: http://www.google.com/support/merchants/bin/answer.py?answer=1344057

An interesting thing happened when I downgraded from PHP 5.4 to 5.3, I got the following error message "could not determine temp directory ..." Solution: make sure your php.ini has the following in it:

safe_mode = off

Google checkout error after upgrade to Magento 1.8.1.0