This tutorial contains 301 redirect checker script which I have created after spending more than half an hour searching the Internet for something suitable I can use for my needs. While I was searching I found only separate web pages with fields where you can check one – two URLs or heavy applications that can be installed, and yet not covering my needs. Certainly, there were paid 301 redirect checker solutions which can be used to check 301 Permanent Redirect, but I am not sure whether it does worth to buy such. The purpose of this checker in most cases is only to see if a particular pages redirect permanent covering the SEO (Search Engine Optimization) requirements.
In my case part of the OnlineHowTo.net is moving over Free Tutorials Submit dot com, and from SEO perspective, the tutorials moved have to be with 301 redirect to their new URLs.
I decided to write it as Linux bash 301 redirect checker script because it can be run on almost any Linux machine with curl installed and easily added to the Crontab. I believe the script is rather easy to be understood and used, even by not so experienced administrators.
What the redirect script is doing is to gather the URLs you want to check from an external file (in my case: /opt/scripts/301URLs.txt ), where every URL is placed on a separate line like this:
-
[…]
-
http://www.freetutotrialssubmit.com/how-to-make-the-taskbar-buttons-switch-to-the-previous-active-window-in-windows-7/1644
-
http://www.freetutotrialssubmit.com/how-to-disable-the-winkey-shortcut-keys-in-windows-7/1725
-
http://www.freetutotrialssubmit.com/speed-up-windows-7-by-forcing-it-to-unload-unused-dlls-and-delete-cached-ones/1786
-
http://www.freetutotrialssubmit.com/remote-access-software/2080
-
http://www.freetutotrialssubmit.com/ftp-voyager/109
-
http://www.freetutotrialssubmit.com/how-to-restore-lost-plesk-password-on-unix-linux-server/112
-
http://www.freetutotrialssubmit.com/how-to-mount-cd-under-linux/206
-
http://www.freetutotrialssubmit.com/online-community-2/2142
-
http://www.freetutotrialssubmit.com/scrolling-social-plugin/2144
-
http://www.freetutotrialssubmit.com/explained-hardware-servers-vs-cloud-hosting/2151
-
[…]
I chose this way as most of the get-all-urls-from-domain applications provide the listing like this, but certainly you can choose (and then modify a little bit my script) different format.
Then using simple bash loop it reads every line one by one, processing it with CURL extracting the headers (curl -I option).
Every header contains information similar to:
-
Date: Thu, 18 Aug 2011 17:23:57 GMT
-
Server: Apache/2.2.3 (CentOS)
-
Location: http://www.freetutorialssubmit.com/ftp-voyager/109
-
Connection: close
-
Content-Type: text/html; charset=iso-8859-1
Once we have this information we can easily egrep for the 301 Moved Permanently part and if it exist, the 301 redirect checker script continues with the next URL. If the redirect is something different it writes down the URL into the /tmp/301report.txt file.
Finally the script checks whether the /tmp/301report.txt file exist – it will be created only if there is even one different from 301 Moved Permanently redirect – and will send it via e-mail to the address you have set in the EMAIL variable. If there are no such file, the script will end with no report.
Here is the whole script which you can freely use and modify as per your needs:
-
#!/bin/bash
-
-
#The subject of the mail message – can be anything.
-
#For example something that triggers your e-mail client filters.
-
SUBJECT="301 Redirect Compromised"
-
-
#Here you must change with your e-mail address.
-
EMAIL="mail-to-report-to@yourdomain.com"
-
-
#Temporary file I am using to store non 301 redirected URLs.
-
LINKS=/tmp/301report.txt
-
-
#Here I make sure this file does not exist before processing the URLs.
-
rm -f /tmp/301report.txt
-
-
#All URLs which should be checked are in /opt/scripts/301URLs.txt one per line.
-
cat /opt/scripts/301URLs.txt | while read line; do
-
L=`curl -I -s $line | egrep '^HTTP\/1\.1 301'`
-
sleep 5
-
if [ -z "$L" ]
-
then
-
echo $line >> $LINKS
-
fi
-
done
-
-
#Check if there were non 301 redirects stored and e-mail them if any
-
if [ -f /tmp/301report.txt ]
-
then
-
mail -s "$SUBJECT" "$EMAIL" < $links
-
fi
-
-
#EOF
To be honest, I am using a little bit modified 301 redirect checker script, which is also checking whether the redirected URL is the one I need, but it is more complicated and may confuse you if you have to modify it.
Also, more elegant solution will be if you decide to use array, but this redirect checker script fits my needs perfect.
Also I have put sleep command as if the number of the URLs is too big, it may lead to server abuse and blocking the IP you are checking the redirect from. If you do not need the script to wait 5 seconds for every URL, just remove that line.
Hope this scrip will cover your needs as well. Your comments and suggestions are welcome.
/opt/scripts/301URLs.txt