Follow this tutorial in Python and create an automated site checker with very little code.
If you often find yourself fetching data from websites, you should probably consider automating the process. Sometimes referred to as "web scraping", it's a common process for sites that don't provide an official API or feed. Of course, you won't get anywhere if the site you're trying to fetch isn't available.
If you are running your own site, chances are you may have had to deal with downtime before. It can be frustrating, resulting in lost visitors and interruption of any activity that your site may be responsible for. In such circumstances, it is useful to be able to easily check the availability of your website.
Python is a great language for scripting, and its concise and readable syntax makes executing a site checker a simple task.
Create your own custom website checker
Website Checker is specifically designed to accommodate multiple websites at once. This allows you to easily switch between sites you no longer care about, or start checking out sites you'll launch in the future.
The validator is an ideal "structural application" that you can build on further, but it demonstrates a basic approach to fetching web data.
Import Libraries in Python
To start the project, you have to import the Orders library in Python using the import function.
import requests
The application library is useful for connecting with websites. You can use it to send HTTP requests and receive response data.
Storing URLs in a list
Once the library is imported, you have to specify the site URLs and store them in a list. This step allows you to keep multiple URLs, which you can check with a website checker.
import requests
website_url = [
"https://www.google.co.in",
"https://www.yahoo.com",
"https://www.amazon.co.in",
"https://www.pipsnacks.com/404",
"http://the-internet.herokuapp.com/status_codes/301",
"http://the-internet.herokuapp.com/status_codes/500"
]
The website_url variable stores a list of URLs. Inside the list, select each URL you want to check as an individual string. You can use the example URLs in your code for testing or you can replace them to start checking your own sites right away.
Next, store the messages for common HTTP response codes. You can keep it in a dictionary, and index each message by its corresponding status code. Your program can then use these messages in place of status codes for better readability.
statuses = {
200: "Website Available",
301: "Permanent Redirect",
302: "Temporary Redirect",
404: "Not Found",
500: "Internal Server Error",
503: "Service Unavailable"
}
Create a loop to check the status of the website
To check each URL in turn, you'll need to iterate over the list of websites. Inside the loop, check the status of each site by sending a request through the requests library.
for url in website_url:
try:
web_response = requests.get(url)
print(url, statuses[web_response.status_code])
except:
print(url, statuses[web_response.status_code])
where:
- for url... iterates over the list of urls.
- url is the variable that the for loop assigns each URL to.
- Try/except to handle any exceptions that may arise.
- web_response is a variable that provides a property with response status code
Entire code snippet
If you prefer to review the entire code in one go, here is the complete code list for your reference.
import requests
website_url = [
"https://www.google.co.in",
"https://www.yahoo.com",
"https://www.amazon.co.in",
"https://www.pipsnacks.com/404",
"http://the-internet.herokuapp.com/status_codes/301",
"http://the-internet.herokuapp.com/status_codes/500"
]
statuses = {
200: "Website Available",
301: "Permanent Redirect",
302: "Temporary Redirect",
404: "Not Found",
500: "Internal Server Error",
503: "Service Unavailable"
}
for url in website_url:
try:
web_response = requests.get(url)
print(url, statuses[web_response.status_code])
except:
print(url, statuses[web_response.status_code])
Programming capabilities in Python in scraping the web
Third-party Python libraries are ideal for tasks like web scraping and fetching data over HTTP.
You can send automated requests to websites to perform different types of tasks. This may include reading news headlines, downloading photos, and sending emails automatically.