How Web Scraping Is Used To Extract Walmart’s Store Location?

In this blog, you will learn about extracting information of Walmart’s store locations which is one of the largest retail stores in the U.S. We will search for the Walmart.com for store locations that are based on zip code and fetch the following details:

• Store name
• Store ID
• Distance from a particular address/Zipcode
• Address
• Zip Code
• City
• Contact Details

There is a total 4674 number of Walmart stores in the USA. The below screenshot shows the notification of the information that will be extracted from Walmart.

You can fetch more information from the store detail page that includes timings, operating hours and days, departments, and services.

Data Representation

Go to https://www.walmart.com/store/finder?location=20005&distance=50 in any browser. This is the link to find Walmart store locations in the zip code ‘20005′ and that is within a 50-mile radius area.

When you right-click on almost any link on the webpage, select Inspect Element from the menu bar. The browser will display the HTML Content of web Pages in a toolbar that is attractively designed. To remove all the queries from the Request table, go to the Network tab and click Clear.

Then type in the zip code you’re looking for and hit the Search button. For the time being, let’s use the zip code 20005. Select XHR and then the following request:

stores?singleLineAddr=20005&servicceTypes=pharmancydistance=50

The Request URL will be shown.

https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr=20005&serviceTypes=pharmacy&distance=50.

If you open the hyperlink in a new window, you’ll see the raw data. Download the JSON formatted add-on to see the data in JSON format.

Developing The Scraper

For this lesson, we’ll need Python 3. If you’re running Python 2.7, the code won’t be running. Python 3 and PIP must be installed on your PC.

Python is pre-installed on most UNIX web browsers, such as Linux and Mac OS. However, Python 3 is not installed by default on all Linux operating systems.

python --version

Type the code given above and hit enter. You have Python 3 installed if the result looks like Python 3.x.x. You have Python 2 if it says Python 2.x.x. If it gives you an error, you most likely don’t have Python installed.

Install Python 3 first if you don’t already have it.

The Code

import csv
import requests
import json
import argparse
import traceback
def locate_stores(zip_code):
"""
Function to locate walmart stores
"""
url = "https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr=%s&serviceTypes=pharmacy&distance=50"%(zip_code)
headers = { 'accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'accept-encoding':'gzip, deflate, br',
'accept-language':'en-GB,en;q=0.9,en-US;q=0.8,ml;q=0.7',
'cache-control':'max-age=0',
'upgrade-insecure-requests':'1',
'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36'
}
stores = []
print("retrieving stores")
for retry in range(10):
try:
get_store = requests.get(url, headers=headers, verify=False)
store_response = get_store.json()
stores_data = store_response.get('payload',{}).get("storesData",{}).get("stores",[])
if not stores_data:
print("no stores found near %s"%(zip_code))
return []
print("processing store details")
#iterating through all stores
for store in stores_data:
store_id = store.get('id')
display_name = store.get("displayName")
address = store.get("address").get("address")
postal_code = store.get("address").get("postalCode")
city = store.get("address").get("city")
phone = store.get("phone")
distance = store.get("distance")
data = {
"name":display_name,
"distance":distance,
"address":address,
"zip_code":postal_code,
"city":city,
"store_id":store_id,
"phone":phone,
}
stores.append(data)
return stores
except:
print(trackback.format_exc())
return []   
if __name__=="__main__":
argparser = argparse.ArgumentParser()
argparser.add_argument('zip_code',help = 'zip code to search')
args = argparser.parse_args()
zip_code = args.zip_code
scraped_data = locate_stores(zip_code)
if scraped_data:
print ("Writing scraped data to %s_stores.csv"%(zip_code))
with open('%s_stores.csv'%(zip_code),'wb') as csvfile:
fieldnames = ["name","store_id","distance","address","zip_code","city","phone"]
writer = csv.DictWriter(csvfile,fieldnames = fieldnames,quoting=csv.QUOTE_ALL)
writer.writeheader()
for data in scraped_data:
writer.writerow(data)

Run the entire script with script name followed by zip code:

python3 walmart_store_locator.py zipcode

For instance, to search all the Walmart stores in and around Boston, Massachusetts, for example, we would use the input 20005 for zip code:

python3 walmart_store_locator.py 20005

A file named 20005 stores.csv should be created in the same directory as the script. This is what the output file will look like.

For professional help on how to extract Walmart store location data, you can contact LocationsCloud.