Data Drudgery

In the last part we went through how to get the WoW auction data using the developer APIs. The auction data dump (auctions.json) is updated once every hour. As I noticed, that this dump is updated just before the hour in UTC. So scheduled job to get the updated auction dump every hour should work fine. In this section we will use Spark to do some basic analysis on the auction data. Simple items have a row like the one shown below. Items like legendary equipment, pets will have additional fields like bonusLists, petSpeciesId etc. Let's take a look at a row of auction data. {"auc":1018440074,"item":41114,"owner":"Lougincath", "ownerRealm":"Dalaran","bid":507084,"buyout":1000000,"quantity":1, "timeLeft":"LONG","rand":0,"seed":0,"context":0} Next we will put the auction json data into a dataframe. As the datadump has some additional me...

I will now delve into real data and the dataset I have chosen is the auction data for World of Warcraft. Each realm has its own auction house, and I will start with the auction house of a single realm. There are more than two hundred realms in NA region alone. There are three more regions, Europe, Asia-Pacific and China.  To fetch the data, one will need an API key, which can be easily obtained by registering on Blizzard Dev . Below is the simple code to get the data and print it. Please note that this API returns the metadata, the response contains the location for complete auction data, which has to be fetched next. This response also contains the last modified time which can be persisted so that it can be used to find when the auction data dump has been updated. Auction metadata The data dump takes quite a while to retrieve (few minutes on my broadband connection), so it would be nice to add a progress status message. Below is a nice utility function to do that. The print...