Do simple things

I recently decided that I wanted an easy way to check if a given game is available on game pass or ps plus since I subscribe to both. I wanted something faster than checking the official sites, booting up my consoles or going to some other site. There are sites like https://gamepasscounter.com/ and https://psplustracker.com/ but they don't show both at the same time and there is no search function. I thought that it would be a great idea to take this opportunity to give Next.js a shot for something else than this blog.

First problem was getting the data. Microsoft and Sony respectively have sites that show these games so let's start there. Microsofts site requires Javascript to function and isn't especially easy to scrape. I managed to do it with a Puppeteer script that burps out the games in JSON, for both PC and Xbox. The Sony site was a bit easier and now I had two json files with games. Now what? Obviously I need to deduplicate them since I don't want the same game to show up twice right. To do this I used N8N. N8N calls the web service for extracting the games and then deduplicates them. That took care of the worst of the problem but there were still duplicates. Turns out that they are spelled differently sometimes and sometimes they even include things like "xbox version" or "ps4 & ps5". I spend some time cleaning that up using regular expressions and then I had a(not perfect) version that spits out the JSON that I wanted.

Whats next? Storing the data obviously. I can't run this thing every time since it takes 5 minutes to run due to Javascript scripting and pagination. Where do I put the data though? I have this thing that I use to generate RSS feeds. It can take any data in JSON format and store it. It can then output the data as anything: JSON, XML, raw text, RSS, etc. Lets use that! It took me maybe 10 minutes to wire up something that saved the games there. OK so now I have the games in a workable JSON format. Now all I have to do is show them and make them searchable.

I already mentioned that I wanted to use Next.js so let's use that. I spend some time wiring up something that looked decent using Tailwind(my preferred CSS framework for simple projects) and then an API route to search in the JSON blob I had. First problem: My feed service doesn't have search. Rather than adding that I thought that I'd fix two problems at once and put it in the Cloudflare cache. I thought that people shouldn't have to wait for a roundtrip to Europe if they are located in the US. Why not let Cloudflare cache this thing for me and then the search can be done on the users machine? You can do all of this client side with no issues since the number of games is only like 1500 or so. So what do we do now? We have this Next.js site that I need to run on Cloudflare Pages or Cloudflare Workers, but that doesn't have a Node runtime so I can't use API routes. Right? Not quite! Cloudflare has the ability to have serverless functions in this setup as well if you put them in the functions directory of your project. This means that you have to use the beta version of wrangler however. That turned out to sort of work but it was buggy. I still decided to give it a try.

What I wanted to do is save the games somewhere and then use Cloudflare KV storage to act as a fast cache. I decided on saving the games in a Google Sheet, since that has a nice API and works well with N8N.

After a few more hours of wrangling I had something that resembled a working serverless function. On a request it tries to access the games in the Cloudflare KV storage with a cache TTL of 3600 seconds. If that doesn't exist, or if the cache has been busted, then it fetches the games from the Google Sheet using an N8N webhook. Now we're in business! This was Very fast since everything was cached on the edge on Cloudflares network. For a few minutes I was very happy with this setup. Until I had to change something that is. Developing this locally in any meaningful way turned out to be a hassle and any change to the brittle code in the functions meant breakage in a number of ways.

It was at this point that I had a thought: "Wait a minute, what was the purpose of this thing again? It was to be able to search for a title of a game and find out if it is available on Game Pass or PS Plus. Why do I need all this stuff just to do that?. I have this typesense index that I use to index all kinds of documents that I want to search for. Can't I just use that?"

30 minutes later I had a working version of exactly what I wanted to accomplish. No edge functions; no brittle setups; no beta releases.

The lesson: Do simple things. Solve the problem at hand. Prefer things that you can run on your own machine. Do not go down weird rabbit holes unless that was the purpose in the first place.