An introduction to reverse engineering and unofficial APIs
- Is there an integration for that ?
- Do they have an API ?
- There HAS to be a way to automate that !
- Do I need to upgrade my Zapier/IFTTT/Integromat plan again ?
If you have wondered this at any point, you landed on the right post !
In my last article, I already tried to help these pain points by showing you how to deploy your own custom API in a few minutes.
Today, we are going to talk about unoffcial APIs, a way to reduce even more frustration of not being able to automate a process on a given SaaS.
I made a whole video about this post if you want to see the process live !
Before we get into it, let’s talk about a tangible example where reverse engineering can be applied to solve automation needs : Notion.
(UPDATE: their’s finally a public API !!)
Notion has become one of the top, if not the top, workspace/organizer app in the span of a few months. Its design is top notch and endlessy customizable.
Now that you’re reading this article, you probably know that it has also a major flaw. (aside from being slow at times)
It has no official API, so there is no official integration, no Zapier, nothing !
That is until you discover an unofficial API.
An unofficial API is basically a reverse engineering of the official API using the frontend of a service.
If you manage to intercept the calls made when you add, update or delete an entry on Notion’s interface, you can reproduce them with any programming language.
In essence, it’s a technique that is very close to scraping.
For common services, there are community made unofficial APIs like this one easily found on GitHub.
As mentioned on the repository, the creator of this unofficial API uses Notion as a CMS and his API to download the notion pages.
He then uses a flow to convert the downloaded data to html and then publish his new page automatically.
This means we can start building cool stuff like :
- Create a card when sending a message to a Slack bot
- Change a item status in Notion based on Github actions
- Create an api of a Notion table like you would with Airtable
As another example, we recently made the switch to webflow at Quable, after the OVH burnt server fiasco.
We hosted Quable’s homepage on Wordpress hosted on a burnt OVH server, and we had no backup !
We had to react fast, in particular not to lose our SEO efforts…
We had over 300 blog posts, in French and in English to redirect somewhere and not to a dead 404 page !
The plan was to set up an emergency website on Webflow, and redirect as much links
The problem is : Webflow has an API, but they do not do bulk redirections, so the only solution would be to add every single redirection by hand …
That is unless you know about a little trick related to unofficial APIs.
The idea here is to automate the process of:
- Finding the 300+ links that need to be redirected somewhere (because remember: we didn’t have website to even look links up)
- Adding them to webflow in bulk without doing it by hand
Finding the 300+ most relevant links
I used APIfy to scrap the first 300 Google links returned by the query `site:quable.com` which hopefully gave me the most important links of the website.
Once I had my links, I needed to automate my redirections, and I did so unofficialy, with the help of two things : the F12 key and this cURL syntax converter tool.
Adding the redirections in bulk
To add multiple redirections, I had to reverse engineer how my browser actually sends a request to Webflow’s server to add a redirection and then write a script to execute the requests over my scraped links.
To know how my browser handles the request, I use the F12 key which shows the Developer Tools on any common browser, and checkout the Network tab.
To watch the request, I clear the console by clicking on the “forbidden” icon and I add a random redirection like /testA to /testB :
Here it’s pretty easy : the route is explicit and it’s called “redirect”.
To replicate it, I right clicked on it, clicked on “copy as cURL” and used the cURL converter tool to translate it easily to Python:
(Careful when sharing this kind of information especially here where there are cookies containing login data !)
When pasting, the Python code appeared instantly, and in there you can clearly see the our “testA” and “testB” redirection paths !
The rest is simple as I just needed to make a function out of this thing and loop through with my scraped links:
And that’s how I automated the whole process creating a mini unofficial webflow API to suit my needs !
Closing words
Now that I have explained the magic, this type of reverse engineering comes with a few warnings:
- The script will probably not last in time mainly because the login cookies will expire sooner rather and later, and it would break anytime when Webflow updates their API.
- Again : manipulating login cookies is dangerous, be very cautious with them because they give full access to your account
- You can easily trigger alerts to the service you are reverse engineering if you bombard them with requests, and they might ban you outright : That’s why Instagram automation is very risky nowadays.
In this specific case, I put random timeouts between each added redirection between 10 and 100 seconds, and everything went well !
Let me know if there are specific services you would like to see automated this way !
Until next time,
Anas
Originally published at https://www.anas.link.
