Have you ever been in a situation where you try writing a web scraping script to pull some data of a webpage but face a lot of problems as the websites HTML is not structured or the websites are written in some JS frameworks like React JS or Angular JS which make the HTML code really difficult to understand.
I have a neat trick which can pull you out of this situation almost always and the best part is we don’t have to scrape. In fact, it is better to first try & implement this trick before trying out web-scraping.
Airflow has an Operator called “Kubernetes Pod Operator” which would ideally connect to a cluster, create a pod run the desired program and delete the pod. This is a much-required feature where we will be able to run our post-load/Intermediate jobs or process in the Kubernetes cluster which can scale up and wide. This saves a lot in terms of cost as the pods will be up only when we are running the job. …
Wouldn’t it be cool if you can speed up your program by just adding a decorator to the function? Wouldn’t it be cool if you don’t have to worry about running the data in a list as a parallelly?
Today we are going to write a python decorator which exactly does these automatically for you, so that you can concentrate more on the logics of your code than worrying about multi-threading issues.
Some basics on python multi-threading before we start.
A programmer who likes to innovate. I write code when I am happy, sad or angry. It is a stress buster for me.