How do I Handle Heavy Data Processing in RESTful API

Google image: sorry, forgot the link 😭

Phase 1

We used the aggregate function to process the big data calculation on our database. So, the load for it is on the database. We didn’t realize it until the nightmare comes. Yeah, the CPU load of our database is spiking, and make a whole application is slow.

Illustration image:

Phase 2

OK. Now, we decide to make the calculation on the app-level. So, we get all the data from the database first, then loop the data to get the result and return it as a response of the API. And of course, we use Redis as our cache system.

Angry Spongebob Squarepants:

Phase 3

I found the Asynchronous Request-Reply pattern from here: Very interesting design pattern, especially for my case.

Asynchronous Request-Reply pattern:



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store