Currently, the issue of optimization is one of the most difficult challenges faced by a programmer. This is due to the ever-increasing amount of data. The problem is not just providing them, but specifying the relevant information that we analyze. Even more complicated is the provision of data in the so-called “on the fly”, which means analysis while information is being processed or retrieved from them. All this to obtain what is important to the user, e.g. financial analyses, resource charts or current production events. The time of data delivery increases to such an extent that the application is often unusable, and maintaining it on its own resources becomes unprofitable. It is necessary to analyze and use tools that will facilitate this process enough for the data to be useful again.
Let’s assume we have a system that collects information about prices and products from many distributed systems via API. This causes significant delays on the website that is responsible for providing and presenting this data to the user. Extended data access time causes the user to resign from using the system. It is therefore necessary to introduce appropriate optimization and preventive mechanisms.
One of the mechanisms may be the use of data recording on the side of the data access server (a server that returns data directly to the user) and periodic querying of external systems (e.g. via the mentioned API) for the necessary data and updating them on an ongoing basis. This solution enables immediate access to data, which directly affects the comfort of working in the system. Of course, the entire process should be preceded by an architecture analysis, appropriate calculations and a check of available resources.
Analyzes and calculations are an important step to using appropriate resources such as technology, server memory or type of communication mechanisms. The database itself is also crucial during optimization, and above all, its appropriate selection for the processed/stored data. I often encountered an inappropriately selected database, which caused later difficulties. To avoid such situations, it is important to understand the difference between SQL and NOSQL databases.
The SQL database is a relational database where data and tables have a certain relationship. For example – a database of a company selling through its representatives. Let’s assume we have three tables: Company, Representative, Product. The representative is an employee of the company, so we have a one-to-one relationship, where we have a key connection and we build an appropriate relationship. Then this representative has many products that he offers. This means a one-to-many relationship.
The NOSQL database is a document database. I will also use an example here. Let’s assume we have a company with a complaints system. Each complaint is unique and receives a number and date of receipt. It may also contain information about the person who entered it, but this is often variable because the company has a high employee turnover. We register the complaint and no other connection is important to us. This is a document that we keep. Of course, the NOSQL database makes it possible to link documents, e.g. by number, but it is also a specific way of using it, e.g. PZ and WZ documents, which we link with the order number.
Sebastian Wąsik
