eCloudvalley implements a new data pipeline and improves the process to be more streamlined. We are able to bring data to use at a fast speed. The glue job required 37 minutes to execute in the past, and now it completes in less than 7 minutes.
About Red Dot Payment
Founded by a group of visionary financial technology payment experts in 2011, Red Dot Payment has grown into a trusted online payment company providing premium payment solutions and expertise to the brightest merchants across the Asia Pacific.
Red Dot Payment is headquartered in Singapore with offices in Thailand and Indonesia. Many of their customers come from hotels, consumer services, and regional e-commerce. Red Dot Payment provides payment gateway services, establishes connections with merchants, and deals with all payment requirements such as Visa, Mastercard, and e-wallets, such as PayPal, WeChat, Alipay.
Red Dot Payment’s Challenge
As the first step in the online payment process, Red Dot Payment plays a crucial role in helping customers more easily accept online transactions. The speed of transactional data transformation positively influences customer satisfaction. However, they met the following challenges:
- Facing pressure from slow operating performance in Extract-Transform-Load (ETL) transformation. For example, it takes 37 minutes for running a job
- Slow transaction speed from the payment gateway to bank or acquirer caused the revenue loss from end-users because of the inefficient payment process
- Long duration for report generation due to waiting for ETL transformation, Red Dot Payment’s customer could not start the in-time analysis.
- Requires elimination of unnecessary costs for the business to focus on product innovations and the creation of value-added services for their customers
- In need of streamlining IT infrastructure for the payment gateway platform in a cost-effective way
Solutions from eCloudvalley
- Replaced Amazon Elastic MapReduce (EMR) instead of Glue to conduct ETL.
- Utilized Apache Airflow as workflow management tools
- Optimized current Glue script to improve running performance
- Developed 4 to 5 Directed Acyclic Graphs (DAGs) for data pipeline and orchestration.
- The previous glue script is optimized, and it helps Red Dot Payment reduce 75% time of transactional data transformation by leveraging the solution eCloudvalley provided.
- Red Dot Payment saves up to 98.23% of the cost compared to the previous infrastructure.
- With Apache Airflow managed orchestration service, Red Dot Payment can easily schedule and monitor sequences of processes and tasks. For example, engineers are able to see the dependency in all the valueless glue through a straightforward interface.
- With the new data ingestion framework, managed workflows automatically scale their workflow execution capacity to meet the needs. Engineers can follow this set of services without having to spin off any instance and deploy the software.
- The solution speeds up the transaction process from the payment gateway to banks or acquirers. Higher revenue and customer satisfaction are achieved in terms of an efficient and secure payment process.
- Red Dot Payment and their customers can monitor transactions by real-time dashboards.