long banner inside article post image 6

Greenfeed - data optimization solution with AWS

Established in 2003, Greenfeed's mission is to provide consumers with a clean food chain from farm to table with the 3F Plus process - FARM FEED FOOD. During the time operating the process, Renova Cloud is honored to cooperate and accompany with Greenfeed to provide AWS solutions.

web 610x400 11

INDUSTRY:

Agriculture

technology:

AWS

CUSTOMER CHALLENGE

GreenFeed produces and sells agricultural products in a large scale in multiple South East Asian countries. To keep track of the sales and performance of the company, the data stored in various locations and services needs to be combined in reports.

In the past, to make daily sales reports, GreenFeed used the templates on Excel and collected information from various data sources such as: SAP system, Solomon ERP system at factories in Vietnam, Laos, Cambodia, Myanmar and from many excel files.

Because the data is disintegrated and was managed by different departments, it took a lot of time and had to work manually step by step to collect information and complete a synthesis report.

                                                                                                                 lWdRkMsERzdrmrIaw 0G44nvAqVJzGCOgnh2B62bYiP1fIg 2HZfQ5OiP2Vfv3RqhvWZ XxjV2 PslofQbLVfrtlDchj6sB1m3WMxRfJxr4wIEQdhuKHDgw0JxXlEtyul PYYFSckgeA5k CA

Following the current trend, when data becomes one of the biggest and most valuable assets of enterprises, the methods of storing and analyzing data can influence business decisions. Advanced analytics and business intelligence (BI) reporting tools are the basis which organizations are applying to promote the development in current competitive economy. Therefore, GreenFeed has the following requirements:

  • Build a single data lake, which can store all data from all different sources to support for making reports:  SAP system, Solomon system, Excel files.
  • Build an automated reporting solution that will automatically update the latest data every day and create a daily sales revenue report. Directorate and users will access through the web portal or mobile phone to view the reports.
  • Data lake to have unlimited storage space and can store data from other source data in the future such as other databases, external source data (from partners, network social),…

SOLUTION

The key questions are:

  • How can we extract data from the SAP system?
  • How can we extract data from the Solomon system with all foreign sites?
  • How will the data from Excel affect reports?
  • How can we automate the process?
  • And how can the solution be native to AWS so that it will be easy for customers to use?

First, to be able to extract data from sources to AWS, we will need to establish a secure connection because we cannot push the data out to the internet for such extraction, which is very dangerous. So we will use AWS’s Site to Site VPN to create VPN routes  from the AWS network down to GreenFeed’s office, as well as in foreign sites.

Regarding data extraction, we can make use of AWS DMS (Database Migration Service) service, but unfortunately, DMS does not support SAP and GreenFeed’s Solomon SQL Server is an old version which DMS does not support. So we can only build a script to extract the data and Python will be the main language.

For SAP source, we create applications using SAM CLI. By using SAM CLI, we only need to take care of our coding, SAM will help us deploy to Lambda Function AWS. With SAP source, we will divide into 2 sources and extract as follows:

  • Data for Dimension Tables
    • With the data for Dimension, we will make a call to SAP’s Odata API to fully load the data into S3 as csv file. Then we will COPY data from S3 into Redshift.
  • Data for Data Table
    • With data for Fact Table, we will also call SAP’s Odata API to load historical data first, then load daily incremental data put it in S3. From S3 we will also COPY the data into Redshift

For the Solomon source, we make a Python Script to extract the data and put it into S3. Because Solomon’s data is quite small, we can use Redshift Spectrum to query data from Redshift to S3 without the need of putting data from S3 into Redshift. For more convenience in querying, we will have a Glue Job to adjust the Struct Type of the entire table and convert it to Parquet format.

For Excel, we upload directly to S3 and load into Redshift using the COPY Command.

Next, we will automatic above steps:

  • For SAP, we will put Lambda Functions created from SAM CLI into a Step Function for each data type: Data for Dimension, Data for Fact. With Fact data, Step Function will have an extra step of getting the current date and running a stored procedure to load data into Redshift incrementally. With Dimension Data Step Function, we will load fully into S3 and then Evenbridge will trigger a query to run Storing Procedure to load Dimension data into Redshift.
  • For Solomon SQL Server, the script is located in an EC2 server, so we will use the Window Scheduler to schedule the data extraction. AWS Glue can schedule itself to convert back to parquet with the new data.
  • For Excel, we also use EventBridge to trigger stored procedures to load data into Redshift.
  • After the data from the sources has been loaded into Redshift, we will trigger the Store Procedures to load the data from Redshift’s Staging into the Data warehouse
diagram

BENEFITS

  • The data lake/data warehouse system on AWS and the reports on the Power BI platform have brought great benefits to GF such as:
  • A data lake system can store all structured and unstructured data. AWS tools/services will support GF to ETL different data sources to data lake.
  • Easily view automated BI reports. The data ETL processes have been set up to flow automatically to the data lake every day, then Power BI will automatically connect and build the daily sales reports which is required by the GF (the picture below is an example of the sales report on mobile interface)
  • The scalability of the data lake system can fully meet GF’s future advanced analysis requirements, as well as apply AI/ML technologies to make important reports and decisions of the enterprise’s business activities.

                                                                                                                   gNM 4oN7DagElgmxolCWp7PO 337PE5lK aybxMb8HIQqK6qdT9ae7biRvxnmOTSm2afvQDQLZ knwWCb 3vBk qBExI8EZNMkgz0tDNo9 QywqkNpjr5 aofaXew7fr6ZG uWn6WvWs72pvg

RELATED STORIES

6 1 7
Cloud Reselling White 1

Ngân hàng Thương mại Cổ phần

Joint Stock Commercial Bank successfully accelerated growth by migrating to AWS

The competitive digital race among banks is increasingly fierce, focusing on attracting technology-savvy customers. The commercial bank wanted to stay ahead of the competition and required scalable, reliable and cost-effective back-end infrastructure for rapid growth.Read more>
IMV web 610x400 1
Cloud Reselling White 1

IMV

IMV’s Business Intelligence Empowered by AWS Data Management Solution

The former infrastructure of IMV lacked the scalability required to accommodate the growing volumes of data and the increasing demand for business intelligence and analytics. Read more>
web 610x400 5
Cloud Reselling White 1

E-wallet

Vietnam’s Prominent E-wallet Migrated Its Two Most Critical Workloads to AWS

The E-wallet in question is one of Vietnam’s largest online payment platforms, developed and published by one of the most dominant technology players in the region. Read more>
web 610x400 8
Cloud Reselling White 1

F88

F88’s Journey to a Secure and Centralized Data Management with AWS Cloud Solutions

Established in 2013, F88 stands as the forerunner network of secure lending in Vietnam, focusing on serving micro & small enterprises as well as individual customers. Receiving substantial financial support from renowned global Read more>
web 610x400 12
Cloud Reselling White 1

Greenfeed

Greenfeed Data optimization solution with AWS

Greenfeed's mission is to provide consumers with a clean food chain from farm to table with the 3F Plus process - FARM FEED FOOD. Read more>
web 610x400 3
DevOps Transform White 1

Startup oi

RENOVA – STARTUP OI – SUCCESS STORY

Startup Oi is a new social media and tech talent platform. Startup Oi has an ambitious goal to focus on professional needs of tech engineers and developers, building a digital community and connecting with job opportunities and tech events. Read more>
web 610x400 5
Cloud Reselling White 1

Vietcetera

VIETCETERA – MIGRATION JOURNEY TO AWS

Founded in 2016, Vietcetera is a first-of-its-kind multimedia digital platform producing editorial content along with video series and podcasts in both Vietnamese and English.Read more>
imv photo
Cloud Reselling White 1

IMV

IMV – ON-PREMISE TO CLOUD JOURNEY

International Minh Viet Joint Stock Co. is a distribution company which entails general trade, modern trade, as well as an e-commerce channel. Read more>
smartpay illustration photo
Cloud Reselling White 1

Smartpay

SmartPay Rehost and Replatform Migration on AWS

SmartPay is an important contender in the Vietnamese payment apps landscape and experiencing growth with new vendors, partners, and users. Read more>
pepsico case study product scaled
Cloud Reselling White 1

Pepsi Co Myanmar

Accelerate SAP S/4HANA Transformation With AWS

Pepsi Co Myanmar was looking to modernize their existing systems and launch a new platform for their users and applications utilizing the up-to-date technology solutions and choose SAP S/4HANA on AWS. Read more>
engineers working photo
Cloud Reselling White 1

Nutifood Group

AWS Immersion Day with Nutifood Group

The goal of this educational workshop was to empower N Group's IT team and leadership to confidently use the AWS platform for their benefit, and unlock new business opportunitiesRead more>
photo 1571247865791 9d7ed2ddf033
Cloud Reselling White 1

Tadiran

Serverless application implementation on AWS for Tadiran Group

Tadiran is a leading technical and electrical appliances manufacturer company. Some of the company’s more famous product lines include batteries and air conditioners. Read more>
webinar blog banner
Cloud Reselling White 1

Scratchpay

Scratchpay Security best practices and well-built architecture on GCP

Evaluation and assessment of the existing GCP environment for best practices and well-built architecture methodologiesRead more>
UAB CLOUD NATIVE APPS 1
uab white 1

UAB - Connect. Create. Change.

UAB BANK CLOUD NATIVE APPS

As a part of the modernisation and digitalisation strategy, UAB is expanding its services to build mobile apps for payment and e-Wallet purposes to serve the growing Myanmar consumer banking marketRead more>
YOMA MIGRATION MANAGED SERVICES 1
yoma white 2

Yoma

Yoma Transformation to AWS Cloud

The increasing demand for Yoma’s applications and business across different verticals requires to have a digital transformation journey with an effective migration to the cloud.Read more>
AQUA AUTOMATION DEVOPS 1
aqua white 1

AQUA

AQUA – Migration & CI/CD

AQUA is able to improve operations as result of the migration and CI/CD on AWS as well as able to release new features faster with minimal time and disruption to the end users.Read more>
SOVIGAZ PROTECTS CRITICAL SYSTEMS ON AWS 1
sovigaz white 2

SOVIGAZ

Sovigaz – migrating to AWS

The agility of the new infrastructure help Sovigaz to relieve the burden of on premise infrastructure management and limitations as well as access to advanced services offered on AWS platform. Read more>
KAOPIZ AUTOMATION CI CD 1
kaopiz white 2

KAOPIZ

Kaopiz’s Infrastructure Automation

Automate the processes on AWS and create an environment to test new features to support the Development & QA team to deliver jobs faster, automated and without a deep level of experience and skills on AWS.Read more>
NAVIGOS MIGRATION BACKUP 1
navigos white 2

NAVIGOS

Navigos’ journey to the Cloud

As part of the company commitment to it’s customers, it is essential to secure a robust disaster recovery process. Navigos’ application requires to have a fast and effective reaction in the disaster recovery scenarios.Read more>
SABECO MIGRATION WELL ARCHITECTED 1
sabeco white 1

SABECO

Sabeco Case Study

Having migrated to AWS, Sabeco is capable of quickly responding to increased traffic and services’ usage when necessary, making running marketing campaigns and launching new features less risky.Read more>
Nkid case study image 1
Nkid white 1

NKID’S

Nkid’s journey to the Cloud

Nkid Group is running a number of Microsoft workloads in various environments including on-premises, local data center in Vietnam, Microsoft Azure and AWS.Read more>