Greenfeed - data optimization solution with AWS

Established in 2003, Greenfeed's mission is to provide consumers with a clean food chain from farm to table with the 3F Plus process - FARM FEED FOOD. During the time operating the process, Renova Cloud is honored to cooperate and accompany with Greenfeed to provide AWS solutions.

INDUSTRY:

Agriculture

technology:

AWS

CUSTOMER CHALLENGE

GreenFeed produces and sells agricultural products in a large scale in multiple South East Asian countries. To keep track of the sales and performance of the company, the data stored in various locations and services needs to be combined in reports.

In the past, to make daily sales reports, GreenFeed used the templates on Excel and collected information from various data sources such as: SAP system, Solomon ERP system at factories in Vietnam, Laos, Cambodia, Myanmar and from many excel files.

Because the data is disintegrated and was managed by different departments, it took a lot of time and had to work manually step by step to collect information and complete a synthesis report.

                                                                                                                 

Following the current trend, when data becomes one of the biggest and most valuable assets of enterprises, the methods of storing and analyzing data can influence business decisions. Advanced analytics and business intelligence (BI) reporting tools are the basis which organizations are applying to promote the development in current competitive economy. Therefore, GreenFeed has the following requirements:

  • Build a single data lake, which can store all data from all different sources to support for making reports:  SAP system, Solomon system, Excel files.
  • Build an automated reporting solution that will automatically update the latest data every day and create a daily sales revenue report. Directorate and users will access through the web portal or mobile phone to view the reports.
  • Data lake to have unlimited storage space and can store data from other source data in the future such as other databases, external source data (from partners, network social),…

SOLUTION

The key questions are:

  • How can we extract data from the SAP system?
  • How can we extract data from the Solomon system with all foreign sites?
  • How will the data from Excel affect reports?
  • How can we automate the process?
  • And how can the solution be native to AWS so that it will be easy for customers to use?

First, to be able to extract data from sources to AWS, we will need to establish a secure connection because we cannot push the data out to the internet for such extraction, which is very dangerous. So we will use AWS’s Site to Site VPN to create VPN routes  from the AWS network down to GreenFeed’s office, as well as in foreign sites.

Regarding data extraction, we can make use of AWS DMS (Database Migration Service) service, but unfortunately, DMS does not support SAP and GreenFeed’s Solomon SQL Server is an old version which DMS does not support. So we can only build a script to extract the data and Python will be the main language.

For SAP source, we create applications using SAM CLI. By using SAM CLI, we only need to take care of our coding, SAM will help us deploy to Lambda Function AWS. With SAP source, we will divide into 2 sources and extract as follows:

  • Data for Dimension Tables
    • With the data for Dimension, we will make a call to SAP’s Odata API to fully load the data into S3 as csv file. Then we will COPY data from S3 into Redshift.
  • Data for Data Table
    • With data for Fact Table, we will also call SAP’s Odata API to load historical data first, then load daily incremental data put it in S3. From S3 we will also COPY the data into Redshift

For the Solomon source, we make a Python Script to extract the data and put it into S3. Because Solomon’s data is quite small, we can use Redshift Spectrum to query data from Redshift to S3 without the need of putting data from S3 into Redshift. For more convenience in querying, we will have a Glue Job to adjust the Struct Type of the entire table and convert it to Parquet format.

For Excel, we upload directly to S3 and load into Redshift using the COPY Command.

Next, we will automatic above steps:

  • For SAP, we will put Lambda Functions created from SAM CLI into a Step Function for each data type: Data for Dimension, Data for Fact. With Fact data, Step Function will have an extra step of getting the current date and running a stored procedure to load data into Redshift incrementally. With Dimension Data Step Function, we will load fully into S3 and then Evenbridge will trigger a query to run Storing Procedure to load Dimension data into Redshift.
  • For Solomon SQL Server, the script is located in an EC2 server, so we will use the Window Scheduler to schedule the data extraction. AWS Glue can schedule itself to convert back to parquet with the new data.
  • For Excel, we also use EventBridge to trigger stored procedures to load data into Redshift.
  • After the data from the sources has been loaded into Redshift, we will trigger the Store Procedures to load the data from Redshift’s Staging into the Data warehouse

BENEFITS

  • The data lake/data warehouse system on AWS and the reports on the Power BI platform have brought great benefits to GF such as:
  • A data lake system can store all structured and unstructured data. AWS tools/services will support GF to ETL different data sources to data lake.
  • Easily view automated BI reports. The data ETL processes have been set up to flow automatically to the data lake every day, then Power BI will automatically connect and build the daily sales reports which is required by the GF (the picture below is an example of the sales report on mobile interface)
  • The scalability of the data lake system can fully meet GF’s future advanced analysis requirements, as well as apply AI/ML technologies to make important reports and decisions of the enterprise’s business activities.

                                                                                                                   

RELATED STORIES

Greenfeed

Greenfeed Data optimization solution with AWS

Greenfeed's mission is to provide consumers with a clean food chain from farm to table with the 3F Plus process - FARM FEED FOOD. Read more>

Startup oi

RENOVA – STARTUP OI – SUCCESS STORY

Startup Oi is a new social media and tech talent platform. Startup Oi has an ambitious goal to focus on professional needs of tech engineers and developers, building a digital community and connecting with job opportunities and tech events. Read more>

Vietcetera

VIETCETERA – MIGRATION JOURNEY TO AWS

Founded in 2016, Vietcetera is a first-of-its-kind multimedia digital platform producing editorial content along with video series and podcasts in both Vietnamese and English.Read more>

IMV

IMV – ON-PREMISE TO CLOUD JOURNEY

International Minh Viet Joint Stock Co. is a distribution company which entails general trade, modern trade, as well as an e-commerce channel. Read more>

Smartpay

SmartPay Rehost and Replatform Migration on AWS

SmartPay is an important contender in the Vietnamese payment apps landscape and experiencing growth with new vendors, partners, and users. Read more>

Pepsi Co Myanmar

Accelerate SAP S/4HANA Transformation With AWS

Pepsi Co Myanmar was looking to modernize their existing systems and launch a new platform for their users and applications utilizing the up-to-date technology solutions and choose SAP S/4HANA on AWS. Read more>

N Group

AWS Immersion Day (N Group)

The goal of this educational workshop was to empower N Group's IT team and leadership to confidently use the AWS platform for their benefit, and unlock new business opportunitiesRead more>

Tadiran

Serverless application implementation on AWS

Tadiran is a leading technical and electrical appliances manufacturer company. Some of the company’s more famous product lines include batteries and air conditioners. Read more>

Scratchpay

Scratchpay - Financial services

Evaluation and assessment of the existing GCP environment for best practices and well-built architecture methodologiesRead more>

UAB - Connect. Create. Change.

UAB BANK CLOUD NATIVE APPS

As a part of the modernisation and digitalisation strategy, UAB is expanding its services to build mobile apps for payment and e-Wallet purposes to serve the growing Myanmar consumer banking marketRead more>

Yoma

YOMA STRATEGIC HOLDINGS

The increasing demand for Yoma’s applications and business across different verticals requires to have a digital transformation journey with an effective migration to the cloud.Read more>

AQUA

DỊCH CHUYỂN & CI/CD

AQUA is able to improve operations as result of the migration and CI/CD on AWS as well as able to release new features faster with minimal time and disruption to the end users.Read more>

Kymdan

Kymdan- Hành trình dịch chuyển Microsoft workloads lên AWS

Automate the processes on AWS and create an environment to test new features to support the Development & QA team to deliver jobs faster, automated and without a deep level of experience and skills on AWS.Read more>

SOVIGAZ

SOVIGAZ HÀNH TRÌNH DỊCH CHUYỂN SANG AWS

The agility of the new infrastructure help Sovigaz to relieve the burden of on premise infrastructure management and limitations as well as access to advanced services offered on AWS platform. Read more>

NAVIGOS

NAVIGOS GROUP VỚI HÀNH TRÌNH DỊCH CHUYỂN & SAO LƯU

As part of the company commitment to it’s customers, it is essential to secure a robust disaster recovery process. Navigos’ application requires to have a fast and effective reaction in the disaster recovery scenarios.Read more>

KAOPIZ

KAOPIZ SOFTWARE TỰ ĐỘNG HÓA & CI/CD

Automate the processes on AWS and create an environment to test new features to support the Development & QA team to deliver jobs faster, automated and without a deep level of experience and skills on AWS.Read more>

SABECO

SABECO - DỊCH CHUYỂN VÀ TỐI ƯU HÓA KIẾN TRÚC HẠ TẦNG

Having migrated to AWS, Sabeco is capable of quickly responding to increased traffic and services’ usage when necessary, making running marketing campaigns and launching new features less risky.Read more>

NKID’S

Nkid’s journey to the Cloud

Nkid Group is running a number of Microsoft workloads in various environments including on-premises, local data center in Vietnam, Microsoft Azure and AWS.Read more>