Process for Creating and Close a New Data Model

I. First ticket stage

Clarify Questions

1. To ask the assigner any question from the request prompt that is not clear.
2. Ensure that you know 
   • what is the role of the model in the business flow 
   • what is the intended outcome
   • Which domain does this model belong to (ffm or warehouse etc…)
3. Also, if the problem are too complex, try to draw flow chart 
4. Do not refer to sql models in this stage

Write Ticket in 30 mins

1. Background: Provide a brief description of the current situation. For example, explain that there are orders in a ‘processing’ state that need to be updated to ‘cancelled’ because they couldn’t be delivered. Additionally, the shipment status of these orders in the CRM needs to be updated to ‘returned’ as the items have been sent back to the warehouse.
2. Objective (Business Sense): Outline the business objective, such as ensuring accurate and real-time updates in the CRM system to improve inventory management and provide timely updates to customers.
3. Deliverable: Define the expected deliverable, which in this case is a new data model that automates the status update of orders and shipments. Include name of models and its required columns
4. Action Item: List the required actions.

II. Refine Ticket

Update Details: 

1. Expand on the initial ticket with more detailed information about the requirements and scope and actions. 
2. Check with similar models at the same level.
3. Create substicket if required
4. Include specifics on the 
   • data model design: dimension (e.g order or order x items) and coverage (all time or monthly or last months or daily)
   • integration points with other applications (redefine data contract)
   • any constraints or considerations: for example: date_time format (e.g YYYY-MM-DD or DD/MM/YYYY) or status_name format (‘delivered’ or ‘DELIVERED’ or ‘to-be-shipped’ or ‘to_be_shipped’)
5. Use tables or sheet to show the intended output if needed
6. Need to be clear:
   • What is the source of truth of the info in the models
   • What is the dimension of all the models that this models is referring
   • When can we close the ticket?

Request Support from Relevant Departments: 

1. Identify and reach out to the departments that will provide necessary information or resources. 
   • Data contract: when creating API interface, request data contract from DE.
   • Staging tables: if staging is not updated, or missing staging data, you can request DE to re update or recheck. Ensure to use expectation vs reality with example
   • CRM: if requires extra info in CRM, request DE

III. Follow Action Items and Code

Develop the Data Model: 

1. Start coding based on the refined ticket. 
2. Create the new data model. 
3. Use correct column name
4. Write yaml, and make it easier to understand
5. For API interface, add data constraints

Testing: 

1. Referring with expected data created above and see if data is expected
2. Remember to test if the model is runnable on all test environments (bb_vn_test and va_vn_test).

IV. Create Merge Request and Resolve Merge Request Comments

1. Create Merge Request: Once the coding and testing are complete, create a merge request to integrate the new data model into the main codebase.
   1. Original ticket: ticket that direct this models
   2. Main ticket (if have): The ticket capture the main topic
   3. 1 liner about the changes
   4. Screenshot (if needed): pgadmin display of the new model
   5. Check for the changes to be as incremental as possible
   6. If any part of the code is unclear, comment on why you are doing so.
2. Resolve Merge Request Comments: Address any feedback or comments from code reviewers. 
   • Make necessary adjustments and improvements based on the feedback to ensure the data model meets all requirements and standards.
3. Merged

V. Work with orchestration

1. Create a ticket to direct DE to the model that is approved to integrate into orchestration.
2. Show an example of a row of data and expect outcome on the different systems (crm, shopee, lazada seller centres, etc…).
3. If there is an error with the model, work with DE to clear the error. Do not let DE handle the exceptions, all of them should now be handled at the T layer.
4. After DE finish integration, check the new changes in different system and see if it fix in with our expectation
5. If needed, wait for next instance that use this model and test if it work automatically
6. Close the ticket if the condition to close the ticket satisfied

 

Final word

This article follows the same principle that is mentioned in the training slides provided. There are 4 steps in solving a business problem:

• Scoping (1. And 2.)
• Wireframe (2.)
• Implement (3.)
• QC (3. And 4. And 5.)

Here, there is some additional focus on refining tickets (enhancing scoping and wireframe) and QC (the QC steps take extra time due to working with other departments).