Initial Phase of Implementing Airbyte

Context

This blog post is about the thought process behind our first steps towards implementing Airbyte. We have yet to implement it, as we are still exploring whether Airbyte is a good fit within our tech stack.

The Discovery Process

One of the tasks assigned to me was writing a ticket regarding the implementation of Airbyte as part of the company’s data ingestion pipeline. Initially, this task raised many questions in my mind. What exactly is Airbyte? Why does the company want to use it? Why not continue using the current pipelines? What does the current pipeline even look like?

The Data Ingestion Pipeline

Rather than immediately jumping into writing the ticket, I realised that I first needed to properly understand the problem and the surrounding context. To answer these questions, I began researching independently. I watched demo videos to understand how Airbyte works and explored the different connectors available on the platform. At the same time, I went through Vatico’s internal slides and diagrams to better understand the company’s business operations and existing data architecture.

With the help of the company’s in-house chatbot and internal documentation, I was gradually able to piece together how the current data pipeline operates and why the company is exploring a more standardised ingestion solution. Through this process, I also gained a better appreciation of how data engineering decisions are closely tied to operational scalability and maintainability.

Problem Solving

This experience reinforced an important lesson about problem-solving, especially when dealing with unfamiliar systems or technologies. Asking questions is extremely important. Instead of assuming understanding, it is often more effective to break the problem down systematically. Personally, I like to start with the “5Ws and 1H” approach — Who, What, When, Where, Why, and How. This framework helps me structure my thinking, identify knowledge gaps, and build a clearer understanding before attempting to propose solutions or write technical documentation.