Data Engineering
This page highlights a diverse portfolio of projects I've completed as a Data Engineer. It showcases my expertise in leveraging data languages like SQL, Python, and Spark to perform all stages of the ETL process, culminating in data visualization for actionable insights.
Project 1
In this project, I served as a Data engineer for a betting company which wanted to implement AI into their data. Therefore the code snippets below are Python Code which connects to a Ai source, configures its settings and trains to query a SQL database containing football data.
Dataflow:
Data is Extracted from a SQL Database
Connected to Open AI
Trained AI to understand the SQL Database (using SQL) and how queries should be structured
Web hosted interface was created using “Gradio” so that non-technical users can simply use.
Project 2
In this project, I worked as a Data Engineer, connecting to a cryptocurrency API to extract data. Using Notebooks in Microsoft Fabric I leverage Python to identify the top 3 gainers, color-coded them with HTML, attached the latest news about the coin and sent the results via email.
Dataflow:
CRYTPO Data is Extracted from a API
Transformed using python to find the top 3
HTML Colour code a email
Since I was using Microsoft Fabric Notebook I scheduled the notebook to run every week.
Project 3
In this project, I served as a Data Engineer for a restaurant company, building a web-based interface to integrate OpenAI for seamless, real-time interaction and insights from their business data.
Dataflow:
Data is Extracted from a SQL Database
Connected to Open AI
Trained AI to understand the SQL Database (using SQL) and how queries should be structured
Web hosted interface was created using “Gradio” so that non-technical users can simply use.
Project 4
In this project, I web-scraped data from a clothing website to identify products matching specific parameters. The goal was to notify users via email about products classified as "deals." Below are code snippets demonstrating the implementation.
Dataflow:
Data is Extracted by web scrapping using “Beutiful soup”
Transformed by Python code
HTML coded to be sent to email
