Pujan Dahal

• Data Ingestion: Designed and implemented system for daily email ingestion into AWS S3.
• Data Processing: Used medallion architecture in AWS to clean and process emails with Spark and AWS Glue scripts for scalable data handling.
• Prompt Engineering: Engineered prompts for LLMs to extract keywords and build lexicons from emails and web articles.
• Data Storage: Worked with MongoDB for storing and managing extracted lexicon data.

For internal departments,

• Data Integration: Built pipelines and scripts to ingest data from SQL databases, CRM, APIs, and Google Sheets into AWS S3.
• Data Modeling and Architecture: Designed medallion architecture for scalable data pipelines, ensuring efficient ETL and data integrity.
• Data Processing: Used Spark with AWS Glue for data cleaning, and transformation, and AWS Athena for complex querying.
• Data Analysis: Created Apache Superset, and Google Looker Studio dashboards for monitoring and insights.
• Data Analysis: Designed metrics for hiring process, and CRM, reducing time and costs significantly.

For a government client,

• Data Integration: Built pipelines to ingest SQL data into Fabric Lakehouse using medallion architecture and Fabric Notebooks.
• Data Modeling: Created optimized data models based on existing client models to streamline API calls.
• Data Quality: Developed data quality framework for completeness, consistency, and uniqueness, ensuring high data standards.
• Real-Time Data Processing: Leveraged Fabric Event Stream for real-time API integration with Azure SQL Database CDC.
• Expertise in Azure ADLS, Azure Functions, Data Factory, Fabric Notebooks, KQL, Warehouse, and Lakehouse.