1. Configured and Implemented Full Data Engineering and Analysis pipeline: - Implemented asynchronous tasks with Celery to upload incoming sensor data to a data lake, ensuring timely and accurate data storage. - Performed data quality assessments to validate dataset accuracy and reliability, ensuring integrity in subsequent processing and analysis. - Integrated validated datasets into a centralized data warehouse, simplifying access for analytics and reporting. - Automated the whole process using Apache Airflow, resulting in a 20% increase in performance. - Configured Airflow alerts and monitoring tools to provide real-time notifications on workflow status. 2. Statistical analysis, Data analysis and Machine learning model implementation: - Analyzed 1M+ rows of data from about 300+ users daily. - Developed an algorithm capable of accurately detecting complex driving maneuvers, including u-turns, rapid turns, overtaking, and lane changes. - Integrated the algorithm into the existing data processing pipeline, ensuring seamless operation and real-time behavior analysis. - Together with the team, utilized the algorithm’s outputs and implemented a driving behavior scoring algorithm for usage-based insurance, significantly enhancing road safety. - Developed a supervised SVM machine learning model with Fourier and wavelet transform preprocessing, achieving a 92% accuracy rate in distinguishing between road and sidewalk classes. - Communicated real-time findings by utilizing Python’s Streamlit and R-Shiny. - Utilized Tableau for scorecards dashboard.

더보기

- Conducted market research to analyze target market, customer behavior, and industry trends, supporting business operations. - Utilized BeautifulSoup for data extraction from URLs, ensuring cleanliness and consistency, and stored data in a data lake. - Collaborated with the development team to design and implement new databases and tables, enhancing data management processes. - Communicated findings and insights with stakeholders to inform strategic decisions.

더보기