Developed Hadoop eco-system
- Component system: HDFS, Flume, Solr, Zookeeper, HBase, Redis, MySQL, Kafka, Thrift, Cloudera manager, Avro, GlusterFS, Sqoop, Oozie, OpenTSDB
Developed data acquisition system and parser for manufacturing equipment
- Developed high performance data collector via FTP written in C (support 2,000 server connection concurrently per 1 collector server)
- Developed data parser using Morphlines
- Developed work flow notification system using Kafka
- Developed build script and run script
- Developed test code
Developed NoSQL DB system
- Developed HBase schema to enhance insert/search performance
- Developed data compression in HBase
- Developed meta data storage using MySQL
- Developed cache system using Redis
- Developed data selection service using Thrift
- Developed HBase test code
Set up Hadoop eco-system server cluster
Developed monitoring & management system architecture and developed
- Developed clustering architecture including leader election system
- Developed RESTful API(message protocol) via WebSocket
- Developed configuration management system
- Developed watchdog system
- Developed failover system
- Developed asynchronous logging and searching system
- Developed user interface using Bootstrap
