Hardware and Software Projects
Mini Places Challenge - Image Classification: The goal of this challenge was to identify the scene category depicted in a photograph. The data for this task comes from the Places2 dataset which contains 10+ million images belonging to 400+ unique scene categories. Specifically, the mini challenge data for this consisted of 100,000 images for training, 10,000 images for validation and 10,000 images for testing coming from 100 scene categories. The images were resized to 128*128 to make the data more manageable. Further, while the end goal is scene recognition, a subset of the data contained object labels for building better models. For each image, algorithms will produce a list of at most 5 scene categories in descending order of confidence. The quality of a labeling will be evaluated based on the label that best matches the ground truth label for the image. The idea is to allow an algorithm to identify multiple scene categories in an image given that many environments have multi-labels (e.g. a bar can also be a restaurant) and that humans often describe a place using different words (e.g. forest path, forest, woods). [Additional info here]
Our entry to the 2016 Miniplaces challenge used an ensemble of residual networks. We combined the outputs of three of these CNNs by averaging the outputs of the final softmax layer. Incorporating the outputs of the final 1000- dimensional layer of a state-of-the-art object recognition network trained on the ImageNet database was key to the success of our system. The softmax outputs of the scene and object networks were used as features for a Random Forest classifier resulting in a significant increase in accuracy. To combat overfitting due to the small size of the Miniplaces dataset, we implemented additional data augmentation: we added random rotations, translations, shear transforms and zooms. Our final system achieved a Top-1 error of 0.4397 and Top-5 error of 0.1644 on the Miniplaces test set. For additional information, read the report here.
JetsonBot: The majority of machine learning and AI projects available for hobbyists are strictly software based and hosted primarily in the cloud. I wanted a tangible, inexpensive, and easily modifiable platform through which I could explore vision-based robotics algorithms (e.g. object recognition, SLAM and navigation, human interaction) that would otherwise not be possible if working strictly on web apps or cloud applications. Thus, I built a Jetsonbot, which is an inexpensive and small indoor robot with enough compute for running online machine learning implementations. The Jetsonbot is built on top of an iRobot Create base for movement. The main computer is an integrated CPU/GPU Nvidia Jetson TX1. In addition to the bump and drop sensors available in the Create base, the robot is outfitted with an RP Lidar unit, and an Orbbec infrared depth camera. The plastic platforms allow for the robot to be modified with additional sensors and other peripherals (e.g. a platform for holding food or other items useful for humans).
The code is available here.
Self-Balancing Bot: A two-wheeled self balancing robot that can be piloted via remote control. The 3D printed PLA chassis is built in three sections in order to fully enclose the electronics, motors, and battery. This Arduino based inverted pendulum robot uses an MPU-6050 gyro/accelerometer for its self-leveling feedback. It uses two brushed DC motors with 34:1 gear reduction for increased torque. The attached encoder units can also optionally be used in a secondary feedback loop for greater leveling control. Additionally, a Wii Nunchuck was modified with the addition of an Arduino Micro and a Bluetooth module in order to serve as a self-contained wireless controller to pilot the robot.
The goal of this project was to explore Proportional-Integral-Derivative control loops through a relatively inexpensive (<$150) robot. The code is available here.
Waiter Robot: The goal for this robot was to create an automated waiter that served drinks at a party. The robot would wander the party offering guests the opportunity to place an order. After collecting orders, the robot would return to the bar and submit the order list to the bartender, who would load up the robot’s tray with the proper drinks. The robot would then return to each of the guests delivering the proper drinks.
The waiter robot was built upon the Turtlebot platform. A Turtlebot Roomba base included all the mobility hardware, as well as bump sensing capability. An Asus Xtion Pro camera provided infrared (IR) imaging capability, which was used for non-contact object detection and avoidance. A Logitech C920 HD webcam mounted at the top of the Turtlebot frame served as the primary sensor for detecting people. A netbook running Ubuntu was used as the brains of the robot. The Robot Operating System (ROS) was used in order to compartmentalize all of the sensors and actions that the robot was capable of, and a combination of OpenCV and the ar_track_alvar ROS package were used for environment detection.
We made several mechanical alterations to our robot to make it more suited to its task. We also included a vacuum molded mask constructed using a 3D scan of John Harvard’s head. The speaker was placed behind the John Harvard mask in order to give the impression that the robot was actually talking.
For additional information, read the report here.
Robotic Localization and Mapping: Robot localization and mapping is an on-line filtering problem within the field of Artificial Intelligence that utilizes sensor information to approximate the current state of the robot, given an initial state estimate and a sequence of observations. A popular algorithm is particle filter localization (aka Monte Carlo localization), which is a method for sifting through a set of states in a Hidden Markov Model to find the best state estimate. Given a set of observations, a particle filter estimates the posterior density of the state variables. We chose to use this method in order to simulate a robot localizing itself in Harvard engineering buildings. We implemented a particle self-localization procedure, where a robot has knowledge of the map layout but no knowledge of its location within this layout. We also implemented the start of a SLAM procedure, where the robot has no knowledge of the map layout, but uses its sensors and odometry information to construct a perceived map.
The primary goal of this project was to implement a Monte Carlo localization based procedure that allows our simulated robot to estimate its location within any floor plan. Given an image of the map layout, the procedure would allow the robot to estimate its position on the map based on information that it gathers from moving around and sensing its environment. Additionally, we wanted to extend this functionality to include mapping. If the robot were denied access to the map layout should be able to use SLAM procedure in order to generate a perceived map of the walls and obstacles that it comes across. For additional information, read the report here.
A Sensor System for Autonomous UAV Landing: Unmanned Aerial Vehicles (UAVs) have recently surged in popularity and are now seen as viable tools for use in commercial delivery, search and rescue operations, and planetary exploration. A challenge in utilizing UAVs for these purposes is the limited battery capacity: an average-sized drone has 25 minutes of flight time, which results in limited flight range and payload capacity. Implementing an array of charging stations would extend UAV flight range, but GPS navigation would not provide the resolution necessary to reliably find and land on these stations. This project focuses on developing a computer vision-based sensor system that would provide onboard sensing mechanisms and path planning in order to accurately detect and land on a charging pad. This solution aims to be lightweight, unobtrusive, self-contained, and more accurate than relying on GPS and IMU alone. This proof of concept sensor system can be adapted to the form factor of mission-specific UAVs.
For additional information, read my undergraduate senior thesis here.