In the News: Intelligent Scribbles Software Edits Out Images in Real Time

According to the Washington Post, only about 50 percent of new businesses are still open after four years and during 2013, ADSC proved that it deserves to be in that top group after four years of remarkable growth and significant accomplishments.

ADSC began in 2009 with a vision of bringing Illinois innovation to Singapore and has now grown to a world-class research lab with over 50 full-time researchers, 14 active Illinois faculty and over 100 published papers. With research in the fields of interactive digital media and the smart grid, ADSC is having an impact in areas such as computer vision, video tracking, image processing and analysis, as well as smart grid security and monitoring.

ADSC Research Scientist Jiangbo Lu demonstrates ADSC’s tele-immersive meeting technology, which helped lead to the development of the PatchMatch Filter.

Over the past year, Professor Minh Do’s team of researchers at ADSC have developed a new visual PatchMatch Filter, which is an advance in computer vision, with many future applications for fast correspondence field estimation. The idea is to match up the same regions or `patches’ in multiple images, for example, stereo views to find the distance of objects as humans do with two eyes or to track the motion of objects through multiple video frames. The technique is based on a computational framework that integrates the randomized search and propagation with efficient edge-aware image filtering, running much faster than other competing methods and achieving state-of-the-art correspondence accuracy.

“What’s special about ADSC’s patch-match filter technology is that it combines randomized search to find the best match very efficiently, while simultaneously finding and preserving the edges or motion discontinuities between the object and the background,” said Doug Jones, who became ADSC director in July. “This new technology will have many uses, from practical `green screening’ without the green background for home movie-making or video-conferencing, to vision-based collision-avoidance or self-driving automobiles.”

Additionally, Professor Narendra Ahuja‘s video analytics research team has made several key advances over the past year in tracking many moving and interacting objects in a complex video scene (such as all of the players on a football field), and making sense of this complex data. For example, they are applying research to sports video analysis to automatically recover information like what type of football play occurred, which players were involved and statistics on the performance of individual players.

Automated video analysis will save thousands of hours of tedious work for coaches who now have to do this manually, or for forensics investigators who currently have to watch hundreds of hours of security videos to identify criminals, such as the Boston Marathon bombers.

ADSC Research Scientist Gang Wang demonstrates the pill identification technology that was developed at ADSC and licensed to Singapore spinoff I3 Precision.

“The research has spawned a great deal of interest from coaches of professional and college football teams, including the University of Illinois, and they soon plan to spin off a start-up company to commercialize the technology,” Jones said.

Researchers at ADSC have also been continuously adding to the number of copyrights on their work, including adding ones for automated jersey number detection and recognition, entity search, energy analysis system, automated assistant for physical therapy and real-time elastic streaming analytics.

Furthermore, one of ADSC’s goals is to contribute to the forward-thinking research climate in Singapore by working with industry partners to further develop ADSC’s research. During the past year, ADSC’s pill identification research, led by researcher Gang Wang, was licensed by I3 Precision, a spinoff company from ADSC and KooPrime. I3Precision obtained a $250,000 Proof-of-Concept commercialization grant from Spring Singapore to develop the technology. Similarly, ADSC researchers are working with companies to further develop the people and restaurant entity searches developed by Professor Kevin Chang’s ARISE group.

Looking forward, Jones anticipates more accomplishments and further collaborations with Singapore partners, as well as continuing to strategically invest in research areas that will benefit Singapore and the University of Illinois.

“In the past year or so, ADSC researchers have received five best paper awards or nominations for top research, and have been winners or finalists in at least seven competitions to apply research results to real-world problems,” Jones said. “I’m very pleased that many ADSC researchers are being recognized as among the best in the world and also that our research is showing the potential to have real-world impact in many application areas.”

ADSC faculty member and ECE Associate Professor Minh N. Do has been announced as an IEEE Fellow for 2014. Do was recognized for his contributions to image representation and computational imaging. In addition to his position as an associate professor of ECE, Do has a full-time appointment with the Coordinated Science Lab, a part-time appointment with the Beckman Institute, an affiliate associate professorship with the Department of Bioengineering and is an active ADSC faculty member leading the ITEM: Immersive Telepresence for Entertainment and Meetings research group.

IEEE Fellows are elevated by the IEEE Board of Directors, which chooses the fellows based on nominees’ records of accomplishment. The Fellow grade is the highest grade of membership within IEEE. Of more than 400,000 members from around the world, 293 individuals were named IEEE Fellows this year.

The IEEE cited Do’s contributions to the area of image representation as instrumental to his elevation to fellowship status. One of Do’s image representation projects in particular stands out. With his first group of PhD students, Do developed a new way of representing images that can capture geometrical structures, which are typically the dominant features in images and multi-dimensional data. He published this theory, along with software that allows users to readily apply it. The project’s popularity far surpassed the team’s expectations.“Google Scholar showed more than 2,700 citations, and our software received, typically, 50 to 100 downloads per month,” Do said.

Do’s other crucial contributions cited by the IEEE are in the area of computational imaging. Do’s work in the area emerged from the interdisciplinary nature of ECE and the rest of the College of Engineering.

“The goal is to extract information from this massive amount of visual information,” Do said, “and to have ways we can present it in a very compact form, so in the end, a human can quickly look at it and make sense of these data.”

The practice has applications in a wide range of disciplines. Some of Do’s interests are in biomedical imaging and biology. When observing cell behavior over a period of several hours or days, researchers can extract an enormous amount of data in images of the cells, too much to analyze directly. This is where computational imaging comes into play.

“We can automate all of this, and we can then develop a theory of how the cells evolve,” Do said.

Do is also interested in exploring emerging possibilities of computational imaging involving the unprecedented number of images available on the Internet.

“A smartphone can take a lot of images, and often we don’t have time to look through all of them,” Do said. “Every day, almost half a billion photos are uploaded on Facebook.”

Do believes computational imaging can make more sense of these images.

“We can take the tools and make a real impact for both professionals and everyday consumers,” he said.

Do said that he is honored to be an IEEE Fellow and humbled to be an ECE faculty member. He said many ECE ILLINOIS faculty earn fellowship status with the IEEE, often by the time they become senior faculty. The department now has 42 IEEE Fellows.

“That’s really a statement of how strong of a reputation the department has,” he said.

The IEEE is the world’s largest professional organization for advancing technology, and is a leading authority on a wide variety of areas, including aerospace systems, computers and telecommunications, and biomedical engineering. The IEEE also publishes 30 percent of the world’s literature on electrical and electronics engineering and computer science, and is a leading developer in the international standards of telecommunications, information technology, and power generation products and services.

Editor’s note: media inquiries should be directed to Brad Petersen, Director of Communications, at bradp

ADSC researchers recently took second place at the 2013 IEEE International Workshop on Machine Learning for Signal Processing (MLSP) Bird Classification Challenge.

ADSC software engineers Hong Wei Ng and Thi Ngoc Tho Nguyen developed a machine-learning algorithm to predict the presence of a set of bird species in audio clips that were recorded in a number of locations within Oregon’s H. J. Andrews (HJA) Long-Term Experimental Research Forest.

ADSC software engineers Thi Ngoc Tho Nguyen (left) and Hong Wei Ng took second place in the 2013 MLSP Bird Classification Challenge. They developed an algorithm to predict the presence of a set of bird species in audio clips accurately.

Bird behavior and population trends are important topics to study as birds respond quickly to environmental changes and are able to tell researchers about other organisms, while being easy to detect. However, traditional methods of collecting data on birds often require manual effort. There are many advantages to using acoustic monitoring to study bird populations, such as increased temporal and spatial resolution and extent, applicability in remote sites, reduced observer bias and potentially lower costs.

Since the recordings were taken outside of a lab environment, the recordings included sounds that the researchers were unable to account for, such as wind, rain, insects or airplanes flying overhead. This was further challenging because there would often be more than one type of bird singing in an audio clip and sometimes the birds would sing together.

“Our technique had to be able to distinguish these sounds from those of the birds that we were interested in, in order to make accurate predictions,” Ng said.

The 79 teams that participated were given 322 audio clips for training and another 323 for testing. Each audio recording was 10 seconds long and all of the data files were collected in HJA over a two-year period. Each audio clip in the training set was labeled with the set of bird species present or otherwise given no labels.

The problem is formulated as a multi-instance, multi-label machine learning problem where an audio clip can potentially contain a range of anywhere from zero to multiple instances of bird vocalizations and the classifier has to predict probabilities indicating the presence of 19 species of birds in the test clips.

To complete the challenge, Ng and Nguyen’s algorithm first detected segments of interest in each spectrogram computed from the given audio clips. Next, audio and image features were computed for each of the segments and summarized using a “bag-of-words” model. These features were further augmented with encoding statistics relating species of birds to the environment, such as the probability that each of the 19 bird species would appear in the recording location of a given audio clip. Finally, Extremely Randomized Trees classifiers were trained using the binary relevance method to make the required predictions.

The key factors of Ng and Nguyen’s algorithm were the ability to segment very faint signals in the spectrogram and the ability to convert the multi-instance multi-label problem into single-instance multi-label problem. In addition, the algorithm was able to account for audio clips that did not contain any bird calls, but only background noise and sound from other sources such as insects and rain.

The team, named Herbal Candy, was awarded $600 for their runner-up placement. Ng and Nguyen, who took part in this competition as a side-project are considering further developing some of the techniques they learned during the competition for more general audio surveillance and event classification purposes.

“ADSC aspires not just to do theoretical research, but also to do research that has real world impact,” ADSC Director Doug Jones said. “These contests are an important step in that direction. They’re narrower than real world, but they’re based on real data with real application. The techniques our researchers used were related to their ADSC machine learning and audio research, but this was just an extra challenge for them.”

A team of ADSC smart grid researchers recently took second place in the Energy Efficiency for Everyone (E3) Hackathon, sponsored by Urban Prototyping Singapore, Energy Market Authority and Singapore Power.

ADSC research scientists Binbin Chen and Nils Tippenhauer, along with software engineer Varun Badrinath Krishna and senior research engineer William Temple competed in the three-day event to create prototype applications to encourage energy efficiency in Singapore’s residential sector.

ADSC researchers William Temple and Binbin Chen work on “Efficionado,” their web application to encourage energy efficiency in Singapore’s residential sector during the E3 Hackathon.

The competition attracted over 150 participants on 18 teams and brought together energy experts, developers, designers and researcher to develop creative solutions to help Singapore residents to be more energy conscious at home. As runners up, the team received four Belkin WeMo Home Automation Systems, which they hope to use in living laboratory related research.

“The challenge for the hackathon was coming up with an innovative concept that you can actually execute, to some degree, during the competition,” Temple said.

The teams were given several data sets and asked to create a prototype web or mobile application that encouraged energy efficiency in Singapore’s residential sector. Participants had access to more than 24 million electricity records and 8 million gas records spanning 36 months, as well as related information such as weather forecasts, typical appliance energy consumption and census data. The winning teams may be granted access to this real data after the hackathon to continue developing their prototypes.

“In the commercial and industrial sector, if a company wants to improve their energy standing, they do an energy audit,” Temple said. “So why doesn’t this happen with homeowners? Unfortunately, it’s too expensive.”

To combat that problem, the ADSC team developed a web application and business model, called Efficionado, which is an affordable, do-it-yourself, home energy audit using rented smart plugs.

A consumer would rent the plugs for two days and plug them into key appliances at their home. During day one, the plugs measure typical energy consumption over a 24-hour period. On day two, the user implements targeted energy efficiency strategies and measures the consumption again. Users receive an analysis report when they return the plugs showing exactly where and how they can most easily cut down on their energy usage, as well as personalized energy efficiency labels to stick on the appliances to remind users of how much they can save if they keep practicing energy efficient behaviors.

ADSC researchers (L-R) Binbin Chen, Varun Badrinath Krishna, William Temple and Nils Tippenhauer took second place in the Energy Efficiency for Everyone (E3) Hackathon in September.

“The competition encouraged ADSC and the community to be more conscious about their technologies appealing to ordinary homeowners, who demand something easy and inexpensive to use,” Smart Grid Program Director David Yau said. “Ideas like what we demonstrated will allow ADSC to monetize our research in energy monitoring and optimization and achieve broad impact through the mass market.”

ADSC smart grid researchers are currently highly involved in other energy efficiency research, as well. They have created a living laboratory in the ADSC offices to monitor energy consumption and are using an indoor sensor network to infer information about occupancy, thermal comfort and energy waste.

“Energy efficiency is an important research area in Singapore at the moment, as evidenced by the support from key government and industry players,” Temple said. “I think from that perspective, this competition was an important event to participate in and it is great that we were able to do well.”

For more information about the E3 Hackathon, view the following links:

Team Efficionado’s presentation

Energy Efficiency Energy efficiency ideas at E3 Hackathon – AsiaOne

EMA – Energy Market Authority: E3 Hackathon to Crowd-Source Energy Efficiency Apps – 4-traders.com

EMA – Energy Market Authority: E3 Hackathon Yields Promising Ideas for Energy Efficiency – 4-traders.com

ADSC and A*STAR recently approved four new research projects that will be conducted at ADSC’s offices in Singapore. The ADSC Technical Review Panel (TRP) and Program Management Steering Committee (PMSC) accepted these new projects during their meetings in August and September, respectively.

ADSC is moving in the direction of cyber security, data analytics and data visualization, which are important as they build upon ADSC’s research developments to-date by moving the human to be the center of focus.

“The beneficial premise is that such human-centered systems will be particularly relevant in urban environments where technology is crucial to improving the quality of life,” ADSC Director Doug Jones said. “Today’s pure information technology remains largely disembodied from the physical world and in locales in which, and with which, citizens interact, live and play. The center’s basic research direction will be to seize the leadership ground in extending the reach of certain technologies into this physical world.”

Jones and ADSC visiting senior research scientist Rama Ratnam will lead a new “Acoustic Event Detection, Classification and Analysis” research project. This project will focus on developing key signal processing and analytical tools to make audio-based monitoring and analytics practical and effective on a large scale, as there has been relatively little research in this area. This project will be a part of ADSC’s new real-time data analytics initiative.

Similarly, ADSC faculty member Deming Chen’s “Heterogeneous Architecture and Design” project will be extended into a new “Next-Generation Compilers and Architectures for Computation Acceleration with Energy Efficiency” project. He and his team will build on the framework, infrastructure and tool flows, in relation to FPGA and GPU programming, that they established in the initial three years of their project. They will be working to develop a complete and mature CUDA/OpenCL to FPGA compilation flow, as well as continuing development of the new high-level synthesis engine and work on the advanced GPU compiler. Their ultimate goal is to provide new tools, compilers and architecture for computation acceleration of complex computer vision and analytics with GPUs and FPGAs.

Faculty member and former director Marianne Winslett will begin a new project, called “Scalable, Real-Time Analytics for Challenging Data,” which will work to design and develop RESA, a general framework for performing common real-time analytics tasks on challenging data. RESA will support real-time processing of fundamental analysis tasks, such as analyzing surveillance video to identify criminals on the run or determining where roadblocks are during major natural disasters to enable first responders to get to the scene quickly.

Additionally, ADSC Associate Director Ravi Iyer, along with co-PIs Zbigniew Kalbarczyk and David Yau, will be conducting research on the smart power grid in a project called “Towards a Resilient Smart Power Grid: A Testbed for Design, Analysis, and Validation of Power Grid Systems.” This project will look at some of the challenges of incorporating a system-wide solution for the power grid that would reduce physical and cyber vulnerabilities and enable rapid recovery from disruptions. Iyer, Kalbarczyk and Yau will analyze these challenges by building a testbed and associated tools to facilitate rapid analysis and characterization of error attack resiliency of software and hardware components of the power grid structure.

“ADSC’s new projects address new directions that are just now emerging as key challenges for creating more intelligent and interactive digital systems that make modern life safer, easier, more efficient and hopefully just more fun,” Jones said.