To regain full functioning, many stroke or trauma patients must continue physical or occupational therapy even after returning home from the hospital. But often, their progress diminishes once they are home, said Rama Ratnam, a senior research scientist at the University of Illinois at Urbana-Champaign.

Rama Ratnam “Patients almost always do well in a hospital, where there’s someone who monitors them to make sure they do their exercises,” said Ratnam, who is also a researcher at Illinois’ Advanced Digital Sciences Center in Singapore. “Home-based rehab has not been as successful because once patients go home, there’s much less accountability in making sure patients are performing their exercises,”

Ratnam is leading a group of Illinois and ADSC researchers who have developed a new system that makes it easier for a doctor or therapist to keep tabs on a patient’s progress remotely. The technology employs Kinect cameras, which capture 3D motion data, and then adds a mathematical layer that eliminates noise and produces a smooth motion, giving health care providers an accurate picture of how a patient is performing an exercise.

A major advantage of the system is that data is stored in the form of a stick-figure representing major joints and body segments, and not as a detailed video that can reveal the identity of the person. This reduces the data bandwidth while ensuring privacy. The system, called Salus (after the Roman Goddess of Wellness), was initially developed by ADSC engineer Alex Khromenkov. The ADSC team includes Vignesh Ramkrishnan and Stefan Winkler, who are continuing to develop the technology and target specific applications.

The research comes at a time when health care costs are driving therapy patients to reduce time spent in in-patient rehabilitation care and return home before regaining full functioning. In addition, as people continue to live longer, there may not be enough providers to care for patients in traditional settings in years to come.

Ratnam’s at-home rehab care system would help overcome these challenges by allowing physicians to access video of patients’ therapy at a convenient time. After the patient performs an exercise, physicians could then download the data from a cloud server and play back the video. The interface also includes a calendar that would enable physicians to schedule specific exercises on certain days.

The Salus system

According to Ratnam, the mathematical layer is what makes the system unique. Kinect cameras excel at capturing motion in 3D, but do not have a holistic concept of the human body. They only see a series of points in space. The algorithms developed by Ratnam’s team imposes constraints on the data that rules out movements that humans are physically incapable of doing.

“You can’t rotate your neck 180 degrees like in ‘The Exorcist,’” he said. “The camera also doesn’t assume that a person’s arm length doesn’t change from frame to frame. It’s important to smooth out unwanted noise and make sure the system understands what is possible and what isn’t.”

“You can’t rotate your neck 180 degrees like in ‘The Exorcist,’” he said. “The camera also doesn’t assume that a person’s arm length doesn’t change from frame to frame. It’s important to smooth out unwanted noise and make sure the system understands what is possible and what isn’t.”

In order to get a good picture of what a movement should look like, Ratnam is working with colleagues at the University of Illinois to apply the technology in specific populations. For example, Prof. Jacob Sosnoff, a professor of kinesiology and human health, is using the system to assess and identify elderly at a high risk of falling. Another kinesiology professor, Robert Motl, is working to develop home-based exercises and remotely monitor compliance in multiple sclerosis patients. Ian Rice and Laura Rice, whose research focuses on wounded veterans, are collaborating with Ratnam and Jake Sosnoff to monitor wheelchair transitions, such as a patient moving from a wheelchair to a chair or a bed, with the aim of minimizing shoulder problems. Each of these populations has specific needs, and the Salus system along with the body model functions as the core engine in powering the various applications.

Researchers are currently working to develop a prototype, which they hope to test at Illinois and in Singapore.

“While there are some limitations to what this technology can do, it is non-invasive, touch-free, and guarantees privacy while monitoring movements,” Ratnam said. “We intend for the system to relieve the burden currently on health care professionals by enabling remote physical rehab in the home along with the crucial capability to monitor compliance.”

Through two new projects funded by the Energy Market Authority of Singapore, researchers at the Advanced Digital Sciences Center are tackling real-world issues to improve the cyber security and resiliency of power plants and the smart grid.

The first project, “PoPSeCo: Power Plant Security by Advance Sensing and Computing,” seeks to create more secure information and communication technologies (ICTs) by designing innovative sensing and computing methods. ICTs, which are widely used in power plants, have the potential to improve system efficiencies, but they also increase cybersecurity risk.

In this project, an ADSC research team, including Adjunct Senior Research Scientist Rui Tan (co-principal investigator, assistant professor at Nanyang Technology University, Singapore), Postdoctoral Researchers Yang Li and Xin Lou, and Senior Software Engineer Sreejaya Viswanathan, will develop accurate and secure clock synchronization approaches and an enhanced intrusion detection system (IDS) for centralized supervisory and regulatory controls. The research team has identified vulnerabilities of the clock synchronization protocols (e.g., network time protocol and precision time protocol) that are widely adopted in current power plant systems and a broader range of industrial systems. The team will leverage power grid frequency, an extrinsic yet secure physical signal, to synchronize grid-connected devices to overcome these vulnerabilities.

Researchers will collaborate with Senoko Energy, the largest power generation company in Singapore, to develop an enhanced IDS for power plant industrial control systems. The IDS will detect malicious and inaccurate centralized control commands by predicting the consequences of their execution and checking them against command history to see if they deviate from the norm.

Collaborators include David Yau (principal investigator, professor at Singapore University of Technology and Design), Zbigniew Kalbarczyk (research professor at Illinois), as well as researchers and industry experts from Purdue University, Argonne National Laboratory, and Singapore’s Ministry for Home Affairs. Yau is also ADSC’s Cybersecurity Program Director and Distinguished Scientist.

In the second project, “Securing Last-Mile Communication Systems for Smart Grid,” researchers will tackle the vulnerabilities of last-mile communication systems – such as remote terminal units and smart meters – for the smart grid. The project, a collaboration of the Institute for Infocomm Research, ADSC, and Mirai Electronics, will develop novel multi-layer protection against security threats associated with these systems, which are susceptible to malware, malicious commands, and denial-of-service attacks.

Research effort in this project focuses on the development of a software-only remote attestation scheme for a network of interconnected field devices. The attestation scheme ensures that no malware will be able to hide inside a networked system. Its software-only design makes it easy for the scheme to be applied to protect power grid devices being deployed today, which often do not have built-in secure hardware support.

Compared to prior software-only attestation solutions, the new scheme is much more efficient, reducing the attestation time on a single device by up to 100 times and on a networked system by an additional 10 times. This improvement is critical for preserving the high availability of smart grid systems. The solution is also resilient to potential collusions inside the network by leveraging a novel network-level assurance.

 Another research effort in this project aims to develop an active command mediation module that can make the smart grid more resilient to malicious commands, even if insider attackers or malware compromised the control center (as what happened in the recent Ukraine incident). To facilitate future adoption, the module has been designed from the beginning to be compliant with existing smart grid standards and architectures, as well as common power grid operation practices.

“If successful, the research will enhance the security level of power grids that use intelligent field devices for smarter monitoring and control, both in Singapore and beyond,” said co-PI Binbin Chen, an ADSC senior research scientist. Other ADSC staff involved in the project include co-PI Xinshu Dong, Research Scientist Daisuke Mashima, and Software Engineers Prageeth Gunathilaka and Sumeet Jauhar. Illinois Professor Ravi Iyer joins researchers from the University of Malaga and Hong Kong’s ASTRI as collaborators.

Both projects were funded under the Energy Innovation Research Programme, administered by EMA. The EIRP is a competitive grant call initiative driven by the Energy Innovation Programme Office, and funded by the National Research Foundation (NRF).

The projects build on ADSC’s track record and expertise in power infrastructure cyber security R&D in Singapore.

According to ADSC Project Manager William Temple: “While our past integrative security assessment and resilient smart power grid projects took a more holistic view of power systems, these new projects and our industry partners give us an opportunity to zoom in on specific cybersecurity challenges for generation and field devices.”

Analyzing data in real time is often a struggle for researchers. To address those challenges, a team at the Advanced Digital Science Center, a University of Illinois research center in Singapore, has built a platform that provides an easy solution for analyzing text, video, audio and other types of data quickly and accurately.

“We hope that our analytics can help the decision makers to make decisions as quickly as possible when some special event happens,” said Zhenjie Zhang, an ADSC senior research scientist. “For example, when analyzing video, we want to detect events or faces as quickly as possible. Or, we may hope to monitor social media networks and identify the new trending topics. For sensor networks, such as the power grid, we want to find problems and notify engineers immediately, so they can stop the problem before it gets worse.”

The system utilizes simple programming, so that programmers in other fields can easily integrate their codes into the program.

“There are many fancy computer vision algorithms out there,” Zhang said. “The problem is that the algorithms are too complex and cannot be run in real time. They can only analyze about one to two frames per second, but we can do real-time tracking.”

Another unique aspect of ADSC’s solution is their use of elasticity, which enables automatic scaling as the volume of data increases or decreases throughout the day. Zhang, along with ADSC Research Scientist Richard Ma and Illinois Computer Science Professor Emerita Marianne Winslett, has adopted an elastic cloud-based scheme, where a user pays for service only when they need it.

For example, the researchers ran a demonstration in which they took real-time transportation data produced by Shanghai’s metro, bus, and taxi systems and were able to predict what traffic congestion would be like in one hour. The capability could help commuters know how long their trips might take. The cost of using the tool would also adjust as the volume of data increases, such as during rush hour, when more machines are used.

“This elasticity is the most challenging aspect of the research because when a very fast data stream comes to your system, you need to move the results to a new machine and you want to try to make this migration as quick as possible so it doesn’t affect the current computations,” Zhang said. “We’ve developed our solution, but are always working to improve it.”

The team, composed of a variety of experts in data management, machine learning, and data mining, has almost finished their system platform prototype and is working to add new features and additional applications on top of the platform. They are also working closely with the open source community to add functionalities and improve shortcomings of current platforms.

“As we’ve been developing our platform, we’ve talked with many computer vision researchers and there’s a huge gap between the research and practice,” Zhang said. “They usually focus on the accuracy of the model, but don’t care if they can deliver the prediction on time. We can help supplement the research to help them bridge that gap.”

At any one time, a person is likely to be close to at least three to four wireless devices, such as smart phones and tablets. Those devices would include not just his or her own, but also ones belonging to other people nearby. At airports or shopping centers, the number of people, and thus the number of devices, can be substantially higher. ADSC Research Scientist Binbin Chen is looking into how to capitalize on these close quarters to help phones save energy.

Chen, along with National University of Singapore (NUS) computer science student Girisha De Silva and NUS Associate Professor Mun Choon Chan, recently won a Best Paper Award (networking track) at the 2016 International Conference on Distributed Computing and Networking for their idea, which uses Bluetooth to allow phones to share each other’s energy usage.

“Phone batteries are used up on lots of useless things,” Chen said. “Even when the screen is off, it’s doing things in the background, and up to 60 percent of the phone’s energy can be wasted in this state.”

Low-frequency, low-data-rate background traffic, such as push notifications or incoming emails, can drain phone batteries quickly, but by using low-power Bluetooth radio and sharing cellular bandwidth, Chen and his team have saved up to 90 percent of background traffic energy consumption in urban settings. The team developed a middleware on Android to automatically enable the use of Bluetooth radio for sharing cellular bandwidth, but only during times when it would be useful for collaboration.

While other researchers have been looking into solutions to this problem, Chen’s method takes a different approach by looking into how phones can collaborate with each other.

“Normally phones won’t be alone and will be nearby other phones,” he said. “All devices can form a local community that collaborates, so we’re trying to develop a way for the phones to recognize each other and piggyback on other people’s usage in an energy efficient way.”

Chen said that if only two to three people piggyback off of each other, a phone can save about 10 percent of energy, but if there are 10 devices nearby, a phone can save more than 50 percent of its energy. To help ensure equality between users, the team developed an algorithm that schedules users to take turns to become contributors to the collaborating system based on each user’s individual use patterns and the team’s proposed fairness notion.

Chen and his fellow researchers believe the combination of the characteristics of background traffic and Bluetooth is the key to helping save energy, because while Bluetooth consumes little power, it is slow. Luckily, the background traffic is very small in size and can tolerate wait times, which makes Bluetooth ideal for this type of work.

Chen and his team have been determining if this method is feasible through real prototypes and conducting simulations on a larger scale. They have also designed a solution that ensures the method is fair to all users while maintaining as much efficiency as possible.

“Even 10 years ago, we were just beginning to be affected by smart phones,” he said. “Think about how many things have changed since then and what will happen in the next 10 years. We could be surrounded by hundreds of devices that talk and collaborate with each other.”

Chen believes this technology could be incredibly useful in the future, as energy is always a top constraint in developing new Internet of Things applications.

One concern that hasn’t been addressed yet is security challenges that will arise as a result of this type of collaboration. Chen, who works with ADSC’s cybersecurity team, began the project on the side a few years ago, but foresees future research on this topic fitting in with ADSC’s current work.

“We haven’t touched on the cybersecurity aspect of the project yet, but with phones collaborating like this, security is very important,” he said. “Not only is the phone talking to the server, but it now sends traffic through its neighbor, so there are a lot of security and privacy issues that still need to be addressed. It’s a work that can lead to a lot of interesting mobile computing security, Internet of Things security and mobile application security research.”

When exploring a new city, visitors may want to find good places to eat, drink, and shop in a throng of unfamiliar places. Researchers from the Advanced Digital Sciences Center (ADSC, a CSL research center in Singapore) and Singapore Agency for Science, Technology and Research (A*STAR), wanted to make this process easier by combining location-based augmented reality and a large database of online reviews to support an intelligent shopping experience.

The researchers have developed IntelligShop, an app that uses a phone’s camera and location-tool to automatically detect nearby businesses and pull up reviews from multiple sources on the same screen. Users can easily and efficiently scan nearby retailers and peruse reviews all in the same app.

IntelligShop is an app that allows users to automatically detect and search reviews for nearby businesses.

“A positive shopping experience is very important, especially for a tourism city like Singapore—almost 30 percent of the money spent by Singapore tourists is on shopping, food and drink, which translates to $6.8 billion,” said Vincent Zheng, the lead ADSC research scientist for this project. “We are trying to make the shopping process more informative and smarter.”

With IntelligShop, users are able to view a variety of peer reviews in order to make an informed decision on where to go—it doesn’t pull from one source, but from many, much like a traditional Google search.

“The problem we saw with just pulling up reviews from one source is that not every business has reviews in all the same place. It might help one business to draw from a single source, but it would hurt another business,” said Zheng. “So we draw from different blogs, forums, social media, and review sites, to integrate reviews like what you would find by searching on the internet.”

This review gathering process is fully automatic, thanks to a new learning-to-query algorithm that behaves just like a human using Google to search for information.

While the app can be used in any setting, it specifically works well in indoor shopping centers. Indoor location sensing is usually a challenge because every phone behaves differently—different phones receive WiFi and other context readings in different ways.

“Indoor tracking is not a difficult problem if everyone has the same kind of devices. But we don’t,” said Zheng. “So we had to build an algorithm that could accommodate many different systems and still work effectively.”

Advanced location-tracking is particularly helpful in dense cities like Singapore that have many indoor malls. The team is currently testing the app in indoor malls in Singapore. After research is completed, the team will release the app to the public.

The IntelligShop project is conducted by a joint ADSC–A*STAR (Institute of Infocomm Research) research program and supported by the research grant for the Human-Centered Cyber-Physical Systems Programme at ADSC from Singapore’s A*STAR. The team members include: Vincent Zheng (ADSC), Miao Lin (A*STAR), Hong Cao (A*STAR/McLaren Applied Technologies APAC), Yuan Fang (A*STAR), Aditi Adhikari (ADSC/Illinois), Shenghua Gao (ADSC/ShanghaiTech University), Kevin C. Chang (Illinois), and Shonali Krishnaswamy (A*STAR).

There is a growing trend toward personalized consumer experiences, and this includes an emerging demand for personalized fashion shopping and purchasing. Imagine finding an image of someone wearing an item of clothing, putting that photo in an app, and being instantly presented with dozens of images of garments with similar colors, patterns, and shapes. Each item is linked to a website where customers can purchase it. This highly customizable shopping experience is within reach.

FashionMatch allows users to instantly search for visually similar apparel items from a large database of images.

ADSC researchers Vassilios Vonikakis and Siddhanta Chakrabarty and their colleagues have commercialized FashionMatch, a fashion-optimized tool that analyzes apparel in an image to instantly find visually similar items and where to purchase them from a large database.

The technology uses an algorithm that can identify humans, analyze posture, estimate the proportions of the garments relative to the body, and separate the garment from the background, all without user intervention.

“The algorithm has mastered many computer vision challenges. Even with shadows and cluttered backgrounds, the algorithm can identify which pixels of the image correspond to the person, to the garment, and to the background,” said Vonikakis, an ADSC research scientist. “It can then identify the skeletal structure of the person, which makes it possible to tell the fashion-related proportions of the garment, like how long the sleeves are, the proportion of the neckline to the waist, general shape, and more.”

The selections draw from a large database of images, and users can customize their results based on their preference for certain similarity attributes. For example, users can rate how important it is to them that selections match the garment’s color, pattern, or shape, and receive different results based on those selections.

The research for this technology began in CSL by a team of graduate students, led by Bernard Ghanem, an ECE Ph.D. student at the time, in Professor Narendra Ahuja’s Computer Vision and Robotics lab. Ghanem handed the project—then called FashionLatte—to ADSC for commercialization.

With a $285,000 USD grant from A*STAR, an agency in Singapore that encourages research commercialization, eight engineers were hired and split into two teams to work in parallel on developing two versions of FashionMatch for the marketplace.

One team was led by Vonikakis to evolve FashionLatte, and the other team worked on a similar algorithm developed by former ADSC research scientist Wang Gang. Vonikakis’ team focused on making the algorithm robust for real-life conditions, while Gang’s group focused on the business development side.

After the grant ended, Chakrabarty, an ADSC software engineer, joined Vonikakis’ team to make the technology more user-friendly for administrators. The two teams have already sold the license of FashionMatch to five companies, including Jet Solutions, IQnet, and Image Science, and are looking to sell to more.

“We started with a good algorithm, and then we upgraded specifications to meet the needs of clients. They wanted the app to function on mobiles and ‘in the wild,’ which meant identifying the garment accurately even with complications like shadows and cluttered backgrounds,” said Vonikakis. “Our two groups delivered two complementary solutions to the growing demand for personalized and customizable fashion search and retrieval.”