Ryosuke Harakawa: Using AI to Solve Real-World Problems

Artificial Intelligence (AI) has permeated every aspect of business and our personal lives. A report from McKinsey estimates that around $13 trillion of global economic activities will be driven by AI in 2030. Given today’s unprecedented openness between research institutions and industries, it is exciting to see what AI-related discoveries will be brought to life by this collaboration.

At the Image Information Systems Laboratory (headed by Prof. Masahiro Iwahashi) at Nagaoka University of Technology (NUT), the team of Prof. Ryosuke Harakawa works on various AI applications that uniquely address real-world social challenges. 

To date, they have come up with AI for polishing cutlery, monitoring flood control, and providing road maintenance support.

“When you try to solve problems that are social and environmental in nature, and to some extent global in scale, there is a great need to bring individuals with highly specialized skills, knowledge, technical knowhow, and resources to the table,” he begins.

“The AI applications that we have come up so far wouldn’t be made possible without the collaboration that we had with different departments at the university, government agencies, partner organizations, and other experts,” he continues.

At the laboratory, Prof. Harakawa works in a team of 18 individuals, comprised of Ph.D. candidates, master and undergraduate students. The team specializes on the use of image recognition, social media / data mining, deep learning, and big data analysis to develop AI and IoT technologies.

While AI has been applied to bring business intelligence, automate business process, and deliver a novel approach to customer interaction, the lab focuses on multimedia signal processing for AI-related applications that bring value to society.

01. AI for Polishing Cutlery:
A Collaboration with Hardware Technology

Tsubame-sanjo in Niigata, Japan has been producing cutlery for more than one hundred years and is known internationally. The said industry dates back to the Edo period but due to the aging and falling population, necessary manpower to support the process of polishing cutlery has seen a sharp decline.

“To help sustain the production of these regional products, our team worked with NUT’s Department of Electrical, Electronic and Information Engineering, Prof. Ohishi (an expert on robotic engineering), the regional company of mechanics, and the Ministry of Economy, Trade, and Industry (METI), to develop an AI technology to support the craftsmen in polishing cutlery.”

“To make this happen, we developed a new method to determine the shape of the cutlery, and adopted a hue-based coding method to ensure a more accurate polished product.”

“We used cameras and projector to obtain a 3D surface measurement of the material, and robots to polish the spoon and fork based on a hue-based coding method.”

The existing method of determining the shape of the cutlery is based on luminance captured by the camera.

“Specular reflection, which is the mirror-like effect seen from the surface of cutlery, can serve as cue for the shine, glossiness, or finish of a cutlery. But the specular reflection causes the performance degradation in the existing method for a 3D surface measurement based on luminance.” 

“With a hue-based coding method, we can accurately measure the 3D surface for enabling us to judge the finish of the cutlery,” Prof. Harakawa explains. 

02. AI for Flood Control and Road Maintenance:
A Collaboration with Civil Engineering and Disaster Prevention Science

Nagaoka and Niigata are home to the longest and widest river in Japan, the Shinano River. At 367km in length, the river surrounds many communities and has been extensively used for electricity generation and irrigation.

Despite the benefits, seasonal high rainfall can result to flooding which can claim lives, damage properties, and destroy crops such as rice, which is the principal product produced in the region.

“Using data mining (from weather news, Twitter, and social media), image processing obtained from river surveillance cameras, and deep learning, we analyze river images to accurately detect flood risks. This collaboration project with Prof. Matsuda at the Department of Civil and Environmental Engineering and the company related to the disaster prevention system can contribute to ensure the safety of the residents of Nagaoka and Niigata.”

“Our deep learning technology is also used to enable road maintenance support for human labor. In-vehicle cameras allow us to capture the condition of roadside guardrails and highway fences. And through deep learning, we can classify the degradation level and make a recommendation if we need to replace or repaint.”

03. AI for Plasma Simulation:
A Collaboration with Applied Physics

In the realm of applied physics, Prof. Harakawa’s team has developed an image processing technology that is capable of observing ultrafast phenomena such as the molecular processes in plasmas.

“Plasmas are hard to observe because, like gases, they do not have fixed shape or volume, and have less density than solids or liquids. Also, because they are ultrafast phenomena, they cannot be observed in existing image capturing devices.”

“Our image processing technology can combine overlapped images, and restore ‘real images’ at each time. This collaborative research with Prof. Sasaki at the Department of Electrical, Electronics and Information Engineering can accelerate the plasma simulation or any phenomenon that exceeds the frame rate.”

The team’s research on image processing technology for plasma simulation and hue-based coding method for polishing cutlery were both accepted for publication in the top journal in applied physics and electrical and electronic engineering, respectively.

Prof. Harakawa was originally interested in multimedia information retrieval and recommendation. When he first started the research, social media such as YouTube and Twitter were popular and widely used.

“During that time, terms such as “big data” and “unstructured data” (untagged data such as image, video, and music) attracted wide attention especially in the research community. User-generated media came from different sources. And the space where these data exist was very heterogeneous and unstructured.”

“The problem that time was, if you are going to input queries that are not suited based on the design of the algorithm, then, you are not going to see relevant results.”

To acquire the desired contents based on the query, Prof. Harakawa’s team developed a model that uses consensus clustering of data based on semantic and sentiment similarity estimation.¹

“Our model efficiently analyzes these unstructured data in order to come up with the hierarchical structure of similar content groups. This helps us extract useful information for the user from a sea of ambiguous data,” he explains.

“In sentiment estimation, we look at the emotions or attitudes expressed in a text. I think the easiest example would be classifying a text as something that is ‘positive’ or ‘negative.’ The word love for example can be viewed with a positive sentiment, while hate is viewed, otherwise. Now, when you look at other words that show the same sentiment, that’s when you have sentiment similarity.”

From Prof. Harakawa’s field of expertise and area of research at the laboratory, he considers that AI can become the controller of ‘information flood.’

“Information flood happens when a user of the Internet gets a sea of information, which causes the user to become unable to process all the data that is presented to a user.”

Using COVID-19 as an example, he further explains:

“In unprecedented incidents such as COVID-19, we’ve seen how a lot of information and misinformation (ex. 5G network being the cause of COVID-19) were spread in the cyberspace. As a result, users cannot acquire the necessary knowledge from the Internet, and some people acted inappropriately in the physical space.”

“This is because we can only remember knowledge that we originally knew, and cannot discover new knowledge in the unprecedented scene by the current technology. We developed the AI to overcome this limitation.”

In Prof. Harakawa’s research, the team looked at Twitter and identified keywords that are highly correlated to the number of infections of COVID-19 as the knowledge.

“By automatically understanding the semantics of many tweets and extracting similar tweet groups, people can discover new knowledge efficiently. And at the same time, we can visualize the minority opinion hidden in the majority. In a way, we can see the value of AI when it comes to avoiding human bias.”

“Current students at NUT have a lot of potential, and I have seen many students who are good at quickly implementing software and hardware programming. However, many of them do not know their own potential and do not have the confidence.”

“In the future, I would like to meet students who are patient and have the willingness to challenge known and unknown experiences. Through the fundamental researches and collaboration with other fields and partners such as industries, companies, and/or government, I would like to make them aware of their own potential.”

“And eventually, through their own research, I look forward to seeing them obtain significant research achievements be it in world-class top journals or research that has practical, real-world applications,” he concludes.