Reiko Katayama – Designing for Comprehensibility

For those who cannot hear or have difficulty hearing, making sports spectating more realistic.
The digital technology “Miruoto,” which visualizes the sound of the competition,
is changing the way people watch sports games.
Reiko Katayama, president of Hogaku Co,
pursues designs that convey sound through means other than sound.
To share space, power, and emotion with all people.

 

 

The set of a singing show changed her life.

-How did you establish Hogaku Inc.?

Katayama-san　I originally studied spatial design such as architecture and landscape architecture at an art college. Many of my classmates went straight into construction companies or general contractors, but I wanted to do something more like contemporary art, so I went on to graduate school. After that, I joined a design production company that I had a chance to work for, and I got married at the same time. At that time, I wondered if my life was going to end like this. I wanted to do something new.
That’s when I started looking for a new job. But right around that time, an increasing number of companies were stopping their hiring activities due to COVID-18. Some of the companies I applied to stopped contacting me…. But I had already started looking for a new job, and I had already told the company that I was quitting, so there was nothing I could do, so I decided to go freelance. Butwomen freelancers are often looked down upon…. Women were often thought of as if she was doing it on the side of housework. I was so angry about that. So I thought that if I became a president, I would not be taken lightly, and that is how I started my business (laughs). Many of my relatives are also in business, so I created the company without much resistance.

-What made you decide to study design?

Katayama-san　There is a singing show called COUNT DOWN TV, right? When I was in high school, I watched it and was fascinated by the stage set behind the singers. It was so beautiful, and I had an intuitive feeling that I wanted to make something like that. It was late at night when I woke up my mother and said, “I want to go to art school! ” (Laughs). I started attending a prep school that winter. That really changed my life

-Were you always interested in design and art?

Katayama-san　I liked to draw, but I was not a very good artist. I struggled a lot when I was in prep school and at art school, and I even had a paper model that I worked on for eight hours crumbled in front of my eyes and was thrown away, saying, “This is garbage”. However, I have always been the type of person who, when I get a chance to do something, I say, “Oh, this is it!” and just go for it.

Visualizing Sound

– In addition to Katayama-san, we are joined today by Iwata Lab. at Waseda University (Toriya-san and Gourmelen-san) and Aisin Corporation (Shioura-san), who were involved in the development of “Miruoto”. Please tell us about the technology of “Miruoto,” which won the 29th UPGRADE with TOKYO*.

*UPGRADE with TOKYO: A pitch event in which startups and the Tokyo Metropolitan Government collaborate to solve metropolitan government issues. The theme of the 29th event is “Realization of competition venues where sound can be seen and felt.
Official website: https://upgrade-tokyo.metro.tokyo.lg.jp/event/

Katayama-san　”Miruoto” is an atmosphere cheering visualization system that visualizes and displays sounds of the atmosphere and cheering in the competition in real time. It captures the atmosphere of the competition and the energy of cheering as onomatopoeia and displays it on a large screen or online. By visualizing the sound of rackets hitting the ball and shoes rubbing on the floor, this technology aims to allow those who cannot hear to visually experience the power of the sport and to make it easier to understand the progress of the sport.
*Official website of “Miruoto”: https://miruoto.jp/

Originally, Aisin and Waseda University were working together to create technology to visualize sports sounds. We (Hogaku), who have a long-standing relationship with Aisin and have planned and provided services together with deaf and hard-of-hearing staff, were added to the project. The three parties decided to work together to create something that would better reach those who cannot hear by having our company participate in the “Miruoto” project. This project began in earnest with “UPGRADE with TOKYO.
The division of roles is as follows: Aisin is responsible for the “voice recognition” technology, Waseda University for the technology to determine “where the sound is coming from” using motion recognition technology, and Hogaku for the planning and design part.

– “Miruoto” is now visualizing the sound of table tennis play among many other sports. Why did you choose table tennis?

Katayama-san　First of all, it is an indoor sport. Another is that it is somewhat compact in scale, such as being played one-on-one. These characteristics made it easy to judge the sound of the competition and to estimate the position of the body and where the sound was coming from, and table tennis matched these other detailed conditions as well. We are also working on the next competition to visualize…it is still a secret (laughs). Please look forward to it!

Real-time visualization and sound display of the atmosphere and cheering at the spot during the game
©Miruoto

– What kind of technology is used to determine where the sound is coming from?

Toriya-san(Waseda University)　Our laboratory is working on robot development, Broadly speaking, our theme is “to create robots and systems to connect people with something”. We are doing a wide variety of research related to this theme. For example, we are doing research on rehabilitation systems and robots that treat cancer. Robots that use echoes to examine heart disease are also targets. The technology used to locate and recognize the position of an object on a video image is used to determine the position of the sound required for this “Miruoto”. The AI model accumulates the movements of a human swinging a racket and calculates the position of the sound based on the way the racket moves.

– Table tennis sounds are currently represented by four types of onomatopoeia, but how are the sounds picked up?

Shioura-san(Aisin)　There are four types of sounds: the “konk” and “kak” of the ball hitting the racket or table, the “squeak” of shoes rubbing against the court, and the “crackle” of applause. In order to pick up sound accurately, we use a special microphone to adjust how far the sound is picked up, etc. If only part of the sound is picked up, there is no reality, and if the range is too wide, the surrounding sound is also picked up. We use two main types of microphones: one is a directional microphone placed on the side of the stand, and the other is a vibrating microphone. The vibration microphone is a microphone that is placed on the stand and can respond only to sounds emitted in contact with the stand.
At first, the only sound we visualized was the sound of the ball being hit, but when we actually watched table tennis using our ears, we felt that the sound of the ball being hit alone was not enough to express the intensity and fun of the competition. We came to the conclusion that  it would be better to visualize not only the sound of the ball being hit, but also the atmosphere and excitement of the event. So we came up with the idea of the sounds from shoes and applause.

– What do you pay special attention to in “Miruoto”?

Toriya-san (Waseda University)　Even if a person who cannot hear can see the sound, he/she should be able to understand where the sound is coming from. We are particular about displaying onomatopoeia that accurately identifies where the sound is coming from.

Shioura-san (Aisin)　The other thing is the speed at which the sound comes out. Table tennis matches are played at great speed, and if there is a time lag between the time the game is played and the time the sound comes out, the viewer is likely to wonder what is going on. Therefore, we are developing a system that minimizes the time lag so that people can watch the game without stress. We are constantly accumulating data to improve the accuracy of both the technology for capturing the position of the sound and the technology for minimizing this time lag. We already have a ping-pong table in the lab. Whenever I have free time, I take data and run the AI model…. Everyone got really good at table tennis (laughs).

Katayama-san　Each of us is continuing to grow step by step toward the year 2025, while conducting daily research in our respective fields of expertise. I am glad that the three of us are able to work on “Miruoto” together, as we are members who can mutually enhance each other’s skills.
I took two years to get into college. I entered the workforce two years later than everyone else. But if I had entered the workforce two years earlier, I don’t think I would have participated in this project. So I really think it was a fateful encounter.

-Katayama-san, you are also involved in the design of “Ekimatopeia”*.

*Ekimatopeia: A device that visually represents environmental sounds such as station announcements and train noises as letters, sign language, and onomatope. The catchphrase is “I never knew the sound of a train before.” The project team of four companies, including Fujitsu, worked with children at Kawasaki City School for the Deaf to come up with the idea, aiming to create an experience that is easy for everyone to use and makes daily train use more enjoyable.
Official website: https://ekimatopeia.jp/

“The Sound on the Train, I’ve never heard before.” Ekimatopeia: Ueno Station version
© Fujitsu Limited

Katayama-san　When I started working on “Ekimatopeia,” I did not know anyone who was deaf, nor did I have the opportunity to meet anyone who was deaf. However, I had a chance to talk to a deaf person during the design process. When I first saw the design there, I was told, “This may be cool, but it’s hard to understand”. It was the first time I was told that a design was “hard to understand”… I had always thought that design was about making “cool” things. But there is also the perspective of “easy to understand”. I had a knowledge of it in my head, but I didn’t know how to use it. It was like a new angle was born for me as a designer: something that is “easy to understand” and “beautiful.
I usually post about my work on social networking sites, and I posted about Ekimatopeia as I usually do, and the response was astonishing. I received many comments, especially from people who are deaf or hard of hearing, saying, “Thank you for creating such a wonderful thing”. I had never received such appreciation and feedback for my work before. That made me happy, and I began to want to know more about people who are deaf or hard of hearing. I found myself hiring deaf people as part-timers (laughs).

A place where employees can realize their dreams

-Onomatopoeia is the key to all of these techniques. How is the onomatopoeia designed?

Katayama-san　In the development process, we always include respecting the opinions of the parties involved as an axis of the project. This is not an ego-driven approach. When considering what to do for people who are hard of hearing, it is a prerequisite that we conduct proper hearings with the people concerned before proceeding. The project team for “Miruoto” also includes members who are deaf or hard of hearing from the planning stage.
Our designer, who is in charge of “Miruoto,” has been completely deaf since birth, and I think her sense of sound is different from that of people who can hear. We believed in her sense of sound, which is different from those of people who can hear, and asked her to draw on her own experience in manga and animation to visualize the sound. I thought that if I made an animation of the onomatopoeic sound of the ball hitting the ground, the sound would be conveyed in a way that was closer to the image of the ball. We do not want to just design, we want to create while examining whether or not the expression can be properly conveyed to those who cannot hear.

-Katayama-san, you are also developing “Kikoniwa”*1, a media for people who cannot hear, and “Gratuna”*2, a job site for people who can’t hear.

1 Kikoniwa: Web media mainly by writers who are deaf or hearing impaired. It collects information on lifestyles related to deafness and deaf-blindness, businesses run by deafblind people, and events run by deafblind people, and publishes articles and videos.
2 Gratuna: A job site for people who are deaf. It is designed to facilitate smooth matching between individuals and companies, and offers a “Preparation Program” that includes basic knowledge training before employment, and an “After Support Program” that provides appropriate consideration and advice to companies after employment.

Katayama-san　Of course I want to make all people with disabilities happy. However, since I do not know all of them, it is difficult for me to imagine what I would like to do…. In such situations, I always think, “First of all, I want to make my employees happy”. All of my employees are very nice people. They came to work for such a small company. That’s already a good person, isn’t it (laughs)? When we think about what we can do to make these people happy, services like “Kikoniwa” and “Gratuna” are naturally born. “Kikoniwa” is very lively because the people who are involved in the project are able to communicate what they are actually struggling with. They are trying to shape what they feel are their own issues with society into solutions at my company. I myself am extremely grateful for this, and I am very happy that the company has become a place where everyone can realize their dreams. Creating a place where all employees can do what they want to do is what makes my day-to-day work worthwhile.

 

Opportunity to learn about the parties involved

-The Deaflympics will be held in Tokyo in 2025. What are your expectations for the Games and what goals do you hope to achieve with the Games?

Shioura-san (Aisin)　This may sound very engineer-oriented, but I want to make sure that “Miruoto” is something that people can enjoy. I strongly hope that as many people as possible who cannot hear will be able to enjoy watching sporting events. It was very moving to hear people say that they were able to learn about sounds that they had never heard before.

Toriya-san (Waseda University)　Our laboratory is working to enrich people’s lives and to create a “society where no one is left behind”. I believe that the “Miruoto” project is very much linked to this laboratory’s philosophy. I would like to take the opportunity of the Deaflympics, and in the future as well, to let people who cannot hear enjoy watching sports, and to realize a “society where no one is left behind”.

Gourmelen-san (Waseda University)　I wanted to learn sign language. I came to Japan in April 2023, but I didn’t have a chance to meet people who don’t hear until then. When I first started communicating with people in Japan who could not hear, I wanted to understand more about what they wanted to say, and I also wanted to communicate more. If I could speak in sign language, I might be able to know things I don’t know or have never thought about, and it might lead to research. Deaf people from all over the world will come to Japan for the Deaflympics, and I would like to learn sign language and actively engage in conversation.

Katayama-san　I believe that there is still a great deal of prejudice and knowledge about hearing impairment in the world. I would like to eliminate them one by one. I am approaching the issue through Gratuna and other means, but that is not enough. This time, we have a great opportunity, the Deaflympics, which will create a tremendous opportunity to learn about hearing impairment. I hope that the Deaflympics will provide an opportunity for people to learn about new technologies such as “Miruoto” and at the same time, provide an opportunity to learn more about people who cannot hear. I would like to do my best to create such opportunities widely.

Design can’t be done if you’re stuck at home.

-What does “design” mean to you?

Katayama-san　Please wait a moment…. (After thinking about it) I think design is “communication”. I don’t think there is much difference between design and art, but if I had to say something, I would say that design has a partner. Art may be possible even if you are confined to your house, but I don’t think design can be done if you are confined to your house. Design is not complete without someone, and that someone may be someone who cannot hear us, or it may be the Deaflympics organizers, or it may be someone who can hear us who is looking forward to the Deaflympics. Through communication with everyone, I would like to create designs that make us laugh together.

Reiko Katayama / Born in Tokyo
Startup Manager

Born in 1991 in Omori, Tokyo. After completing the Graduate School of Film and New Media, Tokyo University of the Arts, she worked for a design production company and as a freelance designer before founding Hogaku Inc. in 2021. Based on the idea that “all disabilities exist in society, not in individuals,” she has developed many businesses as a social design company that changes the social structure through the power of design. The company employs approximately 4/5 of its employees who have some form of hearing disability, and pursues truly sought-after design by promoting projects from the perspective of those with hearing disabilities.
Together with Waseda University and Aisin Corporation, they jointly developed “Miruoto” and won the first prize at the start-up pitch event “UPGRADE with TOKYO” with the theme of “realization of a competition venue where people can see and feel sounds”.

X：@denkiry（https://twitter.com/denkiry）
Instagram：denkiry（https://www.instagram.com/denkiry/）