Motion is a significant symbol of living beings, and emotion is a unique feature of human beings. With the rapid expansion of human society, there is a growing need for real⁃time motion and emotion sensing in various real⁃world applications such as intrusion detection, patient sleep monitoring, elder emotion companion and autism treatment. To date, vision⁃based and sensor⁃based sensing methods are the dominating solutions in this area. The former usually employs surveillance cameras to take footage, from which gradient⁃based features are extracted for motion and emotion recognition, while the latter explores virtual footprints caused by human beings from specific sensors for recognizing human activities and moods.
Though significant progress has been obtained, the related research still faces several challenges, e.g., the availability issue (specialized hardware), the reliability issue (light and line⁃of⁃sight (LOS) restrictions), and the privacy issue (physical contacts and feeling violated when being watched). Therefore, researchers are struggling for new paradigms to revolutionize the traditional solutions. The rapid development of wireless communications and embedded platforms has greatly accelerated the approaches for the Internet of Things (IoT) and envisioned a new era of ubiquitous computing. The ever⁃increasing smart devices surrounding our daily lives have offered a great opportunity for building a big data driven system that can recognize human⁃centric motions anytime anywhere.
This special issue deals with the overview, technology, and applications of using big data from various IoT devices for motion and emotion sensing, and aims to stimulate research and development of related areas by providing a unique forum for scientists, engineers, broadcasters, manufacturers, software developers, and other related professionals . The topics addressed in this special issue are diversified, including an overview of current research progresses on IoT, real⁃world applications, security issues, etc. The call⁃for⁃papers for this special issue attracted a number of excellent submissions. After two⁃round reviews, seven papers were selected for publication. These papers are organized in three groups. The first group comprises one overview paper that addresses the most fundamental question in the affective computing, i.e., how humans perceive emotion. The second group consists of four papers addressing the emotion sensing issues by leveraging big data from different perspectives, e.g., electroencephalogram (EEG) sensors, galvanic skin reaction (GSR) wearable sensors, social big data, etc. The last group comprises two papers that present basic methodologies to ensure reliable and accurate big data for emotion sensing.
The first paper “How Humans Perceive Emotion?” directly studies the most fundamental issue of emotion perceiving and reviews behavioral as well as neural findings in human emotion perception, including facial emotion perception, olfactory emotion perception, multimodal emotion perception, and the time course of emotion perception. The knowledge of how humans perceive emotion will help bring artificial intelligence strides closer to human intelligence.
The most⁃commonly used data sources in affective computing are psychological sensors like EEG and electrocardiogram (ECG) sensors. The second paper “Emotion Judgment System by EEG Based on Concept Base of EEG Features” proposes an emotion judgment system by using the EEG feature concept base with premise of noises included. The proposed system is testified on a private EEG dataset and compared with the traditional support vector machine (SVM) classifier. The results proves the efficiency of the proposed system.
The third paper “Emotion and Cognitive Reappraisal Based on GSR Wearable Sensor” studies a similar problem with a different data source, i.e., GSR wearable sensors. Based on the authors’ previous work, they establish a novel GSR emotion sensing system for predicting the emotional state transition and considering the correlation between GSR signals and emotions.
The fourth paper “Multimodal Emotion Recognition with Transfer Learning of Deep Neural Network” tackles the emotion recognition problem from the aspect of multimodality including both images and audios, where audio is encoded using deep speech recognition networks with 500 hours’ speech and video is encoded using convolutional neural networks (CNNs) with over 110,000 images. The following experiments reveal several interesting observations. For example, audio features extracted from deep speech recognition networks can achieve better performance than handcrafted audio features.
Another emerging source for emotion analysis is data from the fastly expanding social networks. The fifth paper “Emotion Analysis on Social Big Data” describes a new method of emotion analysis based on social big data, e.g., multilingual web corpora and its annotated emotion tags. To handle the collected social big data, the authors design an emotion analysis model that integrates the high⁃quality emotion corpus and the automatic⁃constructed corpus created in their past studies, and then tries to analyze a large⁃scale corpus consisting of Twitter tweets based on the model. The proposed system is evaluated via time⁃series analysis on the large⁃scale corpus and model evaluation, and demonstrated effective over previous solutions.
Furthermore, the reliability of social big data is also an emerging problem. Therefore, the sixth paper “Measuring QoE of Web Service with Mining DNS Resolution Data” proposes a novel solution called the First Webpage Time (FWT) algorithm to measure the QoE of the web service. The proposed FWT algorithm is analyzed in theory that its precision is guaranteed. The experiments based on the ISP’s DNS resolution data are carried out to evaluate the proposed FWT algorithm.
Other than reliable data sources, effective classifiers are also important to the emotion analysis system. Therefore, the last paper “An Improved K⁃means Algorithm Based on Initial Clustering Center Optimization” targets on the widely⁃used K⁃means algorithm and proposes an effective algorithm to select the initial clustering center for eliminating the uncertainty of the central point selection.
As we conclude the introduction of this special issue, we would like to thank all the authors for their valuable contributions, and we express our sincere gratitude to all the reviewers for their timely and insightful expert reviews. It is hoped that the contents in this special issue are informative and useful from the aspects of technology, standardization, and implementation.