Welcome to Pengyu Hong's Homepage

Automatic Temporal Pattern Extraction and Association

The applications of this research include video summarization, automatic event detection and recognition, automatic concept learning, and so on.

Publications:

Overview:

Introduction: 

A temporal pattern is defined as a segment of signals that recurs frequently in the whole temporal signal sequence. For example, the temporal signal sequences could be the movements of head, hand, and body, a piece of music, and so on. The patterns of the body movement represent the habit of a person. The patterns of the music represent the melodic phases. Those patterns encode the characteristics of the original temporal sequence and can be used for data summarization and pattern detection.

The extracted patterns encode the characteristics of the original temporal sequence and can be used for data summarization and pattern detection. Since activities cohere typically beyond modalities, a stand-alone pattern is seldom meaningful. The correlations among the patterns across modalities endow the pattern with meanings.

This research investigate how to automatically extract temporal patterns and automatically associate the patterns that extracted from different modalities.

My Approach:

We present an approach for automatic temporal pattern extraction. Threshold Fuzzy State Network is used to model the temporal signal sequence. Scale Pursuing Process is proposed to train the Threshold Fuzzy State Network. The trained Threshold Fuzzy State Network is able to identify the sub-patterns of the patterns. The sub-patterns are then grown into the whole patterns.

The proposed approach is applied to automatic analysis of multiple synchronized temporal sequences from several input modalities. The timing information of the extracted patterns are used to learn the correlations of the patterns across multiple modalities. The learning results lead to automatic concept learning.

Examples:

(1) Automatically model and extract temporal pattern from a mouse trajectory. The user uses mouse to draw two patterns. The mouse trajectory is uniformly sampled.

(1.1) The data

(1.2) The intermediate and final results of mouse pattern extraction are shown as following.

(1.2.1) Local search results

 (Click to see a video clip showing an example of the local search)

(1.2.2) Global search results

After eight iteration, the algorithm extracted the results as below.

(2) Automatically extract and associate temporal patterns from two synchronized temporal signal sequences. One of the signal sequence is the mouse trajectory, which is captured while draw the two pattern shown above. The other one is the annotation sound track, which is recorded synchronously with the mouse trajectory.