Abstract
Dimensionality reduction has been considered as one of the most significant tools for data analysis. In general, supervised information is helpful for dimensionality reduction. However, in typical real applications, supervised information in multiple source tasks may be available, while the data of the target task are unlabeled. An interesting problem of how to guide the dimensionality reduction for the unlabeled target data by exploiting useful knowledge, such as label information, from multiple source tasks arises in such a scenario. In this paper, we propose a new method for dimensionality reduction in the transfer learning setting. Unlike traditional paradigms where the useful knowledge from multiple source tasks is transferred through distance metric, we attempt to learn a more informative mapping function between the original data and the reduced data by Gaussian process that behaves more appropriately than other parametric regression methods due to its less parametric characteristic. In our proposal, we firstly convert the dimensionality reduction problem into integral regression problems in parallel. Gaussian process is then employed to learn the underlying relationship between the original data and the reduced data. Such a relationship can be appropriately transferred to the target task by exploiting the prediction ability of the Gaussian process model and inventing different kinds of regularizers. Extensive experiments on both synthetic and real data sets show the effectiveness of our method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Actually, in the implementation, we are only interested in the learned kernel hyperparameters \(\theta ^i_l\)s.
References
Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans PAMI 19:711–720
Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15:1373–1396
Bonilla EV, Chai KMA, Williams CKI (2008) Multi-task Gaussian process prediction. In: NIPS 20, pp 153–160
Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge
Cai D, He X, Han J (2007) Semi-supervised discriminant analysis. In: ICCV
Cai D, He X, Han J (2007) Spectral regression: a unified approach for sparse subspace learning. In: ICDM, pp 73–82
Chen B, Lam W, Tsang I, Wong T (2009) Extracting discriminative concepts for domain adaptation in text mining. In: KDD, pp 179–188
Duan L, Tsang IW, Xu D, Chua T (2009) Domain adaptation from multiple sources via auxiliary classifiers. In: ICML
Li T, Zhang C, Wang F, Chen S (2008) Semi-supervised metric learning by maximizing constraint margin. In: CIKM, pp 1457–1458
Jurie F, Cevikalp H, Verbeek J, Klaser A (2008) Semi-supervised dimensionality reduction using pairwise equivalence constraints. In: VISAPP, pp 489–496
He X, Cai D, Yan S, Zhang H-J (2005) Neighborhood preserving embedding. In: ICCV, pp 1208–1213
He X, Yan S, Hu Y, Niyogi P, Zhang H-J (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340
Jin R, Wang S, Zhou Y (2009) Regularized distance metric learning: theory and algorithm. In: NIPS, pp 862–870
Maaten LJP, Postma EO, Herick HJ (2009) Dimensionality reduction: a comparative review. Technical report TiCC-TR 2009–005, Tilburg University
Pan SJ, Kwok JT, Yang Q (2008) Transfer learning via dimensionality reduction. In: AAAI, pp 677–682
Pan SJ, Tsang IW, Kwok JT, Yang Q (2009) Domain adaptation via transfer component analysis. In: IJCAI, pp 1187–1192
Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans TKDE 22:1345–1359
Rasmussen CE, Williams CKI (2006) Gaussian processes for machine learning. MIT Press, Cambridge
Rish I, Grabarnik G, Cecchi G, Pereira F, Gordon GJ (2008) Closed-form supervised dimensionality reduction with generalized linear models. In: ICML, pp 832–839
Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290:2323–2326
Sajama, Orlitsky A (2005) Supervised dimensionality reduction using mixture models. In: ICML, pp 768–775
Samaria FS, Harter AC (1994) Parameterisation of a stochastic model for human face identification. In: Applications of computer vision, 1994, Proceedings of the second IEEE workshop on, pp 138–142
Slonim N, Tishby N (2000) Document clustering using word clusters via the information bottleneck method. In: SIGIR, pp 208–215
Sugiyama M, Idé T, Nakajima S, Sese J (January 2010) Semi-supervised local fisher discriminant analysis for dimensionality reduction. J Mach Learn 78:35–61
Tenenbaum JB, de Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290:2319–2323
Tong B, Shao H, Chou B-H, Suzuki E (2010) Semi-supervised projection clustering with transferred centroid regularization. In: ECML/PKDD (3), pp 306–321
Tong B, Suzuki E (2010) Subclass-oriented dimension reduction with constraint transformation and manifold regularization. In: PAKDD (2), pp 1–13
Wang Z, Song Y, Zhang C (2008) Transferred dimensionality reduction. In: ECML/PKDD, pp 550–565
Yang X, Fu H, Zha H, Barlow J (2006) Semi-supervised nonlinear dimensionality reduction. In: ICML, pp 1065–1072
Zha Z, Mei T, Wang M, Wang Z, Hua XS (2009) Robust distance metric learning with auxiliary knowledge. In: IJCAI, pp 1327–1332
Zhang D, Zhou ZH, Chen S (2007) Semi-supervised dimensionality reduction. In: SDM, pp 629–634
Zhang Y, Yeung D-Y (2010) A convex formulation for learning task relationships in multi-task learning. In: UAI, pp 733–742
Zhang Y, Yeung D-Y (2010) Transfer metric learning by learning task relationships. In: KDD, pp 1199–1208
Zhu X (2005) Semi-supervised learning literature survey. Technical report 1530, Computer Sciences, University of Wisconsin-Madison
Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B 67(2):301–320
Gower JC, Dijksterhuis GB (2004) Procrustes problem. Oxford University Press, Oxford
Chen Y, Manjeet R, Ming D, Jing H (2008) Non-negative matrix factorization for semi-supervised data clustering. Knowl Inf Syst 17(3):355–379
Keogh E, Chakrabarti K, Pazzani M, Mehrotra S (2000) Dimensionality reduction for fast similarity search in large time series databases. Knowl Inf Syst 3(3):263–286
Song G, Cui B, Zheng B, Xie K, Yang D (2008) Accelerating sequence searching: dimensionality reduction method. Knowl Inf Syst 20(3):301–322
Tong B, Gao J, Thach NH, Suzuki E (2011) Gaussian process for dimensionality reduction in transfer learning In: SDM, pp 783–794
Zhou Z-H, Li M (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3):415–439
Zhao W, He Q, Ma H, Shi Z (2011) Effective semi-supervised document clustering via active learning with instance-level constraints. Knowl Inf Syst 30(3):569–587
Yan F, Qi Y (2010) Sparse Gaussian process regression via L1 penalization. In: ICML, pp 1183–1190
Si S, Gao J, Tao D, Geng B (2010) Bregman divergence-based regularization for transfer subspace learning. IEEE Trans Knowl Data Eng 22(7):929–942
Author information
Authors and Affiliations
Corresponding author
Additional information
This work is supported by the grant-in-aid for scientific research on fundamental research (B) 21300053 from the Japanese Ministry of Education, Culture, Sports, Science and Technology, and Charles Sturt University Competitive Research Grant OPA 4818. This work was partially supported by the National Science Foundation (Grant No. 61133016), and the National High Technology Joint Research Program of China (863 Program, Grant No. 2011AA010706).
Rights and permissions
About this article
Cite this article
Tong, B., Gao, J., Nguyen Huy, T. et al. Transfer dimensionality reduction by Gaussian process in parallel. Knowl Inf Syst 38, 567–597 (2014). https://doi.org/10.1007/s10115-012-0601-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-012-0601-y