Self-labeling of Fully Mediating Representations by Graph Alignment | SpringerLink
Skip to main content
Springer Nature Link
Log in

Self-labeling of Fully Mediating Representations by Graph Alignment

  • Conference paper
  • First Online:
Artificial Intelligence and Machine Learning (BNAIC/Benelearn 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1530))

Included in the following conference series:

Abstract

To be able to predict a molecular graph structure (W) given a 2D image of a chemical compound (U) is a challenging problem in machine learning. We are interested to learn \(f: U \rightarrow W\) where we have a fully mediating representation V such that f factors into \(U \rightarrow V \rightarrow W\). However, observing V requires detailed and expensive labels. We propose graph aligning approach that generates rich or detailed labels given normal labels W. In this paper we investigate the scenario of domain adaptation from the source domain where we have access to the expensive labels V to the target domain where only normal labels W are available. Focusing on the problem of predicting chemical compound graphs from 2D images the fully mediating layer is represented using the planar embedding of the chemical graph structure we are predicting. The empirical results show that, using only 4000 data points, we obtain up to 4x improvement of performance after domain adaptation to target domain compared to pretrained model only on the source domain. After domain adaptation, the model is even able to detect atom types that were never observed in the original source domain. Finally, on the Maybridge data set the proposed self-labeling approach reached higher performance than the current state of the art.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9151
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11439
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Code available: https://github.com/biolearning-stadius/chemgrapher-self-rich-labeling.

References

  1. Fork of the official sources for the RDKit library (2020). https://github.com/biolearning-stadius/rdkit. Accessed 12 Nov 2020

  2. Baron, R.M., Kenny, D.A.: The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J. Pers. Soc. Psychol. 51(6), 1173 (1986). https://doi.org/10.1037/0022-3514.51.6.1173

    Article  Google Scholar 

  3. Belli, D., Kipf, T.: Image-conditioned graph generation for road network extraction. arXiv preprint arXiv:1910.14388 (2019)

  4. Bruzzone, L., Marconcini, M.: Domain adaptation problems: a DASVM classification technique and a circular validation strategy. IEEE Trans. Pattern Anal. Mach. Intell. 32(5), 770–787 (2009). https://doi.org/10.1109/TPAMI.2009.57

    Article  Google Scholar 

  5. Dalby, A., et al.: Description of several chemical structure file formats used by computer programs developed at Molecular Design Limited. J. Chem. Inf. Comput. Sci. 32(3), 244–255 (1992). https://doi.org/10.1021/ci00007a012

    Article  Google Scholar 

  6. Das, D., Lee, C.S.G.: Graph matching and pseudo-label guided deep unsupervised domain adaptation. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 342–352. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_34

    Chapter  Google Scholar 

  7. Filippov, I.V., Nicklaus, M.C.: Optical structure recognition software to recover chemical information: OSRA, an open source solution. J. Chem. Inf. Model. 49(3), 740–743 (2009). https://doi.org/10.1021/ci800067r

    Article  Google Scholar 

  8. Gao, L., Song, W., Dai, J., Chen, Y.: Road extraction from high-resolution remote sensing imagery using refined deep residual convolutional neural network. Remote Sens. 11(5), 552 (2019). https://doi.org/10.3390/rs10091461

    Article  Google Scholar 

  9. Gaulton, A., et al.: The ChEMBL database in 2017. Nucleic Acids Res. 45(D1), D945–D954 (2017). https://doi.org/10.1093/nar/gkw1074

    Article  Google Scholar 

  10. Gold, S., Rangarajan, A.: Graduated assignment graph matching. In: Proceedings of International Conference on Neural Networks (ICNN 1996), vol. 3, pp. 1474–1479. IEEE (1996). https://doi.org/10.1109/34.491619

  11. Han, C., Mao, J., Gan, C., Tenenbaum, J., Wu, J.: Visual concept-metaconcept learning. In: Advances in Neural Information Processing Systems, pp. 5001–5012 (2019)

    Google Scholar 

  12. Henry, C., Azimi, S.M., Merkle, N.: Road segmentation in SAR satellite images with deep fully convolutional neural networks. IEEE Geosci. Remote Sens. Lett. 15(12), 1867–1871 (2018). https://doi.org/10.1109/LGRS.2018.2864342

    Article  Google Scholar 

  13. Kisku, D.R., Rattani, A., Grosso, E., Tistarelli, M.: Face identification by sift-based complete graph topology. In: 2007 IEEE Workshop on Automatic Identification Advanced Technologies, pp. 63–68. IEEE (2007). https://doi.org/10.1109/AUTOID.2007.380594

  14. Kong, X., Zhang, J., Yu, P.S.: Inferring anchor links across multiple heterogeneous social networks. In: Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, pp. 179–188 (2013). https://doi.org/10.1145/2505515.2505531

  15. Kouw, W.M., Loog, M.: A review of domain adaptation without target labels. IEEE Trans. Pattern Anal. Mach. Intell. 43(3), 766–785 (2019). https://doi.org/10.1109/TPAMI.2019.2945942

    Article  Google Scholar 

  16. Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. Q. 2(1–2), 83–97 (1955). https://doi.org/10.1002/nav.3800020109

    Article  MathSciNet  MATH  Google Scholar 

  17. Locatello, F., et al.: Object-centric learning with slot attention. arXiv preprint arXiv:2006.15055 (2020)

  18. Sadawi, N.M., Sexton, A., Sorge, V.: Chemical structure recognition: a rule based approach. In: Proceedings of SPIE, vol. 8297, p. 32, January 2012. https://doi.org/10.1117/12.912185

  19. Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., Wu, J.: The neuro-symbolic concept learner: interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584 (2019)

  20. McDaniel, J.R., Balmuth, J.R.: Kekule: OCR-optical chemical (structure) recognition. J. Chem. Inf. Comput. Sci. 32(4), 373–378 (1992). https://doi.org/10.1021/ci00008a018

    Article  Google Scholar 

  21. Oldenhof, M., Arany, A., Moreau, Y., Simm, J.: ChemGrapher: optical graph recognition of chemical compounds by deep learning. J. Chem. Inf. Model. 60(10), 4506–4517 (2020). https://doi.org/10.1021/acs.jcim.0c00459

    Article  Google Scholar 

  22. Park, J., Rosania, G., Shedden, K.A., Nguyen, M., Lyu, N., Saitou, K.: Automated extraction of chemical structure information from digital raster images. Chem. Cent. J. 3 (2009). Article number: 4. https://doi.org/10.1186/1752-153X-3-4

  23. Pearl, J.: Causal diagrams for empirical research. Biometrika 82(4), 669–688 (1995). https://doi.org/10.1093/biomet/82.4.669

    Article  MathSciNet  MATH  Google Scholar 

  24. Pearl, J.: Direct and indirect effects. In: Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pp. 411–420 (2001)

    Google Scholar 

  25. Pearl, J.: Causality. Cambridge University Press, Cambridge (2009)

    Book  Google Scholar 

  26. Pearl, J., et al.: Models, Reasoning and Inference. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  27. Pérez, Ó., Sánchez-Montañés, M.: A new learning strategy for classification problems with different training and test distributions. In: Sandoval, F., Prieto, A., Cabestany, J., Graña, M. (eds.) IWANN 2007. LNCS, vol. 4507, pp. 178–185. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73007-1_22

    Chapter  Google Scholar 

  28. Peryea, T., Katzel, D., Zhao, T., Southall, N., Nguyen, D.T.: MOLVEC: open source library for chemical structure recognition. In: Abstracts of Papers of the American Chemical Society, vol. 258. American Chemical Society, Washington, DC (2019)

    Google Scholar 

  29. Sanfeliu, A., Fu, K.S.: A distance measure between attributed relational graphs for pattern recognition. IEEE Trans. Syst. Man Cybern. 3, 353–362 (1983). https://doi.org/10.1109/TSMC.1983.6313167

    Article  MATH  Google Scholar 

  30. Serratosa, F.: Fast computation of bipartite graph matching. Pattern Recogn. Lett. 45, 244–250 (2014). https://doi.org/10.1016/j.patrec.2014.04.015

    Article  Google Scholar 

  31. Serratosa, F.: Computation of graph edit distance: reasoning about optimality and speed-up. Image Vis. Comput. 40, 38–48 (2015). https://doi.org/10.1016/j.imavis.2015.06.005

    Article  Google Scholar 

  32. Serratosa, F.: Speeding up fast bipartite graph matching through a new cost matrix. Int. J. Pattern Recogn. Artif. Intell. 29(02), 1550010 (2015). https://doi.org/10.1142/S021800141550010X

    Article  MathSciNet  Google Scholar 

  33. Staker, J., Marshall, K., Abel, R., McQuaw, C.M.: Molecular structure extraction from documents using deep learning. J. Chem. Inf. Model. 59(3), 1017–1029 (2019). https://doi.org/10.1021/acs.jcim.8b00669. ISSN: 1549-9596

    Article  Google Scholar 

  34. Valko, A.T., Johnson, A.P.: CLiDE Pro: the latest generation of CLiDE, a tool for optical chemical structure recognition. J. Chem. Inf. Model. 49(4), 780–787 (2009). https://doi.org/10.1021/ci800449t

    Article  Google Scholar 

  35. Vezhnevets, A., Ferrari, V., Buhmann, J.M.: Weakly supervised structured output learning for semantic segmentation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 845–852. IEEE (2012). https://doi.org/10.1109/CVPR.2012.6247757

  36. Weininger, D.: SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci. 28(1), 31–36 (1988). https://doi.org/10.1021/ci00057a005

    Article  Google Scholar 

  37. Willett, P., Barnard, J.M., Downs, G.M.: Chemical similarity searching. Chem. Inf. Comput. Sci. 38(6), 983–996 (1998). https://doi.org/10.1021/ci9800211

    Article  Google Scholar 

  38. Xu, J., Schwing, A.G., Urtasun, R.: Tell me what you see and i will show you where it is. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3190–3197 (2014)

    Google Scholar 

  39. Xu, J., Schwing, A.G., Urtasun, R.: Learning to segment under various forms of weak supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3781–3790 (2015)

    Google Scholar 

  40. Xu, Z., Huang, S., Zhang, Y., Tao, D.: Augmenting strong supervision using web data for fine-grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2524–2532 (2015)

    Google Scholar 

  41. Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., Tenenbaum, J.: Neural-symbolic VQA: disentangling reasoning from vision and language understanding. In: Advances in Neural Information Processing Systems, pp. 1031–1042 (2018)

    Google Scholar 

  42. Zhang, H., Xiao, J., Quan, L.: Supervised label transfer for semantic segmentation of street scenes. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6315, pp. 561–574. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15555-0_41

    Chapter  Google Scholar 

Download references

Acknowledgments

MO, AA, YM, and JS are funded by (1) Research Council KU Leuven: C14/18/092 SymBioSys3; CELSA-HIDUCTION, (2) Innovative Medicines Initiative: MELLODDY, (3) Flemish Government (ELIXIR Belgium, IWT: PhD grants, FWO 06260) and (4) Impulsfonds AI: VR 2019 2203 DOC.0318/1QUATER Kenniscentrum Data en Maatschappij. Computational resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government - department EWI. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

Author information

Authors and Affiliations

  1. ESAT - STADIUS, KU Leuven, 3001, Leuven, Belgium

    Martijn Oldenhof, Adam Arany, Yves Moreau & Jaak Simm

Authors
  1. Martijn Oldenhof
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adam Arany
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yves Moreau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jaak Simm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martijn Oldenhof .

Editor information

Editors and Affiliations

  1. University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Luis A. Leiva

  2. Luxembourg Institute of Science and Technology, Esch-sur-Alzette, Luxembourg

    Cédric Pruski

  3. University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Réka Markovich

  4. University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Amro Najjar

  5. University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Christoph Schommer

A Appendix

A Appendix

1.1 A.1 Architecture Summary of Graph Recognition Tool

Every iteration of our method we need to train the graph recognition tool described in Oldenhof et al. [21]. This graph recognition tool is built using a combination of different convolutional neural networks. The first part is a semantic segmentation network to pixel-wise predict every atom, bond and charge type. The second part consists of three classification networks to classify every segment predicted by the semantic segmentation network. After the first step of the ChemGrapher model [21], the segmentation network (Table 3), the predicted segments are processed so that for every segment the center of mass is calculated. These centers of mass would be the atom/bond/charge candidates to be classified by the classification networks (Table 4).

Table 3. Summary of the layers of the segmentation network
Full size table
Table 4. Different layers in the classification network
Full size table

1.2 A.2 Training Details for Graph Recognition Tool

Training details of the graph recognition tool for every iteration of our method are summarized in Table 5. The input images used for training of the different networks are a mix if images from source domain and upsampled rich labeled images from target domain. For pretraining of the ChemGrapher model only images from source domain were used. The training was performed using a compute node with 2 NVIDIA v100 GPUs with 32 GB of memory.

1.3 A.3 Computational Cost per Rich-Labeling Iteration

In the following Table 6 the computational cost for 1 rich-labeling iteration is summarized including all steps: (re)training, predicting and graph aligning rich-labeling.

Table 5. Training details for different networks
Full size table
Table 6. Computational costs per rich-labeling iteration
Full size table

1.4 A.4 Examples of Cases Where Graph Alignment Fails

We would like to showcase some examples where the constrained (max 2 node substitutions or max 1 edge substitution) graph alignment fails. At the same time it is important to note that our proposed domain adaptation method is an iterative method, so if a graph alignment fails in a previous iteration it could succeed in a next one when the new model makes a new graph prediction closer to the true graph.

Fig. 7.
figure 7

Example 1: It is clear that to align the graph prediction \(W'\) with the true graph W more than 2 node substitutions are needed. So no rich labeling is possible for this example in this iteration.

Full size image
Fig. 8.
figure 8

Example 2: It is clear that alignment of the graph prediction \(W'\) with the true graph W can not be solved with only substitutions.

Full size image
Fig. 9.
figure 9

Example 3: It is clear that alignment of the graph prediction \(W'\) with the true graph W can not be solved with only substitutions.

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Oldenhof, M., Arany, A., Moreau, Y., Simm, J. (2022). Self-labeling of Fully Mediating Representations by Graph Alignment. In: Leiva, L.A., Pruski, C., Markovich, R., Najjar, A., Schommer, C. (eds) Artificial Intelligence and Machine Learning. BNAIC/Benelearn 2021. Communications in Computer and Information Science, vol 1530. Springer, Cham. https://doi.org/10.1007/978-3-030-93842-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-93842-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-93841-3

  • Online ISBN: 978-3-030-93842-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics