The future of education

Chemistry and Biology Education Using Commodity Web Augmented Reality -in any device!

Thanks to modern educational content using commodity web-based augmented reality in our moleculARweb site and in Google, students, teachers, and science communicators can now more easily see, grasp, communicate, and interact with the most complex concepts of chemistry and biology. For free, using their smartphones, tablets, and computers, in several languages.

=== The new tools presented here involve contributions mainly from Fabio Cortes and other members of the Dal Peraro lab at @EPFL. ===

Index

1. Introduction
2. Activities available in MoleculARweb and how to use them
3. Examples of 3D objects available in Google to explore them in augmented reality
4. Example of use in science communication: from atoms to tissues and back to atoms
5. Concluding remarks
6. References and links

1. Introduction

Tangible, sizeable models of entities of abstract nature, or imperceptible due to their small size, or of difficult availability/acquisition, are extremely useful for teaching and self-learning, as they make the subject concepts and entities more intelligible. This is especially important in the approach to chemistry, where teachers and students are confronted with complex, hard-to-visualize concepts. Concrete examples range from the shapes of pure atomic orbitals and their combinations to give hybrid and molecular orbitals, to the organization of atoms in space especially in chiral systems, including also the fact that molecules are flexible entities that explore various conformations, passing through the dynamic character of equilibria, the changes of protonation states, the distribution of molecular properties such as charge or aromaticity, and in turn the interaction of all these and other elements in complex molecules such as those that form the building blocks of biology. In these and many other cases, visualization of the entities and mechanisms involved facilitates the communication and understanding of ideas. For this purpose, regular teaching materials and classroom presentations incorporate a variety of illustrations and graphics. But these have two drawbacks: on the one hand, many are two-dimensional representations of intrinsically three-dimensional entities; on the other hand, this material is usually static or in the best cases animated, but rarely interactive, i.e. neither teachers nor students can act on the chemical or biological entities in question. Many courses today partially make up for this deficiency by resorting to molecular computer graphics programs, which allow users to quickly visualize virtually any molecule in pseudo-3D (“pseudo” referring to the fact that the representation is usually on a flat screen). There are even several web programming libraries that allow the insertion of molecular graphics directly inside web pages, with which various educational resources have been created in different branches of chemistry and molecular biology. The main problem with these graphics programs is that they are limited to handling intrinsically three-dimensional objects by means of a mouse on a flat surface, which complicates simultaneous rotation and translation operations on two molecules; thus, it is practically impossible to compare pairs of molecules or to make them interact. In contrast, tangible models of plastic or other materials allow great accessibility and manipulability, where the educator or student can physically move molecules to understand their geometry, stereochemistry, and chirality, and even force rotations around simple bonds, all of which are more difficult to achieve with computer graphics programs. There are even physical models prepared in materials such as plastic that represent nucleic acids and proteins, physical models of cells, and some educators have developed 3D-printed models of atomic orbitals (1). But physical models also have their disadvantages compared to graphics, starting with their relatively high cost, lack of details, and difficulty in modifying the display and annotating information. In particular, molecular models are limited in the numbers and types of atoms available, in addition to the fact that the user must assemble the models which implies the possibility of error for large molecules. Besides, such kits are not largely available and commercial kits are quite costly, at least for students in developing countries.

In recent years, virtual reality (VR) and augmented reality (AR) technologies have become progressively more accessible, fostering many groups to develop software that integrates the best of the virtual world with the best of the physical world in terms of molecule manipulation and visualization. Today there is a varied offer of such applications, in which some limitations are identified in relation to their use: on the one hand, the most complete programs usually require special equipment that, when connected to a computer, provide very immersive VR or AR but at a cost of hundreds to thousands of dollars, which makes it unthinkable to achieve a good reach to students and educators, even in developed countries. On the other hand, the possibility of achieving augmented reality on modern phones, tablets, and computers by simply using the webcam of these devices, allows any user to achieve that experience, which is less immersive than that on VR headset yet much better than using standard computer graphics; and more importantly, they allow better spatial control of the visualization and studied objects. But even here most of the available programs have problems: almost all of them present only static displays, with little or no interactivity with the user; in addition, almost all of these programs and applications must be installed, which may not be possible on shared systems where users do not have administrator privileges -or the user may simply not want or trust the installation of applications.

Trying to bring together the best of the virtual and real worlds in an accessible and user-friendly way, we (myself and WebXR engineer Fabio Cortes) developed a website called moleculARweb (https://molecularweb.epfl.ch) that proposes a series of AR activities useful for teaching and studying various chemistry and molecular biology concepts.(2,3) The activities cover topics from general and inorganic chemistry to structural biology to organic chemistry and physical chemistry. The site does not consist of one-off activities but rather tools with which educators can demonstrate and set up exercises for their students. Each activity on moleculARweb uses a set of markers (Figure 1A) that are printed with common printers, cut, folded, and glued as shown in the instructional videos on the home page. When entering the web page for each activity the user must allow the web browser to use the webcam, and display the printed markers to the webcam. The system will pose the virtual objects relevant to the activity on the markers, which the user can then move to manipulate them in space (Figure 1B).

As shown in section 2, the activities on moleculARweb cover from the most fundamental chemistry (atoms, molecules, their orbitals, mechanics and interactions) to the structure of complex biological molecular assemblies (ribosomes, nucleosomes, viral particles). And as we exemplified in section 3, the Google portal makes available, to any user with a modern smartphone or tablet, models starting just one level above biological molecular assemblies on the scale of biological complexity: from organelles (mitochondria, Golgi apparatus) to cells, tissues and systems (lymphatic, nervous, etc.) and even several complete models of living creatures. Although these models are much less interactive than those of moleculARweb, they extend its content to put into the domain of biology. Thus, together the two tools (moleculARweb and Google-provided models) cover smoothly from chemistry to biology, as we exemplify in section 4 with a path connecting from concepts at the atomic level to models of complex biological systems, inspired by a real activity proposed to visitors of an open science event.

2. Activities available in moleculARweb and how to use them

The content of moleculARweb is divided into 4 thematic modules that we present here with some updates with respect to the version released in 2020, plus what we have called “Virtual Molecular Modeling Kit” of which we present here the prototype of a new version. The thematic modules are “Orbitals and molecular shapes”, “Hydrogen bonding, acids and bases”, “Atomic structure of biological macromolecules”, and “Large biomolecular assemblies”. Each module includes several activities that we have described in a recent article (2) and some of which we present in videos on the YouTube channel Molecules in web augmented reality (4). The activities in these modules (section 2.1) work with the double-sided markers that can be printed from the link given in Figure 1A. Some of these activities also have a markerless mode, less interactive but more immersive (section 2.3). On the other hand, the virtual molecular modeling kit (section 2.2) uses cube markers that can also be downloaded where indicated in Figure 1A (plus, some users provide ways to 3D-print the cube markers in Thingiverse!). In the molecular modeling kit the cube markers allow loading any pair of molecules from their atomic coordinates to visualize them in augmented reality, even simulate their motions, and see how they collide and interact.

2.1 Activities with flat markers on webcam-equipped smartphones, tablets, and computers All activities support a mode where virtual objects are moved with markers that users can print with a normal printer. When the user shows the markers to the webcam the web app will detect the patterns and place the corresponding virtual object(s) on top. The first time the user accesses an activity the browser asks if the use of the webcam is allowed; this must be enabled.

Examples of activities. Figure 2 shows an example of an activity for each module, also illustrating different modes of use. Figure 2A shows an sp³d hybrid orbital (activity “Atomic orbitals and VSEPR theory” from the module “Orbitals and molecular shapes”) manipulated on a phone, taking the picture directly as a screenshot. Figure 2B shows how two water molecules are manipulated by viewing them through the phone (activity “Equilibria of water protonation” from the module “Hydrogen bonding, acids and bases”); in this case with an external photo but which could well be taken directly as a screenshot from the phone itself. Figure 2C shows a user attaching a DNA molecule onto a protein molecule (activity “Protein-DNA complex” from the module “Atomic structure of biological macromolecules”) on a computer that only has a front camera so it only functions as a mirror. Figure 2D shows a study guide about nucleosomes that has an integrated marker so that the student can visualize it directly there in the smartphone (“Nucleosome” activity from the module on “Large biological assemblies”).

Controls. Figure 3A shows the most common controls to all activities, exemplified in a smartphone where there is a greater variety of them. Some of these controls are highlighted here with their possible utilities. First, in case there is more than one webcam, for example in phones with front and rear cameras, the user can switch them in real-time while working. Another control inverts only the image, which can be useful for example when studying enantiomerism (Figure 3B, example taken from the molecular modeling kit). Meanwhile, the rotation controls allow fine-tuning of the object orientation without moving the marker, which can be useful for example when the marker is embedded in a document prepared by the educator as in Figure 2D.

Each activity has some specific controls, usually at the bottom and in order to allow the user to choose between options. For example, Figure 3A shows a smartphone view where options for choosing which orbital of which molecule to visualize are shown at the bottom, along with a control that activates the rotation by hand and indications about the activity. In addition, the activities with static displays have a button with the text “3D” at the bottom right, which activates the markerless mode described in section 2.3. Finally, the virtual modeling kit has several special controls, described in section 2.2.

2.2 The virtual modeling kit with cube markers, also on webcam-equipped phones, tablets, and computers

The virtual modeling kit allows users to load any molecule from its atomic coordinates, to display it in 3D on a paper or cardboard cube that the user can move and rotate freely. Thanks to its six sides, it achieves better handling and visual stability than the flat markers. In addition to displaying the molecules, the modeling kit includes a very simple force field that allows visualization of the conformational changes caused by the coupling of thermal fluctuations. The original version of the virtual modeling kit uses the cube formed by the letters A, B, C, D, F, and a symbol in Figure 1A. Test version 2.0 of the virtual modeling kit works with one or two of the cube-like markers in Figure 1A (one marker is the same, the other is composed of the letters J, K, M, N, L, and P), allowing up to two molecules to be controlled independently.

With the molecular modeling kit the user can load any molecule from its atomic coordinates on each cube. Atomic coordinates can be taken from the preset list or constructed externally (in version 1) or even inside the activity itself (in version 2). Some controls are rearranged in version 2; for example, coordinate inversion can be run specifically on each marker, allowing the same chiral molecule to be loaded on both cubes and then inverting it on only one, useful for comparing chirality.

In addition, version 2 of the modeling kit incorporates a very simple force field that calculates and displays interactions between molecules, both collisions, and attractions. Figure 4 shows a prototype of version 2, where each cube shows a different molecule both interacting via two hydrogen bridges detected in real-time. Readers can now access this version directly from the moleculARweb home page.

In a future Medium story I will cover specifically the upcoming versions of the Virtual Modeling Kit. Stay tuned, because it has lots of features and a huge potential for education!

2.3 Viewing models “in your space” without markers, in smartphones that support WebXR

On smartphones modern enough to support WebXR (most phones available as of 2018), activities that do not have interactive content can also run in a markerless mode. In cases where this mode is available, the user will see a “3D” icon at the bottom right of the screen. When clicked, the activity displays an intermediate screen with the object in question and a button that activates the AR mode (in English it is “See in my environment”). Upon entering this mode the user must scan the floor, a table, or any other flat horizontal surface; the program will then place a virtual representation of the object and leave it static in one place, even if the user moves. This allows the user to explore the object from any angle and even “enter” the objects to see details. All this is accompanied by the ability to take screenshots and record videos from an external program installed on the phone such as AZ Screen Recorder.

Figure 5 shows how to activate the markerless AR mode for the model of a ribosome built from an experimental structure where it was captured with one segment of messenger RNA and two of transfer RNA. From a distance, the user can see the overall shape of the ribosome, the main subunits and its protein and nucleic components. But the user can also move closer and even into the ribosome to analyze its details, as here for example a codon of the mRNA interacting with a codon of one of the tRNAs (in each RNA molecule each base is represented by a stick).

3. Examples of 3D objects available in Google to explore them in augmented reality

The same markerless AR technology described above is exploited by Google to offer a wide variety of augmented reality educational content, openly provided by third parties. By searching on a WebXR-enabled phone, Google will return direct links to predefined 3D objects (note that searches work only in English; the same is true for annotations on models). Figure 6 shows examples where “muscular system” (Figure 6A) or concepts related to the structures of vascular plants (Figure 6B) have been searched. As in the markerless AR models of moleculARweb, in the Google models the user can move freely around the object once it is anchored to the detected surface. This allows exploring in detail certain structures as exemplified in both panels of Figure 6 (detail of the tendon insertion in 6A, and details of the chloroplast in 6B).

4. Example of use in science communication: from atoms to tissues and back to atoms

Finally, in Figure 7 we exemplify how to integrate some of the AR activities presented in the earlier sections into a complete experience for understanding the structure of living matter at different levels of complexity. This particular example begins by exploring the relationship between protonation states and charges, followed by visualizing electrostatic interactions that stabilize DNA-protein complexes, then visualizing how these are assembled into nucleosomes, then seeing chromosomes within the cell nucleus, and slowly zooming out to see a whole cell, then the circulatory system as an example of the tissue level, and finally zooming back into a cell, a mitochondrion, the heme group of a protein, and finally the 3dz² orbital of an atom.

5. Concluding remarks

With more than 26,000 accesses since its release, moleculARweb permeates schools and homes around the world (Figure 8) thanks to its availability in multiple languages, its operation on mobile devices and regular computers, and its easy direct access simply through the Internet like any other website because it is based exclusively on web programming (6-8). Published tests (9), automated online statistics, our own in-person evaluations, and feedback received from high school and university level teachers demonstrate the ease of use of the moleculARweb platform and its pedagogical reach. In this sense, many teachers expressed the importance of having this tool since it improves perception, interaction, and above all motivation on the part of the students, crucial to facilitate the understanding and learning of concepts that are difficult to visualize, both in chemistry and biology.

Google and content-creation companies are continually adding new models, now including animated models of biological processes and even entire animals. These technologies allow for more immersive, didactic learning than ever before, available to anyone having just a smartphone and internet connection. On our side, we are currently completing the virtual molecular modeling kit to formally release its version 2.0 and creating a system with which educators will be able to create their own web AR content for more complex objects. In future stories I will go deeper into the different modules of moleculARweb, and possibly on the programming aspects together with the main developer Fabio Cortes.

6. References and links

moleculARweb Website: https://molecularweb.epfl.ch/

YouTube videos about moleculARweb:

• Printing and preparing the markers: https://youtu.be/Fo0WpTmUJ4s
• Using the activities: https://youtu.be/GuYMqG8kce8
• Main channel, “Molecules in web augmented reality”: https://www.youtube.com/user/laa2515
• Key videos about moleculARweb: https://www.youtube.com/watch?v=myBC_f6OJDQ , https://www.youtube.com/watch?v=ntrLXEaHRUU , https://www.youtube.com/watch?v=SHAPsCMwPrA , https://www.youtube.com/watch?v=FYZWQlp7iF4 , and https://www.youtube.com/watch?v=vfiHI4uT_Wg

Peer-reviewed paper on the first release of moleculARweb: https://pubs.acs.org/doi/10.1021/acs.jchemed.1c00179

References

(1) de Cataldo, R.; Griffith, K. M.; Fogarty, K. H. Hands-On Hybridization: 3D-Printed Models of Hybrid Orbitals. J. Chem. Educ. 2018, 95 (9), 1601–1606. https://doi.org/10.1021/acs.jchemed.8b00078.

(2) Cortes Rodriguez, F.; Frattini, G.; Krapp, L.; Martinez-Huang, H.; Moreno, D.; Salomon, J.; Stemkoski, L.; Traeger, S.; Dal Peraro, M.; Abriata, L. MoleculARweb: A Website for Chemistry and Structural Biology Education Through Interactive Augmented Reality out of the Box in Commodity Devices. ChemRxiv 2020.

(3) Cortés Rodríguez, F.; Frattini, G.; Krapp, L. F.; Martinez-Hung, H.; Moreno, D. M.; Roldán, M.; Salomón, J.; Stemkoski, L.; Traeger, S.; Dal Peraro, M. MoleculARweb: A Web Site for Chemistry and Structural Biology Education through Interactive Augmented Reality out of the Box in Commodity Devices. J. Chem. Educ. 2021.

(4) Abriata, L. A. Moléculas en realidad aumentada https://www.youtube.com/channel/UCdhOVimtNZmri967KdTEAKg.

(5) Juszkiewicz, S.; Chandrasekaran, V.; Lin, Z.; Kraatz, S.; Ramakrishnan, V.; Hegde, R. S. ZNF598 Is a Quality Control Sensor of Collided Ribosomes. Mol. Cell 2018, 72 (3), 469–481. e7.

(6) Abriata, L. A. Building Blocks for Commodity Augmented Reality-Based Molecular Visualization and Modeling in Web Browsers. PeerJ Comput. Sci. 2020, 6, e260.

(7) Abriata, L. A. Web Apps Come of Age for Molecular Sciences. Informatics 2017, 4 (3), 28.

(8) Abriata, L. A.; Rodrigues, J. P. G. L. M.; Salathé, M.; Patiny, L. Augmenting Research, Education, and Outreach with Client-Side Web Programming. Trends Biotechnol. 2018, 36 (5), 473–476. https://doi.org/10.1016/j.tibtech.2017.11.009.

(9) Martínez-Hung, H.; García-López, A.; Escalona-Arranz, J. C. Modelos de Realidad Aumentada Aplicados a La Enseñanza de La Química En El Nivel Universitario. Rev. Cuba. Quím. 2017, 29 (1), 13–25.

I am a nature, science, technology, programming, and DIY enthusiast. Biotechnologist and chemist, in the wet lab and in computers. I write about everything that lies within my broad sphere of interests. Check out my lists for more stories. Become a Medium member to access all stories by me and other writers, and subscribe to get my new stories by email (original affiliate links of the platform).