2713_C002.fm Page 19 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto: An Application in Ethnographic Studies Hiroya Tanaka, Masatoshi Arikawa, Ryosuke Shibasaki, and Yuki Konagaya CONTENTS 2.1 Introduction 19 2.2 Related Concepts 22 2.2.1 Photo Collage 22 2.2.2 Hypermedia 23 2.3 STAMP: A Tool for Hyperphoto Collage 25 2.3.1 Concepts and System Overview 25 2.3.2 STAMP-Maker 27 2.3.3 STAMP-Navigator 29 2.3.4 Visualization Options 30 2.4 An Application 30 2.5 Conclusion 32 References 33 2.1 Introduction In the field of ethnography, researchers deal with a large number of photos A researcher’s mission is to stay in developing countries, often in Asia or Africa, and describe their environment while sharing experiences with native people During their stay, they try to record their everyday life in detail (Konagaya, 1991; Konagaya, 1998; Konagaya, 1999) Some researchers will stay for about one year in the target locale and take more than 100 photos per day Photos are a particularly useful medium for establishing 19 Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 20 Monday, September 26, 2005 7:35 AM 20 GIS-based Studies in the Humanities and Social Sciences a record of places, people, life, and atmosphere in a target area because they provide a rich source of information about spaces or places in the real world The main advantages of photos in such studies are: • Instant acquisition: An ordinary camera can be used quickly and easily • Rich information: Far from representing abstract data, photos are genuine records of real-world phenomena • Subjectivity: A photo often reveals not only the photographer’s target, but also the subjective point of interest (POI) of the photographer • Scalability: Photos include micrographs, landscape photos, satellite photos, and so on Photos thus enable us to examine the real world on any spatial scale While photos are clearly the most important medium for such studies, several problems make it difficult to effectively manage large collections of photos These problems are related to two aspects of the way we manage photos: • Organization and association • Browsing and interaction A few methods for organizing and associating photos have already been developed For example, one method is to plot locations of photos onto two-dimensional (2D) maps like a typical geographical information systems (GIS) (Figure 2.1) If the location (latitude and longitude) where each photo is taken can be recorded using global-positioning system (GPS) and other technologies, we can plot locations of photos onto a map almost automatically However, such conventional methods not enable us to sufficiently understand the relationships among photos With regard to browsing and interaction, two general approaches are usually taken — the album approach and the slide show approach (Figure 2.2) However, neither of these is essentially interactive or sufficient to describe the dynamic experiences of the photographer In this chapter, we describe a new method for managing photos through computers We have developed and implemented a software that we call STAMP (Spatio-Temporal Association with Multiple Photos) [Tanaka (2002a), Tanaka (2002b), Tanaka (2003)] STAMP allows us to easily create, publish, share, and navigate pseudo-three-dimensional (3D) spaces comprised of multiple photos With this software, users can create not only intensive scenes, but also extensive 3D scenes, which enable interactive and continuous navigation Furthermore, users can relive their own or other’s experiences sequentially Researchers can manage photos based on the context of the photos by using this software Our software can solve Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 21 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 21 FIGURE 2.1 Photos plotted their locations onto a 2D map Time FIGURE 2.2 An album approach and a slide show approach several of the problems mentioned above and advance the use of photo applications In the following sections, we describe two important concepts that underlie our approach — photo collage and hypermedia In Section 2.3, we explain STAMP and discuss the characteristics of STAMP In Section 2.4, we report on an experiment using STAMP, and then conclude in the final section Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 22 Monday, September 26, 2005 7:35 AM 22 GIS-based Studies in the Humanities and Social Sciences FIGURE 2.3 An example of 2D photo collage 2.2 2.2.1 Related Concepts Photo Collage Our approach is based on artistic representation through the concept of photo collage on a 2D canvas; that is, a general method of organizing and associating photos People can connect, overlap, and rotate photos on the 2D canvas to compose a single image that represents a memorable sight or event in the real world Figure 2.3 shows an example of our photo collage David Hockney, a famous artist who created so many photo collages, said that photo collage clearly reveals the photographer’s perception of the world (Hockney, 1985) David named the series of his photo collage works “Moving Focus.” This naming implies his notion about photo collage Photo collage is an effective way to assemble multiple photos into one complete image Moreover, we can read through multiple scenarios in a single photo collage Figure 2.4 shows an example of the visual paths in our photo collage The photo collage approach has shortcomings, though, in that it is only a static, 2D graphic representation Our proposed representation is a pseudoCopyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 23 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 23 FIGURE 2.4 Visual paths in a photo collage 3D and interactive representation, the details of which will be introduced in Section 2.3.3 2.2.2 Hypermedia Our approach is also based on the hypermedia concept In computer science, the hypermedia concept was a remarkable discovery The World Wide Web (WWW), which most of us use daily, is based on hypermedia The hypermedia structure is very simple Only two elements, nodes and hyperlinks, are used to create hypermedia We can apply various kinds of media as nodes, such as texts, images, sounds, movies, and so on Historically, hypermedia concept was derived from hypertext concept Simply saying, hypertext is a content network comprised of texts and hyperlinks (Figures 2.5 and 2.6) There are several views regarding hypertext (Peter, 2003) One view of hypertext technology is that it is an authoring method In other words, hypertext is a data-organization paradigm It augments raw data with a rich, semantically meaningful structure Hypertext also provides the means to create a context for data This viewpoint implies that the meaning of data is not inherent in the data itself, but rather resides in the structures into which the data is incorporated Data is, in itself, only useful when set into a context, structured, and reinterpreted to play a role in a particular setting Some researchers view hypertext technology as an interaction paradigm; for example, browsing as an alternative to query We can browse data by continuously Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 24 Monday, September 26, 2005 7:35 AM 24 GIS-based Studies in the Humanities and Social Sciences Node1 Node2 Hyperlink Starting anchor Ending anchor FIGURE 2.5 Hypertest structure based on nodes and hyperlink FIGURE 2.6 Hypertext network following hyperlinks between nodes Each path, which typically consists of several hyperlinks, tells us an inherent story With the popularization of various media, such as images, sounds, and movies, the hypertext concept has been extended to the hypermedia concept Today, we can associate various kinds of media, which are not only texts but also images, sounds, and movies, by using hyperlinks Hypermedia is easy to use for both authoring and browsing, so it has achieved worldwide acceptance and has enabled the remarkable growth of the WWW Copyright © 2006 Taylor & Francis Group, LLC   2713_C002.fm Page 25 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 25 Our STAMP incorporates the hypermedia concept Our software is based on the hyperphoto concept, which can be considered a kind of hypermedia The details of our hyperphoto concept will be discussed in the following section 2.3 2.3.1 STAMP: A Tool for Hyperphoto Collage Concepts and System Overview Until now, there has been no direct linkage between the photo collage and hypermedia, discussed in Section 2.2 The former is a technique used in the field of art, while the latter is a technique used in the field of computer science However, we have found that these two techniques are similar in important respects That is, both are easy to use and enable us to create multiple scenarios from raw data through simple operations The essential difference between the two techniques is that a photo collage represents a continuous space on a 2D plane, and hypermedia is based on a discrete structure over a computer network We have tried to merge the advantages of both the photo collage and hypermedia in a concept We named our concept hyperphoto collage, which is a combination of hyperphoto and photo collage Hyperphoto means a photo that contains hyperlinks to others Multiple hyperphotos can form hyperphoto networks, which can be considered a kind of hypermedia In our system, users can create spatial associations between raw photos through simple operations Spatial-hyperlink, which is an extension of ordinary hyperlink, is used for creating an association between two photos We will introduce the details of spatial-hyperlink in Section 2.3.2 Figure 2.7 shows the data structure of hyperphoto networks The ability to make hyperphoto networks is an authoring technique of our system Moreover, our system automatically makes a pseudo-3D photo collage from hyperphoto and displays it (Figure 2.8) This allows us to navigate in pseudo-3D space interactively and continuously This is our browsing technique We will discuss this process in detail in Section 2.3.3 We have implemented two components of STAMP: STAMP-Maker and STAMP-Navigator STAMP-Maker is an authoring tool, and STAMP-Navigator is a browsing tool Figure 2.9 illustrates both the flow of editing photos with the two STAMP  components and the relations among the three levels of processing data, that is, raw photo, hyperphoto networks, and hyperphoto collage, as visualization of hyperphoto Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 26 Monday, September 26, 2005 7:35 AM 26 GIS-based Studies in the Humanities and Social Sciences FIGURE 2.7 Hyperphoto networks composed of photos and spatial-hyperlinks FIGURE 2.8 Hyperphoto collage automatically made from hyperphoto Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 27 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 27 Raw photos Stamp-Maker (an authoring tool) Hyperphoto networks Stamp-Navigator (a browsing tool) Hyperphoto collage FIGURE 2.9 A flow of editing photos with stamp FIGURE 2.10 Creating a spatial-hyperlink between two photos 2.3.2 STAMP-Maker STAMP-Maker, one of the STAMP components, is a tool for creating pseudo3D spaces from multiple photos The operations to associate photos are very simple A user specifies the same feature or area in two photos by drawing a rectangle or polygon on each of the two photos In the example shown in Figure 2.10, the same building appears in two photos, and the user has drawn a rectangle enclosing the building on each of the photos Specifying two rectangles on two photos attaches a starting anchor and an ending anchor to create a spatial-hyperlink between the two photos A spatial-hyperlink is comprised of two areal anchors in each photo, which are a starting anchor and an ending anchor Unlike ordinary hyperlink, spatial-hyperlink enables us to make a relationship between two partial areas in each photo Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 28 Monday, September 26, 2005 7:35 AM 28 GIS-based Studies in the Humanities and Social Sciences Thus, users can create associations between any photos, even photos taken by different people or on different dates Users can also create additional spatial-hyperlinks on one photo Therefore, users can create many different routes for spatial navigation (Figure 2.11) A spatial-hyperlink is basically used to specify the same feature or area in two photos to create an association regarding the spatial meaning in the real world A more advanced use, however, is to create an association between two photos taken in different places For example, the same kind of dishes may appear in two photos taken in different restaurants, and a user can create a spatial-hyperlink associating the two photos In such cases, the photos are linked based on semantic relationship over the space rather than a spatial relationship Furthermore, STAMP-Maker also offers advance uses of spatial-hyperlinks, which represent temporal order In short, these are three types of using spatial-hyperlinks — spatial relationship, semantic relationship, and temporal relationship Their mechanisms are the same at the system level, FIGURE 2.11 (See color insert following page 176.) Creating various routes by using spatial-hyperlinks Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 29 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 29 and the differences between them exist only at the application level We will introduce those differences through our example in Section 2.4 2.3.3 STAMP-Navigator STAMP-Navigator is a browsing tool that allows people to navigate pseudo-3D spaces through a simple interface In this component, each spatial-hyperlink is associated with a geometric transformation combining translation, rotation, and distortion The transformation is defined so that a rectangle specified in the photos at one end of a spatial-hyperlink is transformed into a corresponding rectangle specified in photos at the other end of the spatial-hyperlink This navigation system displays one focused photo and several linked photos The focused photo is placed at the center of the display, and linked photos are placed at transformed positions Projection transformation is used for this process The focused photo is opaque, while linked photos are translucent and overlaid on the focused photo When a user selects a photo from the linked photos by directly clicking on it, the scene changes The navigation system displays a short animation that represents a smooth transition (Figure 2.12) In the transition to the next scene, the currently focused photo fades out and the next focused photo moves to the center Simultaneously, the transparency values of all photos are changed This animation gives the user a sense of motion toward the viewpoint of the next focused photo A photo may be linked to more than one photo, which allows scene transitions to fork into multiple paths Thus, users can freely step forward, step backward, step sideward, rotate, translate, and zoom in/out in the pseudo-3D spaces by selecting from among candidate photos This mechanism makes great use of a human’s spatial cognition of perspective scenes STAMP-Navigator thus allows people to experience a sensation similar to walking through the real world A focused photo A focused photo A linked photo A linked photo Animation FIGURE 2.12 Smooth transition from one scene to the next Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 30 Monday, September 26, 2005 7:35 AM 30 GIS-based Studies in the Humanities and Social Sciences Hyperphoto networks Photograph Spatial-hyperlink POI (Point of Interest) Real world FIGURE 2.13 Conversion from the real world to hyperphoto networks 2.3.4 Visualization Options STAMP differs from video in its ability to represent multiple paths in scene transitions, including branches, loops, reverse motions, and terminals In this sense, STAMP is closer to nonlinear video, but it can be created through a process much easier than nonlinear video editing An advantage of STAMP is the simplicity it inherits from photography Also, the structures of hyperphoto networks are very flexible The structure of hyperphoto networks is, in one sense, indirectly structures of a place in the real world In other words, hyperphoto networks are the result of an abstraction of the real world (Figure 2.13) STAMP offers several visualization options of the whole structure of hyperphoto networks Figure 2.14 shows a temporal map and a spatial map of a hyperphoto network Each point means hyperphoto, and each line means spatial-hyperlink By using such maps, we can analyze places and uncover characteristic structures 2.4 An Application We have used the STAMP to create several examples of hyperphoto collage One of us (Konagaya) is a researcher engaging in ethnographic studies and Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 31 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 31 y Time x FIGURE 2.14 Temporal map and spatial map of hyperphoto networks has taken a large number of photos in Mongolia We tried to associate these multiple photos with respect to the meaning of their spatial, semantic, and temporal relationships In ethnographic studies, a research project often consists of three phases — fieldwork, a home stay, and postanalysis Fieldwork is carried out for the purpose of surveying and understanding the spatial and geographic structure of the target area After the fieldwork, researchers decide on the area where they will stay A home stay is the next phase, during which researchers live with native people for about one year After that, they leave the target area, analyze the data collected, and compile a summary of their studies Through this process of postanalysis, the researcher can gain a new understanding regarding the target area We think these three phases correspond fairly closely to spatial observation, temporal observation, and semantic observation, respectively Based on these viewpoints, we organized the many photos taken in the Mongolia studies by using spatial-hyperlinks We can create spatial relationships for photos taken during the fieldwork, temporal relationships for photos taken during the home stay, and semantic relationships for postanalysis Figure 2.15 shows an example of the hyperphoto collage we were able to create We could thus explore the pseudo-3D space and relive the researchers’ experiences In this work, we used semantic relationships to associate photos that contained similar plants and woods By tracing the three uses of spatialhyperlinks, we could easily pass through the various points of interest in the target area Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 32 Monday, September 26, 2005 7:35 AM 32 GIS-based Studies in the Humanities and Social Sciences FIGURE 2.15 (See color insert following page 176.) Hyperphoto network created from photos taken in Mongolia — spatial relationship (left), temporal relationship (center), and semantic relationship (right) 2.5 Conclusion STAMP provides a versatile tool for researchers in the field of ethnographic studies and in other fields, in that it enables them to make greater use of photos in new ways Although STAMP is not intended as a mathematical or physical analysis tool, it should be useful for presenting information regarding daily activities and experiences STAMP is a new type of GIS in that it is flexible and enables us to create storytelling/narrative data structures Copyright © 2006 Taylor & Francis Group, LLC 2713_C002.fm Page 33 Monday, September 26, 2005 7:35 AM A Tool for Creating Pseudo-3D Spaces with Hyperphoto 33 Our STAMP software is available for public use The public-use version of STAMP is named “PhotoWalker.” Users can freely download our software from the Web site www.photowalker.net This Web site also includes additional examples of hyperphoto collages We believe that STAMP will become widely used in the field of ethnography for various applications and open the new way to the future GIS References Hockney, D and Alain, S., David Hockney Photographs, Rizzoli, 1985 Peter, J.N., What Is Hypertext? International Conference ACM, Hypertext and hypermedia, keynote speech, 2003 Konagaya, Y., Innovation of Writing on the Mongolian Area Studies, Geography of the Third World ó from local to global, Kumagai, K eds., 1999 Konagaya, Y., The Mongolian Nomadic World (CD-ROM), TEXNAI, Tokyo, 1998 Konagaya, Y., The Spring of Inner Mongolia, Kawadesyoboshinsha, Tokyo, 1991 Tanaka, H., Arikawa, M., and Shibasaki, R., A 3-D Photo Collage System for Spatial Navigations, Digital Cities II, Computational and Sociological Approaches, lecture notes in computer science 2362, Springer, 2002a, pp 305–316 Tanaka, H., Arikawa, M., and Shibasaki, R., Pseudo-3D Photo-Collage, International Conference ACM, Web graphics category, Siggraph, conference abstracts, 2002b, p 317 Tanaka, H., Arikawa, M., and Shibasaki, R., World-Wide Gallery for Pseudo-3D PhotoCollage, International Conference ACM, Siggraph, Web expo category, www.siggraph.org/s2003/conference/web/expo.html, 2003 Copyright © 2006 Taylor & Francis Group, LLC ... Group, LLC 27 13_C0 02. fm Page 22 Monday, September 26 , 20 05 7:35 AM 22 GIS- based Studies in the Humanities and Social Sciences FIGURE 2. 3 An example of 2D photo collage 2. 2 2. 2.1 Related Concepts Photo.. .27 13_C0 02. fm Page 20 Monday, September 26 , 20 05 7:35 AM 20 GIS- based Studies in the Humanities and Social Sciences a record of places, people, life, and atmosphere in a target area because they... 27 13_C0 02. fm Page 26 Monday, September 26 , 20 05 7:35 AM 26 GIS- based Studies in the Humanities and Social Sciences FIGURE 2. 7 Hyperphoto networks composed of photos and spatial-hyperlinks FIGURE 2. 8 Hyperphoto