Visually Mapping The World Can Improve Our Daily Activities

This panel discussion is from the LDV Vision Summit May 2015. This panel discussion is an excerpt from our LDV Vision Book 2015.

Moderator: 
Erick Schonfeld, Co-Founder, TouchCast

Panelists:
Jan Erik Solem, CEO & Founder, Mapillary
Luc Vincent, Director, Engineering, Google
Alyssa Wright, VP Partnerships & Business Development, Mapzen

Evan Nisselson: I’m excited to bring up again Erick Schonfeld, co-founder of TouchCast, who’s going to be our moderator. And he’s going to talk about how visually mapping the world can improve our daily activities and probably many other subjects that they’re going to talk about. Thank you, Erick.

Moderator: Erick Schonfeld, Co-Founder, TouchCast, Demo, executive producer. Panelists: Luc Vincent, director, engineering, (Lead Geo Imagery Street View, Areal, Satellite Efforts, Street View, Aerial, Satellite), Google; Alyssa Wright, VP partnerships & business development, Mapzen; Jan Erik Solem, CEO & founder, Mapillary [L-R]  © Robert Wright/LDV Vision Summit

Erick Schonfeld: Thank you, Evan. Can we give a big round of applause to Evan for putting this together? Really great sessions. I’m learning a lot. So we have a really great panel here. We’ve got Luc Vincent from Google. He runs many of the geo imaging parts of Google maps and other things he’s going to tell us about. We have Alyssa Wright from Mapzen and Jan Erik Solem from Mapillary.  It’s the end of the day and I really want to give people something to talk about during drinks afterward. We’re at this point where we all use maps every day. I know I do. It’s probably after email and maybe after Twitter, it’s my third most used app. I’m opening it up at least three or four times a day. And we all sort of think, Okay, well, this is pretty good. It’s pretty accurate. I don’t need a GPS in my car anymore. I’ve got my iPhone or my iPad and it gets me there even better than when I have a GPS in my car. So what else is there left to do? Aren’t we done? Yes? We’re done? We’re not done?

Jan Erik Solem: We’re nowhere near.

Erick Schonfeld: No?

Alyssa Wright: Yeah, hi. I’ll jump in.

Jan Erik Solem: Go ahead, jump in.

Alyssa Wright: First of all, there will always be blank spots on the map. Whether that means that things are changing and so we are mapping movement or more detail or a building goes up... But I think also another question is even though it’s mapped, I think it’s also important to look at who owns that data, and who has access to it. It’s not just like the data exists, but what’s the community around it?

Erick Schonfeld: So why don’t you tell us a little bit about where you feel we are with the mapping? Where are we in the evolution of digital mapping and where we are in this?

Luc Vincent: I think we’ve come a super long way. I think if you look back at Google Maps, for example, 10 years ago, we had the US only. When we launched Street View in 2007, we had like five cities. I think we have, at least at Google and elsewhere obviously, we have come a long way. We have the whole world. We have a lot of detail. But I think two things are key here... That first of all, user expectation keeps getting higher and higher. I think people expect now everything to work all the time, to be super accurate, for a system to know where you are and to give you advice depending on where you are. A few years ago that was not the case. In addition to Alyssa’s point, I think the world just keeps changing. So whatever we do, we’re just never done, right? So we have to keep up with the world as it evolves and changes and that’s a challenge we’ll never be done with.

Erick Schonfeld: Okay. Let’s talk about the different types of data that we place on maps. We started with lines and a 2D depth representation of the world. Then that became satellite imagery and photographs. Then we started taking photographs. And whether that’s street view or user-generated, uploaded images, that started to add another layer to that. Can you just talk a little bit about: What are the different layers? As you think about making maps, what is that data stack? And from the most obvious to the ones where we’re just starting to place on the map.

Luc Vincent: Yeah, I can jump in on that one. In terms of visually mapping the world, there are several layers there starting from satellite going down. If you’re looking at it from where we are now, the cars have covered the main roads in the most important countries. So we’ve done the easy part and we’re now getting into, okay, how do we get the rest of the world, how do we get the details, how do we get that updated at a rate that people want to see? And so we’re in the early days, I still think that, even after Google has been leading this for a decade, I still think we’re in early days here.

Jan Erik Solem: We used to think of the imagery as being there to enhance the experience—to help you understand a place, to help you get a good sense for your travel destination, to help you every day. It used to be a layer on top of the map in some sense, and now I think it’s maybe changing. I think now more and more we can derive data from the imagery itself and use it to make the map.

Erick Schonfeld: So we’re all talking about data here and how this real-time map is what we’re trying to aim towards, right? Which, to me is really fascinating. Because if we look back at the

history of maps, we had maps that stood the test of time and were the maps people used for hundreds of years, and then decades. And then even when I was going to school, we had maps in our textbooks that were probably created decades or at least years before. And now these maps are just being updated every day, every hour. And we kind of take it for granted. But even that whole real-time aspect of it is something fundamentally different that we just take for granted. We’re updating this virtual representation of the world as close to real time as possible. Why are we doing that?

Alyssa Wright: Can I answer a data question real fast?

Erick Schonfeld: Can you answer the question that I asked?

Alyssa Wright: Yeah. The goal of mapping, at least the... So I’m the kind of person who needs a map just to walk around the block. I grew up with a friend actually who I would consider to be a mapper. She would read an atlas and it actually seemed to have words on it to her and made sense. And then she would get in the car and just drive around. I thought that was a person who liked maps and I’m the kind of person that gets lost walking around the block. But I ended up finding my way to mapping because I realized that it’s a storytelling tool. It’s a way for us to make sense of our world, our experience, our relationships. And it’s an abstract representation. I think it’s important to have many different kinds of stories and have many different authors. And so I think that... Wait, what was your question exactly?

Jan Erik Solem, Co-Founder, CEO, Mapillary ©Robert Wright/LDV Vision Summit

Erick Schonfeld: Well, forget about my last question.

Alyssa Wright: Yeah.

Erick Schonfeld: So are you suggesting every one should have their own map?

Alyssa Wright: I mean...

Erick Schonfeld: Just like everyone should have their own story?

Alyssa Wright: I mean,I do believe that. Everyone should be able to make their own map, whatever that looks like. Whether that’s a napkin that you’re draw- ing to tell somebody how to get to a direction or something that’s like a more...

Erick Schonfeld: Shouldn’t there be some kind of standard?

Alyssa Wright: Sure, and there are standards bodies that work on that.

Erick Schonfeld: Otherwise we’d be bumping into each other.

Alyssa Wright: Say it again?

Erick Schonfeld: But what does that mean? When you say everyone should have their own map. A map is supposed to get me from one place to another, right?

Alyssa Wright: I think that’s one purpose of a map. That’s not necessarily the entire reasoning for maps. I mean, there could be many different purposes for a map. I think that we’re familiar with, especially with Google, getting from point A to point B. We work entirely in the open source, open data field, and I’m on the board of the OpenStreetMap US. So in the OpenStreetMap community, which is kind of the Wikipedia of mapping, they will respond to a humanitarian crisis with Cloud-source response. So that’s not getting from point A to point B. That’s trying to understand what happened after an earthquake or a hurricane and look at “before” pictures and “after” pictures to help with logistics.

Luc Vincent, Google; Alyssa Wright, VP partnerships & business development, Mapzen; Jan Erik Solem, CEO & founder, Mapillary [L-R] ©Robert Wright/LDV Vision Summit

Erick Schonfeld: Right, and this is probably a good time for everyone to just give a little brief synopsis of what you’re doing with Mapillary, with Mapzen, and maybe you can tell us something surprising about what you’re doing with Google.

Luc Vincent: Can I answer your question first? Why are people doing it? Because I don’t think that people realize the maps that we have now are mainly created manually by humans. So, OpenStreetMap is an open and obvious example of that, but all of the major mapping players, they have lots and lots of employees that sit in front of screens and edit maps all day. It’s a manual process. One of the most important pieces of data for that process is the type of imagery that we collect. And so we shouldn’t forget... Why are people doing it? They’re doing it because we need newer maps, we need updated maps in Nepal when something happens. We need better road infrastructure so that people can get in there with their trucks or whatever it is. It’s a manual process fed by imagery and data that we can extract from that imagery. But it’s far from an automated process.

Jan Erik Solem: So it’s manual, but less and less so.

Luc Vincent: Yeah, exactly.

Jan Erik Solem: As the learning gets better and better, I think the manual piece is going to shrink and it’s shrinking right now.

Alyssa Wright: Have you noticed that with your data teams at Google?

Luc Vincent: Absolutely. So for example, we have published some papers... In fact, Andrea, who spoke earlier, was key to some of this early work on this. From Street View imagery and other sources we can extract house numbers.

And that may seem simple from a computer vision perspective, but it’s doing it at scale, at super-high accuracy, that’s really hard. And from these you can then help users find a place. This is like geocoding. You type an address somewhere on a map and you take the user right there where you saw that address. That’s one example. More and more is happening on this space.

Jan Erik Solem: That’s a great example and you shouldn’t forget that what initially fed that work, at least for looking at it from the outside, was again humans. Because you put it into the recapture system and you had humans proving that they’re not bots to train the initial algorithms that now make better navigation for maps.

Erick Schonfeld: If each of you could give us a little data point about how much of the world you’ve mapped, or your respective projects or companies have mapped, and sort of how far we have to go. Just maybe some data points that give some context to what we’re talking about.

Jan Erik Solem: All right. So at Mapillary, we provide a service that lets anyone crowdsource street-level imagery for mapping and for other purposes. And right now we have 16- million-something photos we launched last spring and about 470,000 kilometers of roads mapped.

Erick Schonfeld: And this is a crowdsourced effort?

Jan Erik Solem: Yes. Most of them use our app for iOS or Android and then they drive around, walk around, bike, collect imagery, and then we combine all of that together.

Erick Schonfeld: Okay.

Alyssa Wright: Hi. I’m coming from Mapzen, which is a startup that operates like a research lab. We’re about two years old, or a year and a half old, and everything that we do is based on open source and open data. I’m simultaneously representing OpenStreetMap as well and some of their initiatives, which is, again, the Wikipedia of mapping. There are two million editors registered on OpenStreetMap, which is used for a lot of international aid organizations. It’s mapped a lot of the world, especially a lot of the world that’s in developing countries that may not necessarily be the market for other mapping providers. So we’ve seen a lot of movement in those areas as well.

Luc Vincent: So Google Maps is obviously everywhere, but with varying degrees of quality and detail depending on the country and the area. My area of responsibility is the imagery, which is sort of key to helping make the map. There we think of ourselves as being the crawler and the indexer for the physical world. We sort of gather imagery from a variety of sources, from Street View cars, from users, from crawling the web, from airplanes, and now from satellites. It’s imagery we either acquire or license or gather ourselves from Skybox. We organize all of this imagery, and make it useful for users in the context of a map for their daily activities. It’s also about using the imagery to make sense and create the map. And so in total scale we have... Street View, which is the key to that effort, has now 66 countries on the order of over 7 million miles. But also multiple time layers for these miles so probably more than 20 million miles if you include all this. In terms of high- res imagery of the world, we have probably 90% of the population covered. Now more and more we have also 3D of key metro areas in super high-res.
Alyssa Wright: Yeah.

Erick Schonfeld: What do you define as “high-res”?

Luc Vincent: I think it’s in the order of less than 30 centimeters. 15maybe. I forgot the specs.

Alyssa Wright: When you said “the data that we put on the maps,” I feel like the 3D aspects of where mapping is going is a really exciting piece. You even said “data layers.” I feel like modern mapping has been a two-dimensional concept. You just put layers on top of each other. And what would it be like if we had a much more immersive mapping experience? Something maybe akin to video games or animation or movies? And so I think those kind of influences on our mapping experience is really an exciting place to be in.

Erick Schonfeld: So can we talk about that a little bit? Let’s start with the photos and the street- view-like experience. Just explain to the audience what the value of that imagery stitched together does for making more of an immersive map. Because from my perspective as a user, it’s, Well, this is cool. I can zoom in. It’s sort of immersive. It’s kind of like a panoramic sort of experience. It’s not like VR, but it’s as close as possible. I feel like it’s accurate in an amazing way. But I think of it as, again, this layer that I zoomed in on. I don’t think of it as something that is making the actual contours of the map any better. Am I wrong about that? Are you actually taking the visual information, from photos or from maybe video, and is that giving you a more accurate map?

Luc Vincent: Well, not directly. What can help you from streetview like data is you also have other metadata. For example, the location where the image was taken, which itself tells you where the roads are. You can use various vision techniques to extract 3D and understand how far buildings are. From this and other sources, you can create these building footprints we have on Google Maps. So that really helps you create the more accurate map. It’s not the end of everything, right? I think that what you said is actually quite right. It’s kind of a precursor to... Or poor man’s VR. But it’s going a step in this direction. It’s giving you a virtual experience. You can preview and understand a place, which is often very useful.

Erick Schonfeld: Right. The question I want to ask you is: I understand why Google does Street View. Why do people do this work on behalf of the crowd? What’s in it for them?

Jan Erik Solem: So, what they get is they get a way to capture the imagery themselves. For example, one of the things when I started this was my hometown doesn’t have a street view. I live in Sweden. Well actually, this goes for a lot of places in Europe. Smaller towns. It’s a town of 10,000 people, just outside of the third largest city in Sweden. We don’t have street view and we probably never will have from the bigger providers, because it’s not cost-effective to go in and drive up and down all the residential roads and map it on any frequency at all. But I can do it myself in two weekends with my bike. And so there’s now a tool and a process that lets anyone who has that frustration to go out and get the imagery they want. There’s a lot of people doing it for that reason, and then they get credit as the photographer. But then there’s also a large crowd doing it for open-mapping purposes. So we have strong ties to OpenStreetMap, where a lot of our power users do it because they want to help create a better open map. For example, the Red Cross and a couple of other initiatives are out in developing countries taking imagery for Mapillary. So it goes into the OpenStreetMap editing tools so that people can sit there and fix roads and things like that. There’s anything from the frustrated local that doesn’t have street view to people who are out recreationally mapping on their vacation to people who are doing it for the work of creating better maps.

Alyssa Wright: I think there’s also an urban planning aspect or potential. I was just a guest critic at a university finals. And they had put a kind of street view on the back of a car in order to record potholes. And so this would be something that they had recommended for city vehicles to have so that they could identify if there really was a pothole, what lane it was in, would it be fixed, etc., etc. And so I think potentially street views can help with city government planning.

Erick Schonfeld: What about indoors? I’m the producer of the Demo conference and one of the more interesting companies that has launched in the past few years is a company called Indoor Atlas that basically creates the magnetic footprint of an indoor space. And there are other companies that do other techniques. But to me the inside of buildings is the next level. When we’re talking about immersive experiences, it’s not just the shapes and the outdoor part of it, but it’s also the inside, too. So where are we in mapping the indoors?

Luc Vincent: I’d say we’re less advanced at this point, but I think there’s a whole bunch of activity all around. In terms of what’s happening at Google, we’ve been trying a few things, or ramping up a few programs. One is going into museums, in particular, in large spaces, where we can replicate the Street View car on some sort of push cart. Essentially put the whole set-up on something you can wheel around and take pictures. That gets you a pretty good experience, really a VR for a museum. Now we have, I think, a few hundred museums that are live-in maps as part of the art project. That approach doesn’t really work for small spaces. For a restaurant or even for a mall, it might not work. So really what you can do there is use either an off-the-shelf 360 camera or use an SLR with a fisheye lens. The process where you just go in and take a sequence of photos and create a kind of virtual tour. It’s happening, but at a slower pace.

Erick Schonfeld: So if we shrunk down the 360-degree camera into something that was more like a GoPro device, then all of your users could go into the most interesting buildings and that could be uploaded to the map as well?

Jan Erik Solem: Yeah,absolutely. And there are a lot of those devices in the $500 range coming out on the market now. So I think we will see an explosion in that area. Absolutely.

Erick Schonfeld: So if people have questions, just raise your hand and somebody will come around and give you a microphone. We’ll continue the conversation up here. So I really want to push you guys on this whole idea of sort of indoor mapping, the immersive experience, the 3D experience. So Google has SketchUp, right?

Luc Vincent: Well, we had SketchUp.

Erick Schonfeld: Oh, you had SketchUp?

Luc Vincent: We don’t have SketchUp anymore.

Erick Schonfeld: What happened? Was that closed?

Luc Vincent: We sold it.

Erick Schonfeld: Who bought it?

Luc Vincent: Trimble Navigation bought it. So what was SketchUp? Ithink in 2006 or 2007, I forget exactly when, the notion was that it was going to be a crowdsource of 3D buildings to enhance maps and Earth. But I think we realized after a while that we couldn’t really get the scale we wanted from this approach. Just too hard to do. Perhaps it was just too hard to do. And in the meantime, I think algorithms have gotten better and better and the ability to do this from automated systems has now surpassed what we were able to do with SketchUp.

Erick Schonfeld: I think you mentioned that you’ve got a certain critical mass of cities or parts of cities that are now in 3D and you can see that on Google Maps sometimes, right? That is based on what technology?

Luc Vincent: Based on the imagery captured from planes at a 45-degree angle. Microsoft has something similar, too. I mean essentially, you take this imagery and using stereo-type techniques, multi-view stereo, you can reconstruct a 3D mesh and then you can texture map the mesh. And you’ve got the 3D map.

Erick Schonfeld: How does the advent of drones and the ability to put pretty sophisticated imaging cameras on drones affect this? Do you use drones?

Luc Vincent: We don’t use drones. We have played with drones, but we haven’t used drones for any kind of commercial application at this point. However, there are a bunch of companies doing this—a number of them in Europe. I think it’s great for applications like... I think they have mining applications where they send a drone and the drone can very quickly sort of map the mine and create a 3D rendering and you understand how much is left of the mine. Lots of application around capturing a landmark building, like a castle. It’s hard I think to take this approach and scale that to a whole city. But it might come one day.

Erick Schonfeld: Are any of your projects using drones for capture?

Jan Erik Solem: We have one or two users that played with it. Low-flying drones. It doesn’t scale right now unless there’s some kind of battery develop- ment out there. It doesn’t really scale. Most of them can fly for 15, 20 minutes. You can cover a castle or a city block or something like that, but if you want to do large-scale mapping it doesn’t scale right now.

Alyssa Wright: There are some initiatives within the Open Street Map community to start an equivalent drone map. It’s called “Open Drone Map.” It’s one of the tools to stitch together imagery. And then there’s another initiative that just restarted called the “Open Aerial Map,” which is like a close sister to Open Drone Map. These I know very little about. I don’t know much about drones. But if you’re interested, I can connect you.

Erick Schonfeld: Do we have any questions or suggestions? Feature requests? We’re taking feature requests up here. What do you guys want to see on maps in the next two to three years?

Audience member: I’m thinking a little bit long term. And I’m wondering if what we now think of maps is a kind of temporary stand-in and in X number of years, 10-20, we’ll just have real-time video imagery of every point in the world? And then maps will just be images of some labels? Or do you think maps would be made for another hundred years as a separate type of representation, which has something which video does not have and will never have?

Alyssa Wright: Are you asking will maps go away?

Audience member(cont’d): I mean, maps... My perspective, may be that they haven’t really changed in a hundred years or more. Maybe it’s time for a change. Maybe you don’t need any more better representation because you have a phone with you at all times, which has all the information. Maybe it can give you some other way of showing what you want to know. If you’re trying to go from point A to point B, do we need to show you a map? Maybe not. We need to show you pictures and maybe some arrows and that’s it. I don’t know.

Alyssa Wright: Yeah, I do think it’s a really exciting time for mapping because it’s so much a part of our experience and we do have a map in our pocket all the time. Or multiple maps. So I think it’s changing. In my opinion, I think we will always have something, some sort of map. Some abstract way of understanding the world that’s not necessarily video. Maybe it’s interactive, some simplified way of understanding what’s happening.

Luc Vincent: Yeah,I agree. I think that the flat yellow map that we see today is not the future of the map, but it’s also going to be much more visual than we see now. But probably not real-time video of every location.

Erick Schonfeld: Are we going to continue to be the center of the map?

Alyssa Wright: When I first started to get into mapping, somebody said to me that they didn’t like what was happening in digital mapping, because when you had a paper map you had to locate yourself in a space, as opposed to the way our maps are designed right now. It’s like you’re in the center and it’s almost like an egotistical view of the world. The world kind of revolves around you. This has been a metaphor that has stuck with me. I believe in being authors of maps, but who we are on maps and how we represent our- selves and the people around us will also be a way that we can shift.

Erick Schonfeld: One last question over here.

Audience member: So it will be a political question. Some maps cause borders, or maps identify borders simply for the regular people. And countries have gotten in trouble for using Google Maps for wars. Border wars. Alyssa, you said something about who owns the data on the map. Is there a possibility in the world that just regular people deciding borders... For example, anarchists border mapping where the borders of locations are actually changing based on who controls the map. And if the map is based with the community, then the community decides what the border of the country is. Or the border of that anarchic state.

Luc Vincent: I think that’s what treaties are for, right? Treaties and international law define boundaries and that’s why maps are important.

Jan Erik Solem: True, but I think he’s saying, “Let’s put some limit on what you can do with crowdsourcing.” That’s dangerous to rely entirely on crowd- sourcing to decide on borders.

Erick Schonfeld: Which is back to my original question. There are certain standards—I expect a border on a map to be the real border. Or I expect Times Square to be where Times Square actually is. Which wasn’t, I don’t think, what you were saying about different maps for different stories. Don’t want to put words in your mouth, but maybe a map for me would show different layers, right?

Alyssa Wright: Sure. I mean, I think there are many different sources of authoritative data, even for a border. You could look at a U.N. border as opposed to what individual countries are saying is a border. It’s a really interesting question. Where exactly is a border? Does it follow a geological formation? What happens if that’s moving? So part of my dream—if we’re going to talk about if we will live in a world without maps—is that there isn’t one map, but potentially a series of looking at the world and lots of different maps. So that there doesn’t have to be one border, but potentially like multiple borders.

Luc Vincent: That’s already the case.

Alyssa Wright: Yeah, yeah.

Luc Vincent: Depending on how you look at Google Maps, for a different country, you see a different border.

Alyssa Wright: Right, yeah. True

Luc Vincent: Or disputed areas.

Erick Schonfeld: So we’ll just leave it at “you really can’t trust anything you see.”

Alyssa Wright: Yeah.

Erick Schonfeld: Thank you very much for taking us through this journey through the world of maps. I’m now thinking about it in a very different way. I hope you all are, too. So, please, big round of applause for our panelists and I’ll bring Evan back up to close out the day.