Tesla's Computer Vision Master Plan

  • If you haven't taken delivery yet or plan on ordering you can still get the 6 months of FREE Supercharging only until December 17th all Model 3s now qualify! Call or email your Tesla delivery advisor and give them our code

John

Tech Founder
M3OC Supporting Member
Joined
Apr 15, 2016
Messages
1,420
Location
California
Tesla Owner
Model 3
Country
Country
#2
My latest work: an in-depth outline of Tesla's plan to give cars the ability to perceive the world and therefore drive autonomously throughout it.

Thoroughly enjoyed this piece, Trent.

I'm especially curious as to how Tesla partitions the self-driving problem between one or more NNs and traditional logic. For instance, does a NN just do lane interpretation and sign/signal recognition, and then the rest is traditional logic? Or is a NN learning how to drive more holistically? How do they chunk out the problem in their control diagram, and are there any "critic" NNs observing and correcting in parallel? One day I'd like to hear Andrej discuss this.
 

KarenRei

Top-Contributor
Joined
Jul 27, 2017
Messages
1,615
Location
Reykjavík
Country
Country
#3
Good article.

One thing you left out, and which I wrote about recently, concerns examining the road surface. This was one of several reasons for me previously having pessimism about full self driving. We constantly examine the road ahead, looking for things that look wet, icy, potholed, gravel, debris, etc. And when there's water on the road, we try to assess its depth (despite not being able to see the road beneath) using context to see if it's safe to drive through, or just a puddle. These things involve very complex reasoning involving a lot of back knowledge, and are not readily offloaded to a neural net for simple visual pattern matching. AI-Hard, so to speak.

However, I think there's two *excellent* solutions for them. Probably without any changes to the current hardware suite:

https://model3ownersclub.com/threads/autopilot-seeing-the-road-in-ways-that-we-cant.6403/#post-86747

I still think there's a massive body of knowledge that AP needs to incorporate before the general public can feel safe that it'll handle most all situations as good or better than they would. If you live in the countryside it might be "recognize that this is a ewe, that's a lamb, and the lamb will likely run to the ewe as you get close". In a suburban area it might be "recognize that this is a child in that yard and determine from his behavior and the presence or lack of a parent how likely he is to run out in front of you". In a bad area it might be "recognize whether these people you see ahead of you are harmless or whether they're acting suspiciously and might be planning a carjacking". In a rugged area it might be "recognize how weak the shoulder looks to be ahead and evaluate how severe the consequences would be if you drive off". Etc, etc, etc.** These things can all be incorporated eventually, but there's a massive corpus of knowledge that we use when driving.

But as for evaluating the road... radar returns and fleet-sourced high-res topography data should let a vehicle evaluate the upcoming road surface and assess water depth, respectively, better than humans can.


** These challenges can even be combined. For example, the worst part of the route to my house has a short but relatively steep slope on a gravel road, and you can't see over the top. The shoulders are weak, so you prefer to drive toward the centre, and there's a ravine on the south side (but not the north). If the gravel is wet or the road is very icy, you want to have momentum going into the slope, as you certainly don't want to have to stop on the slope when climbing (or brake on the downslope), but you have to balance that off against not knowing if someone is coming who might also be driving in the middle of the road to avoid the shoulder on the ravine side.

These decisions are not simple. An AI simply will not make them as good as a human when presented with equivalent information. The AI needs much better information to compensate for its reduced reasoning power. But I'm increasingly convinced that it can get said information.
 
Last edited:

John

Tech Founder
M3OC Supporting Member
Joined
Apr 15, 2016
Messages
1,420
Location
California
Tesla Owner
Model 3
Country
Country
#4
My latest work: an in-depth outline of Tesla's plan to give cars the ability to perceive the world and therefore drive autonomously throughout it.

Trent you should post your latest article on the Cambridge NN stuff. That split video showing semantic interpretation of the roadway and other objects is awesome.
 
Joined
Sep 3, 2017
Messages
13
Location
Baccalieu
Country
Country
#5
Trent you should post your latest article on the Cambridge NN stuff. That split video showing semantic interpretation of the roadway and other objects is awesome.
I assume you mean this? The same split screen video from the Cambridge/Oxford researchers is already in the Medium article that I posted originally!
 
Joined
Sep 3, 2017
Messages
13
Location
Baccalieu
Country
Country
#6
Thoroughly enjoyed this piece, Trent.

I'm especially curious as to how Tesla partitions the self-driving problem between one or more NNs and traditional logic. For instance, does a NN just do lane interpretation and sign/signal recognition, and then the rest is traditional logic? Or is a NN learning how to drive more holistically? How do they chunk out the problem in their control diagram, and are there any "critic" NNs observing and correcting in parallel? One day I'd like to hear Andrej discuss this.
Thank you, John!

I imagine that Tesla at the very least does not use one big end-to-end neural network for everything, for reasons described in this video. I haven’t really heard any refutation of Shashua’s argument in that video; people generally seem to agree. So at a minimum, I imagine Tesla must split different tasks into different neural networks.

Some people on the Tesla Motors Club forum who have hacked into their cars claim that the perception is done by neural networks, and the actual driver input (braking, steering, acceleration) is traditional logic, but I don’t know if what they’re claiming is true. Hard to verify these claims.

On the topic of critic neural networks, have you seen this paper?
 

John

Tech Founder
M3OC Supporting Member
Joined
Apr 15, 2016
Messages
1,420
Location
California
Tesla Owner
Model 3
Country
Country
#7
Thank you, John!

I imagine that Tesla at the very least does not use one big end-to-end neural network for everything, for reasons described in this video. I haven’t really heard any refutation of Shashua’s argument in that video; people generally seem to agree. So at a minimum, I imagine Tesla must split different tasks into different neural networks.

Some people on the Tesla Motors Club forum who have hacked into their cars claim that the perception is done by neural networks, and the actual driver input (braking, steering, acceleration) is traditional logic, but I don’t know if what they’re claiming is true. Hard to verify these claims.

On the topic of critic neural networks, have you seen this paper?
Oooh. I'll have to give a that a read. Thanks!
 

John

Tech Founder
M3OC Supporting Member
Joined
Apr 15, 2016
Messages
1,420
Location
California
Tesla Owner
Model 3
Country
Country
#8
Thank you, John!

I imagine that Tesla at the very least does not use one big end-to-end neural network for everything, for reasons described in this video. I haven’t really heard any refutation of Shashua’s argument in that video; people generally seem to agree. So at a minimum, I imagine Tesla must split different tasks into different neural networks.
Did you see the post on TMC about the multiple NNs someone dug up?
Check it out here.
 

EVfusion

Active Member
M3OC Supporting Member
Joined
Mar 10, 2017
Messages
159
Location
Canberra, ACT
Country
Country
#9
My latest work: an in-depth outline of Tesla's plan to give cars the ability to perceive the world and therefore drive autonomously throughout it.

Trent, I thoroughly enjoyed your piece. It is both a very informative and a lucid account. I must commend you both for the depth of your underlying research and for the very clear way you have presented it.