☃️ Your guide to AI: December 2020

Dear readers,

Welcome to 2021 and the first of a dozen issues of my newsletter, Your guide to AI. Here you’ll find an analytical narrative covering key developments in AI tech, geopolitics, health/bio, startups, research, and blogs over the month of December 2020. 

As always, if you’d like to chat about something you’re working on, share some news before it happens, or have feedback on the issue, just hit reply! I’m looking to add 5 seed stage AI-first companies to the Air Street Capital portfolio this year - let’s make a few of those newsletter members 💎

If you enjoyed the read, I’d appreciate you hitting forward to a couple of friends.

AI applied to medical imaging continues to be a hot topic in medicine. Last year saw the first Medicare/Medicaid reimbursement approval for a deep learning-based stroke detection system produced by VizAI. This month we saw Paige receive their CE mark for a breast cancer detection software that flags suspicious areas for further pathology review. There are now close to 60 startups in this field and many more papers submitted to arXiv. But here’s an important detail that you need to look out for. Most papers will compare a new model’s performance against a panel of medical practitioners, sometimes reporting these as pooled averages, which systematically underestimate human performance. A key question to ask when evaluating such studies is whether the task a model learns is one that doctors actually practice in the clinic. For example, what does it mean if a model predicts the confidence score of binary classification or predicts whether a bad event will happen at 48h but not 72h in the future? Luke Oakden-Rayner proposes a solution: for AI-human comparisons, each human reader is its own experiment. 

Over in Russia, President Putin shared several strategy points on the country’s plan for AI. The government is backing the construction of an AI research center in southern Russia (Imereti Valley) that is designated federal territory and will benefit from special legal and regulatory status. He noted that “AI algorithms in Moscow are used in healthcare, education, security, and smart city technology….and very soon the Government will have to adopt a digital transformation strategy…with practical measures to introduce AI algorithms.” Importantly, he points to tax incentives to accelerate investment: “Let me remind you that we have set ourselves the strategic goal to quadruple investment in Russian software solutions in the coming decade…The Government discussed this issue at length and made an absolutely correct decision regarding two matters: insurance premiums went down from 14% to 7.6% and revenue tax was reduced, I think, in a revolutionary way, from 20% to 3%.” Contrast this policy with the US and UK tax policies that will likely only see hikes for businesses and entrepreneurs (take note!). Putin notes that for computer science education, “school pupils are still studying languages and software elements that were used back in the previous century.” Instead, “it is necessary to substantially expand the existing methods of teaching computer science so that schoolchildren learn how to launch their own startups.” Not a bad idea. 

On the topic of defense, the Turkish military received 500 locally-built multi-copter drones capable of flying 90 mph for 30 mins. The drone carries a LiDAR, daylight, and IR camera, and is capable of autonomous flight and swarming while carrying payloads that can take out buildings, bunkers, and armored cars. Relatedly (but without deadly payloads), a California city police department also deploys regular autonomous drone missions to help them respond to crimes. While more cities adopt this technology offered by startups including Shield AI, Skydio, and DJI (which is now also blacklisted by Trump), the ACLU pushes back on the resulting loss of privacy for civilians. 

It’s also clear that ethical and fair use of AI systems really hit the agenda in 2020. The solutions are likely to be a mix of policy and education: For example, the University of Cambridge launched the UK’s first Master’s degree in the responsible use of AI. Meanwhile, the Los Angeles Police Department has banned the use of commercial facial recognition systems (note they previously were customers of Clearview AI) as more wrongful arrests and jail time are dealt with due to bad facial recognition matches. 

Chinese companies and universities were the targets of a long list of sanctions by the Bureau of Industry and Security of the US Dept. of Commerce. Amongst them was SMIC, mainland China’s answer to TSMC, which said that their ban will badly affect the company’s R&D and production capacity as they move to the 10nm node. Recall that SMIC completed a secondary listing on Shanghai’s Star exchange last year, which saw its valuation soar. 

A consortium of European nations plans to spend $145B to boost their domestic semiconductor industry and (presumably) achieve self-reliance. Though no details on implementation are provided. 

The big story from Google was their effort to tighten controls over what their researchers say or don’t say, in particular as it relates to casting its technology in a negative light. A paper submission co-authored by Timnit Gebru on the valid risks of large language models was halted from publication and she was forced out of the company. Her story is described in MIT Tech Review and has “sparked a debate about growing corporate influence over AI, the long-standing lack of diversity in tech, and what it means to do meaningful AI ethics research. As of December 15, over 2,600 Google employees and 4,300 others in academia, industry, and civil society had signed a petition denouncing the dismissal of Gebru, calling it “unprecedented research censorship” and “an act of retaliation.”’ 

The big news this month was Uber selling off its ATG self-driving group to Aurora while investing another $400M into the company and taking a seat on its board. Uber will now own 26% of Aurora, valuing the business at $10B. It was reported that Aurora extended offers to more than 75% of ATG’s employees, but not the 50 folks at ATG R&D in Toronto (led by acclaimed researcher Raquel Urtasun). 

Waymo opened a larger and more complex testing site in Ohio, which also exposes their vehicles to much more inclement weather than sunny California.  

Zoox revealed their ride-hailing robotaxi, which looks cool but doesn’t have a launch date. Meanwhile, Tesla’s Elon Musk announced a new launch date for the full self-driving subscription: Early 2021. 

The “short Intel, long Arm” trade keeps becoming more significant. Now, Microsoft is said to be working with Arm designs to produce its own data center processor instead of buying from Intel. Their official statement says it all: “Because silicon is a foundational building block for technology, we’re continuing to invest in our own capabilities in areas like design, manufacturing, and tools, while also fostering and strengthening partnerships with a wide range of chip providers.” In a market cycle where the speed of iteration, competition, and capital markets run wild, companies opt for vertical integration (not dissimilar to countries opting to shore up key industries). 

Furthermore, reports claimed that Apple pre-ordered the entire 3nm production capacity from TSMC for their M-series and A-series processors. This manufacturing node offers a 15% performance boost and 25% energy saving compared to its most recent 5nm process. 

AWS announced their second in-house chip, Trainium, which is focused on accelerating training workloads. However, no benchmarks were released.

Autonomous navigation of stratospheric balloons using reinforcement learning, Google Brain, and Loon. This paper looks at how to control the physical position of superpressure balloons at high-altitude. Conventional control techniques are problematic because the weather can change quickly in unexpected ways. Instead, the authors use reinforcement learning to learn a control strategy that can handle high-dimensional, heterogeneous inputs, and optimize long-term objectives. They use real-world examples and simulations for training and even test the learned controllers in real Loon balloons flying for three weeks over the Pacific Ocean.

ReBeL: A general game-playing AI bot that excels at poker and more, Facebook AI Research. We know that AI agents can outperform humans in both chess and poker, but the models they use are quite different. This paper combines RL with search (ReBeL) that works not only in perfect-information games like chess but also in imperfect-information games like poker. ReBeL produces gameplay decisions by accounting for the probability distribution of different beliefs that each player might have about the current state of the game. This approximates how human poker players need to infer the probability that their opponents may or may not have a certain hand. In doing so, ReBeL treats imperfect-information games like a perfect-information game by changing the decision of a game state so that it’s not just defined by a sequence of actions but also includes the probability that different sequences of actions have occurred.

Mastering Atari, Go, chess and shogi by planning with a learned model, DeepMind. This paper is the next generation of the AlphaGo lineage, which can now learn to outcompete humans without being given the rules of the game from the outset. Note that AlphaZero used lookahead search and the rules of the game it was given. By contrast, MuZero doesn’t try to model the environment but instead models aspects that are important to the agent’s decision-making process using a neural network: how good is the current position, which action is best to take, and how good was the last action? 

RLOC: Terrain-aware legged locomotion using reinforcement learning and optimal control, Oxford. In this paper, the authors use the ANYmal system to build a model-based and data-driven approach for legged robot planning and control over uneven environments. Of note, the system doesn’t need to be retrained for specific environments and the control policies can transfer from one robot to another one with different mass and form factor. Footstep policy that generates desired feet position. You can find a video demo here. Thanks for sharing your work, Siddhant! 

Imitating interactive intelligence, DeepMind. This paper explores how to build agents that can interact with humans. To do so, they first create a simulated environment, the Playroom, in which two agents are embedded. These agents can interact, move around, manipulate things, and speak to one another. They do so by consuming images and language instructions for actions as input and produce physical and language actions as outputs. Using human demonstrations, the agents learn policies using a combination of supervised learning and inverse RL to infer reward models, and forward RL to optimize policies using that reward function. These agents can be instructed to do things in language, and their behaviors observed in the simulation. 

CPM: A Large-scale Generative Chinese Pre-trained Language Model, Tsinghua University. This paper describes the largest Chinese autoregressive language model with generative pre-training (essentially the Chinese version of GPT) that has 2.6b parameters. I note this paper to demonstrate the pace with which GPT models are re-implemented in different contexts. In this case, the model was trained for 2 weeks using 64 NVIDIA V100s (which is nothing compared to GPT3).

WILDS: A benchmark of in-the-wild distribution shifts, Stanford, Berkeley, Cornell, Caltech. One of the reasons we see so many startups attacking ML monitoring is due to distribution shifts from the data models being trained on the data they consume in the real world. Moreover, the ML community doesn’t have many datasets that are built to specifically solve this issue. To fix this, this paper offers a benchmark of in-the-wild distribution shifts spanning diverse data modalities and applications, from tumor identification to wildlife monitoring to poverty mapping. The datasets “reflect distribution shifts arising from training and testing on different hospitals, cameras, countries, time periods, demographics, molecular scaffolds, etc., all of which cause substantial performance drops in our baseline models.” As some commented on Twitter, this dataset could be the (Super)GLUE benchmark for real-world model evaluation. Speaking of SuperGLUE - it’s now solved, twice.

Point Transformer, Oxford, Chinese University of Hong Kong, Intel Labs. As the Transformer architecture continues to eat through one ML task after another, this paper describes its use for 3D point cloud processing, which is important for autonomous system navigation, positioning, etc. They develop a self-attention layer for 3D point cloud processing and show that the Point Transformer set a new state of the art on large-scale semantic segmentation tasks, shape classification, and object part segmentation. 

Molecular representation learning with language models and domain-relevant auxiliary tasks, BenevolentAI. Continuing with the Transformer theme, this paper uses BERT and SMILES strings to learn a flexible and high-quality molecular representation for drug discovery problems (virtual screening). 

Training data-efficient image transformers & distillation through attention, Facebook AI, and Sorbonne University. While Transformers have been shown to solve image classification tasks, they require pre-training with hundreds of millions of images and expensive computational resources. This paper introduces a competitive convolution-free transformer by training on Imagenet only on a single computer in less than 3 days. 

Unadversarial examples: Designing objects for robust vision, Microsoft Research, and MIT. Adversarial examples jumped onto the scene 2-3 years ago as data samples that were designed to fool neural networks into making incorrect predictions. The field pointed to adversarial examples as security weaknesses that need to be addressed. In this paper, the authors introduce a framework that exploits this sensitivity to instead create robust (“unadversarial”) data samples. In this case, unadversarial examples are used during training to encourage neural networks to correctly classify objects with high confidence. 

Papers with Code 2020: A Year in Review, Facebook/Papers with Code. Check out the top trending papers, libraries, and benchmarks for 2020!

CellBox: Interpretable machine learning for perturbation biology with application to the design of cancer combination therapy, Harvard, Broad, and Dana-Farber Cancer Institute. This paper develops a hybrid approach for predicting dynamic cellular responses to extracellular perturbations (i.e. what happens inside a cell when it is treated with a drug?). Their dynamic systems biology modeling with RNNs makes for highly accurate and interpretable predictions of drug combinations in cancer cell lines.

ML Lake: Building Salesforce’s data platform for machine learning. This product is “a shared service that provides the right data, optimizes the right access patterns, and alleviates the machine learning application developer from having to manage data pipelines, storage, security, and compliance.”

Using JAX to accelerate our research. This post describes the rapid uptake of JAX, a Python library designed for high-performance numerical computing, at DeepMind. Many bid on JAX as one of the next major ML frameworks. Watch this space. 

The Sunday Times ran an interview on Demis Hassabis: “the kid from the comp who founded DeepMind and cracked a mighty riddle of science.”

Using AutoML for time series forecasting. This post from Google describes how AutoML for feature engineering, architecture search, hyperparameter tuning, and model ensemble run over 2 hours on 500 CPUs resulting in a top 2.5% rank in a time series forecasting competition using Walmart product sales. This compares to human-engineered solutions that take months to put together. 

Algorithmia published their “2021 enterprise trends in ML”, which suggests that annual spends are increasing YoY, use cases focus on process automation and customer experience, the gap between the “haves” and the “have-nots” is widening, and that organizational alignment is the biggest gap in achieving ML maturity. 

Coinbase shared how they built in-house tools to do ML on tabular and sequence data, and early use them together. 

LinkedIn describes DataHub, an in-house built data catalog and runs through predecessor systems that motivated its need to exist. DataHub is now a startup called Metaphor Data. 

Completing the ML loop: an ode to the need for treating ML as a never-ending life cycle.

Another area of big tech-born startups is the feature story. This piece describes real-time feature engineering with a feature store.

Two great reads from Chip Huyen: machine learning is going real-time (how it works and why) and the MLOps tooling landscape v2 (terrific resource). 

Geometric ML becomes real in fundamental sciences. 

Maithra Raghu (who presented at RAAIS 2020!) shared her reflections on a (prolific!) 6 year PhD in ML that she just wrapped up. Congrats!

Let’s kick off this hyperactive holiday wrap-up with a cheeky shout-out to two Air Street Capital portfolio companies! AI chipmaker Graphcore raised a $222M Series E led by Ontario Teachers’ Pensions Plan and computer vision data platform V7 Labs raised a $3M Seed to scale its auto-labeling, training, inference, and lifecycle management service. 

Horizon Robotics, a Chinese AI chipmaker, raised $150M of a reported $700M target Series C, led by 5Y Capital, Hillhouse, and Capital Today. This follows their $600M Series B two years ago. 

Scale, the data annotation company for AI, closed a $155M Series D led by Tiger Global. The 4-year old company, which now counts hundreds of customers that annotate tens of millions of data points per week, at $3.5B post-money. Scale also announced its acquisition of Helia, a small computer vision startup founded in 2019 with a focus on video understanding, to accelerate Scale’s Nucleus data lifecycle product. 

Genesis Therapeutics, an AI-first drug discovery company focused on small molecules, raised a $52M Series A led by crossover investor Rock Springs Capital. The company is particularly strong on property prediction and graph neural networks. 

BioAge, a biotech company focused on finding drugs to treat age-related diseases, raised a $90M Series C led by a16z and Elad Gil (who is also an investor in Spring Discovery, another aging focused drug disco startup). The company has human aging cohorts with blood samples collected up to 45 years ago alongside -omics data that is tied to extensive medical follow-up records including detailed future healthspan, lifespan, and disease outcomes.

TuSimple, a Chinese autonomous trucking company, closed a $350M Series E from a range of strategic investors (logistics companies). Earlier in the year, TuSimple was reportedly preparing for a US public listing in 2021. 

WeRide, a Chinese AV startup, raised a $200m strategic round from Chinese bus maker, Yutong, to make autonomous minibusses and city busses. 

Microblink, a 7-year old Croatian computer vision startup automating data extraction, raised a $60M round from Silversmith Capital. 

Anyscale, the makers of the Ray distributed execution framework, raised a $40M Series B led by NEA. Their managed service lets developers build distributed applications from their laptops. 

Ultimate.ai, a virtual customer service agent builder, raised a $20M Series A led by Omers Ventures. 

Tecton, the managed feature store for machine learning, raised a $35M Series B co-led by a16z and Sequoia, just 7 months after announcing their $20M Series A. 

Caja Robotics, a warehouse automation company with robots for lifting and moving items, raised a $12M Series B led by New Era Capital. 

Tonic, makers of a synthetic data service, raised an $8M Series A led by GGV. 

Truera, which builds model explainability software, raised a $12M Series A. 

Arthur.ai, another model explainability startup, raised a $15M Series A led by existing investor Index Ventures.  

Reface, the app that does what it says on the tin raised a $5.5M Seed round led by a16z. 

WaveOne, a video compression/superresolution startup, raised a $6.5M Seed round led by Khosla. In the years since Magic Pony built related technology before joining Twitter, the problem of bandwidth-efficient media communications can only have grown more significant. 

Anomalo, a data quality, and validation startup raised a $6M round led by First Round Capital and Foundation Capital. 

Innoviz, an Israeli LiDAR startup, is the latest of its kind to agree to a SPAC with Collective Growth Corporation. The combined company will be worth $1.4B. 

Ouster, a 5-year old US LiDAR maker that made $19M revenues in 2020, agreed to go public via a SPAC called Colonnade Acquisition Corp. This values Ouster at $1.9B. LiDAR’s are a hot target of SPACs, which to date include Velodyne, Luminar, and Aeva, all reaching combined valuations of $2-4B with very little revenues today (if any). 

Boston Dynamics, the markers of the quadruped Spot Mini, sold an 80% position for $921M to South Korean automotive giant Hyundai. 

In the weeks (or months?) before Roblox finally goes public, the global online platform for game creation and playing acquired Loom.ai, a startup specializing in real-time facial animation technology for 3D avatars.

—

Signing off, 

Nathan Benaich, 17 January 2021

Air Street Capital | Twitter | LinkedIn | State of AI Report | RAAIS | London.AI