😎 Your guide to AI: June 2021

Dear readers,

Welcome to Your guide to AI. Here you’ll find an analytical (and opinionated) narrative covering key developments in AI over June 2021.

Last weekend was a distracting double header of finals for Wimbledon + Euros, so forgive the tardiness of this issue 😉

If you’d like to chat about something you’re working on, share some news before it happens, or have feedback on the issue, just hit reply!

If you enjoyed the read, I’d appreciate you hitting forward to a couple of friends 🙏

In September last year, I wrote an op ed entitled AI has disappointed on Covid. I made the case that despite lots of hype around drug repurposing, literature mining, and various computer vision-based diagnostics, AI methods hadn’t really mobilised to make a dent on the pandemic. Now, a paper has dived into this topic further. By reviewing hundreds of reports on the use of computer vision on lung X-ray data to study and diagnose coronavirus pathology, the authors found that none of them have any clinical use at all. They all have methodological errors, issues with their training data and labels, robustness issues, validation problems and bias. This further highlights the importance of framing a relevant clinical problem and workflow, intimately understanding your data, and generating high-quality labels that are clinically realistic and meaningful (plug: check out the V7 data platform :-). 

In the US, President Biden expanded the Nov 2020 Executive Order (“EO”) with respect to the “threat posed by the military-industrial complex of the People’s Republic of China” by adopting a “sustainable and strengthened framework”. The US can now prohibit US investments in Chinese companies that “undermine the security or democratic values of the US and its allies”. In particular, the EO covers 59 Chinese entities involved in defense and surveillance technology.

The Biden Administration also launched a National AI Research Resource Task Force that will “write the road map for expanding access to critical resources and educational tools to spur AI innovation and economic prosperity nationwide.” We can expect two reports to Congress - one in May 2022 and another in November 2022. A further National AI Advisory Committee is due to be established shortly. 

The US Government Accountability Office surveyed 42 US federal agencies that employ law enforcement officers about their use of facial recognition technology. They found 20 agencies that use facial recognition, either an owned system or another entity’s system. This includes the Federal Bureau of Prisons, the Drug Enforcement Administration, FBI, Customs and Border Protection, amongst others. The fourteen agencies that reported using facial recognition to support criminal investigations confirmed that these systems were built by non-federal entities. 

Graphcore and the Oxford-Man Institute of Quantitative Finance in Oxford demonstrated how next-gen AI hardware can dramatically accelerate the training of price prediction models using limit order book data (bid/ask price levels).

SiFive, the company productising RISC-V CPU (the main shot against Arm) allegedly received an acquisition offer from Intel for more than $2B. What’s interesting here is that Intel’s CEO has recently gone on the offensive with tens of billions of new R&D expenditure focused on its US and (potentially European) foundry projects. Intel trails both Samsung and TSMC on the fab side of the business. Intel will also manufacture a 7nm SiFive-based RISC-V CPU to attempt direct competition with Arm in 2022. It looks like the NVIDIA-Arm alliance - and the resulting encroaching by this group on Intel’s CPU market - has driven Intel closer to the next best future-proof option: SiFive. 

After a slew of SPACs for LiDAR (see: exit section!) and electric vehicle companies, next up are the autonomous vehicle businesses. In June, two major players - Aurora (OEM passenger vehicles, now trucks) and Embark (trucks) - were in process or announced SPAC mergers. Embark agreed on a $5.2B SPAC deal, following TuSimple ($11B valuation, +35% from IPO). Aurora began talks with Reid Hoffman’s SPAC that was confirmed in mid-July, valuing Aurora at $11B with $2.5B cash on the balance sheet.  

In an effort to meet the global “chip crunch”, employees in Taiwan’s semiconductor fabs were found to work in obscene conditions and under constant surveillance.

Large-scale AGI research has for many years been dominated by two poles: DeepMind and OpenAI. The former has ballooned to 1,125 full-time employees since its acquisition in 2014 and OpenAI today counts 195 staff. Together they’ve published thousands of papers, including many covers of Nature and Science. At OpenAI, a company that originally set out to pursue AGI research in an open manner for the benefit of humanity, the focus has increasingly shifted towards creating products. The GPT-3 API and Azure partnership are two key examples. This past month, a core group of OpenAI team members focused on AI safety, large-scale language models and policy parted ways to form a new public benefit corporation called Anthropic. Funded with $124M, Anthropic signals a splintering of the bi-polar power dynamics of DeepMind and OpenAI. It’s likely that we will see a multi-polar AI research world emerge as a result of a) more risk-seeking capital markets, b) power struggles at the helm of organisations with very focused leadership, c) and the diffusion of AI across the economy, giving rise to organisations with “thematic” research focus (e.g. health, energy, safety etc). Indeed, a product-level example of such multi-polarity is the diffusion of large language models from the major Western labs into China. Wu Dao 2.0 from the Beijing Academy of AI is 10x larger than GPT-3 and can perform a variety of tasks including text generation, image recognition and generation.  

Relatedly, the world of AI research frameworks has also undergone remarkable changes. TensorFlow launched in late 2015 and dominated the framework landscape for almost two years, only to relinquish its spot to PyTorch, which came out in late 2016. At Facebook, the birthplace of PyTorch, the entire company’s AI systems are being migrated to PyTorch. This covers more than 1,700 PyTorch-based inference models in full production that serve trillions of inference operations a day. 

Following the Amazon-AWS playbook, ByteDance is cementing its dominance in AI-first consumer products by externalising its suite of AI technologies for third parties in China dubbed Volcano Engine. The suite includes “personalized recommendation, fast A/B testing, beauty filters, machine translation, automatic content generation, video streaming optimizations, data-driven insights, and conventional cloud computing offerings like compute, storage, and virtualizations.” The company has also set up BytePlus, which appears to be the same as Volcano Engine, but focused on customers outside of China. 

Microsoft’s GitHub and OpenAI released a code review and recommendation devtool called Copilot. The product is trained with GitHub repos and code on the internet in JavaScript, Python and TypeScript. It has drawn controversy because it is a commercial product trained on open source code. 

Here’s a selection of impactful work that caught my eye, grouped into categories:

Predicting enzymatic reactions with a molecular transformer, University of Bern and IBM Research. Enzymes are a bedrock of chemical reactions in which useful products are made. Engineering approaches are common to generate increasingly effective enzymes, but prediction which modifications make them better or worse than their parents is hard. This paper uses multi-task learning and transformers on SMILES strings of enzymatic reactions to develop a model that predicts the structure and stereochemistry of enzyme-catalysed reaction products. 

A graph placement methodology for fast chip design, Google Research and Stanford. In this Nature paper, the authors of Google’s chip design paper from April 2020 expand on their results and describe the use of their method to design the next generation of Google’s TPU. It’s neat to see rapid research to production use for advanced RL-based technologies like this one. 

HuBERT: Self-supervised representation learning for speech recognition, generation, and compression, Facebook. This model learns both acoustic and language models from continuous data inputs. It’s trained by consuming masked continuous speech features to predict predetermined cluster assignments. The predictive loss is applied over only the masked regions, forcing the model to learn good high-level representations of unmasked inputs in order to infer the targets of masked ones correctly.

CoAtNet: Marrying convolution and attention for all data sizes, Google Research. This paper addresses the generalisation issues that self-attention based computer vision models have versus convolution-based models due to weaker inductive bias. It does so by creating a family of models that hybridize convolutions with self-attention. They note that “convolutional layers tend to have better generalization with faster converging speed thanks to their strong prior of inductive bias, while attention layers have higher model capacity that can benefit from larger datasets.”

ICLR 2021 Keynote: “Geometric deep learning: The Erlangen Programme of ML”, Michael Bronstein, Imperial College London. A tour-de-force 38 minute lecture on geometric deep learning theory and use cases. 

NWT: Towards natural audio-to-video generation with representation learning, Cash App Labs/Square/Dessa. This paper presents amazing results on speech-to-video: check it out.

The affective growth of computer vision, Indiana University Bloomington. This paper studies the impact that process of doing computer vision research has on researchers themselves. Of note, their “analysis of over 50 responses found tremendous affective (emotional) strain in the computer vision community. While many describe excitement and success, we found strikingly frequent feelings of isolation, cynicism, apathy, and exasperation over the state of the field.” Indeed, I hear more and more that deep learning research today is demoralising and not interesting because it often boils down to a) throwing more compute at a problem and b) knob tuning. 

Collusion rings threaten the integrity of computer science research, Brown University. The author alerts the community of growing problems lurking in the peer-reviewed conference publication process. Here’s a summary of the process, and a reminder to question what you read…

Waymo, the self-driving company, raised a further $2.5B round from a range of growth and crossover investors including a16z, Alphabet, CPPIB, Fidelity and others. Another potential SPAC/IPO candidate in the making. 

Celonis, the category-leading process mining company based in Munich and NYC, announced a $1B Series D led by Durable Capital Partners and T. Rowe Price Associates. The 10-year old business is now valued at $11B and counts thousands of customers. Celonis is, amongst other things, a gateway drug to robotic process automation (e.g. UiPath) and is now clearly an IPO candidate.

Anduril, the American defense technology company, raised a $450M Series D led by Elad Gil. This values the business at $4.6B post-money. Elad shared interviews with high-level individuals involved in US defense and policy. 

Tractable, the AI-first insurance company, raised a $60M round led by Insight Partners at a $1B valuation, making the company the first computer vision unicorn in fintech. 

CMR Surgical, a surgical robotics company, raised a $600M Series D led by SoftBank Vision Fund 2. 

Gong, the AI for sales team analytics and coaching, raised a $250M Series E led by Franklin Templeton. This brings the company’s valuation to $7.25B. 

Insilico Medicine, an AI-first drug discovery company, raised a $225M Series C led by Warburg Pincus. Their first clinical indication is idiopathic pulmonary fibrosis.

Alation, the data search, discovery, and governance company, raised a $110M Series D led by Riverwood Capital at a $1.2B valuation. It’s revenue stands above $50M. 

Cognigy, a low-code conversational AI company, raised a $44M Series B led by Insight Partners. 

Gideon Brothers, the Croatian warehouse robotics startup that commercialises an autonomous pallet transportation robot, raised a $31M Series A led by Koch Disruptive Technologies. 

Waabi, a new autonomous vehicle company founded by Raquel Urtasun who led Uber’s ATG in Toronto, raised a maiden $83.5M round led by Khosla Ventures. 

Dusty Robotics, a construction robotics company, raised a $16.5M Series A led by Canaan Partners. Their first autonomous product prints out plans on the floor of construction sites. 

Scythe, a landscaping robotics company, raised a $18.6M Series A round led by Inspired Capital. Their first product is an autonomous lawn mower. 

Akur8, an insurance automation company, raised $30M Series B.

MachineMetrics, an industrial data company, raised a $20M Series B led by Teradyne. 

Allcyte, the AI-first functional precision oncology company, was acquired by the leading AI-first drug design company, Exscientia for €50M. Allcyte was the largest investment for Air Street Capital - I shared my journey with the company and why the acquisition is a big deal for end-to-end AI-first drug discovery. 

Babylon Health, the British telehealth provider, announced its intentions to go public via a SPAC deal that values the business at $4.2B. Babylon says its services cover 24M patients, generating revenues of $79M in 2020. Their investor deck is available here. The deal brings $575M in gross proceeds, including a $230M PIPE that includes Palantir (which itself has been a very active SPAC investor, in exchange for long-term paid software deals with their targets). 

Quanergy, a LiDAR and 3D perception software company, entered into an agreement with a SPAC to list for $1.4B. 

DeepMap, a startup that built high-definition maps for AVs in the US, entered into an acquisition agreement with NVIDIA. The business had raised $90M, employed 160 people, and was worth $450M on paper - the acquisition price was not disclosed. The strategy here from NVIDIA is to integrate DeepMap’s capabilities into its NVIDIA DRIVE suite, which includes training and validation systems as well as cloud and in-car compute. 

CryptoNumerics, a data privacy startup, was acquired by Snowflake for $7.1M. 

—

Signing off, 

Nathan Benaich, 18 July 2021

Air Street Capital | Twitter | LinkedIn | State of AI Report | RAAIS | London.AI