Week 4: My Interest in Machine Learning

One of my strongest correlations with feeling down or out of sorts is whether I have been reading a book recently. I will notice that I feel a little out of sorts for a day or two. I will then also recognize that I have not been doing any reading on my kindle of late. Its hard to disentangle whether being down is causing me not to read or whether I am down because I am not reading. It does seem like reading helps me have a balanced head space though. Reading gives me a topic that is not my work which I can reflect on.

Having projects or curiosities I am exploring outside of work is important for me to feel balanced. When I spend too much time thinking about work it becomes hard to settle my brain down or reach a place of calm. Reading is one of the ways I can achieve that balance. Another one is spending time with friends. Generally when I am in a rut it also correlates with me spending less time being social and connecting with others.

It is important for my well being that I am balancing my work life with other pursuits. Reading and being social are two mechanisms that help me live a balanced day-to-day life. It also feels important for me to have some intellectual curiosities that I am building on outside of work. This can manifest as certain topic that I am trying to read a lot about or learn more about. It is rewarding to have one theme that I am making progress on or understanding more about.

My first curiosity that was in this category was Machine Learning. During my junior year of college my friend sent me a Wait But Why series on Artificial Intelligence. Reading those pieces from Tim Urban really struck a cord with me. Urban did a wonderful job of both highlighting the importance of the topic while also outlining the unknowns. During that time I was interviewing at different trading firms to go work in finance and decided to bail on that and see how I could explore ML.

My interest in ML was so strong that to date I have oriented a lot of my career around it. I got a job doing ML adjacent research in college before becoming a data scientist professionally. Even while I was working as a data scientist full time, I still spent time reading ML papers outside of work. Machine Learning fully captured my intellectual curiosity.

But recently, I have spent much less time thinking about Machine Learning. It is less clear that I want to orient my career around it and in my free time I am mostly exploring other topics. I am hopefully that by examining this transition in my curiosity, I can understand more about myself. Some questions I am interested in understanding are:

• What makes a topic something that I am attracted to?
• Why did Machine Learning go from something I was spending a lot of time thinking about to something I am not thinking about anymore?

An important place to start this journey is understanding why I was so attracted to machine learning in the first place.

Machine Learning felt more exciting than anything I was learning in any of my computer science classes or economics classes. In 2016, as a junior in college all my classes felt so divorced from something that would actually impact someone’s life. I was learning the theory of software construction and the impact of the federal interest rate. These topics did not motivate me or engage my curiosity. They felt like solved problems where someone was just transferring their knowledge to me. Machine Learning though felt completely different. It was new territory where we were still figuring out the implications of the technology.

Machine Learning as a field felt like it was speaking to some of the most fundamental questions I could be interested in. How do we build a system that is intelligent and can accomplish non trivial tasks? I was thrilled by the idea of understanding what makes a system intelligent. We still lack such a grasp on where human intelligence comes from and how human intelligence is different from other species. Studying intelligence gets at fundamental questions of our world and has implications for how I live my life.

As I was first learning about Machine Learning it was immediately evident how impactful the technology could be. ML can be used in many different ways to make lives better. It can automate tasks that people do not want to do. It can answer questions that humans cannot on their own. It can make predictions about the world. Machine Learning is able to leverage data in a way that is so different from humans. ML can be so impactful because it is an intelligent system that operates in a different way than humans. This potential impact of ML as a tool, enthralled me. I wanted to work on something that had a huge ability to help people.

In 2016, ML was making huge strides every year. Performance on certain datasets was improving so quickly. New techniques were getting developed that far outstripped the state of the art. Problems that seemed impossible merely years before were being solved. It felt exciting to be working on something that was getting better so quickly. That by doing research and working in Machine Learning there was the opportunity to really push the state of the art forward.

Machine Learning stood out in contrast to everything else I was learning about in my classes. It felt like a topic that was exploring fundamental questions about how we move through the world. It was a technology that had an opportunity to improve how we live our lives and that was making an immense amount of progress quite quickly.

Sitting here in 2022, the field of machine learning is in a very different place than it was in 2016. ML has graduated into its teenage years. It is being used in many different places and there are plenty of people making lots of money using ML. Neural networks have shown the ability to accomplish many different tasks and large language models have amazing text generation capabilities.

ML’s success has in some ways made the research field quite narrowly focused. Most research that is happening is related to deep neural networks and how to make step wise improvements on them. Since we have lots of evidence that NNs are quite successful more research has been focused on going down that same path. It is much less risky to work on something that has a track record of success. If you have to publish research papers every year, then it makes sense to spend your time on something that is already proven.

But my attraction to the field was not about making iterative improvements on NNs. I am attracted to the field because I want to better understand intelligence. Improving NNs does in some ways help us understand intelligence, but that is not the framing motivating the research. I can understand why papers do not have that focus since it is not defined what it means for a paper to help us understand intelligence. When evaluating a paper it is much harder to determine if a paper actually pushes that understanding forward than it is to look at an improvement in a few metrics. When I am in the trenches of day to day work though, it is necessary for me to have a larger motivation I can tie the work to motivate me to push through the less desirable parts of the work. I would be more attracted to ML work if it worked harder to tie the content back to understanding intelligence.

We could compare the success Machine Learning has had with neural networks to the idea of a resource curse for developing countries. The resource curse theory posits that when a country is abundant in a natural resource, in some situations this can actually be bad for growth. The country might become over reliant on that one resource which means it lacks a diversity of industries. We might think that similarly, ML has over indexed on one research paradigm. ML as a field is mostly focused on how can we use neural nets and make them even larger to solve even harder problems. It is possible that creating large NNs will really push the field forward, but it is not the type of research that I am most interested in.

The success of ML means that it is being used every where in industry. Almost every tech company uses ML in different parts of their business. The ecosystem around using ML has become so built out that it can be utilized by any software engineer. In just a few lines of code you can train models on datasets.

From my experience, successfully leveraging ML models is not dependent on a deep knowledge of ML algorithms. Being knowledgeable about algorithms is helpful, but often the key constraint is the quality of the data. If you have data with really high signal, then training the model is the easy part. But with data that is noisy, there is very little improvement you can make by messing around with the algorithm a lot.

While in academic research, the dataset is often fixed, in industry there is the opportunity to shape your dataset. Academic research has defined datasets and then researchers work on getting better scores with that fixed dataset. But in industry the dataset is often quite malleable. Additional data can be collected, the data can be formatted in different ways or the problem you are trying to solve can even be changed. Successfully leveraging ML in industry is often determined by your ability to frame the problem in the correct way and collecting the necessary data rather than a knowledge of algorithms.

As a field, there should be more focus on how to create the right data to solve a problem. Working in industry my success is not dependent on knowing the cutting edge of research and being able to implement it. It is based on making smart decisions about what data to collect and how that data relates to the problem we are trying to solve. This aspect of doing ML work gets talked about a lot less but is quite important.

It feels disheartening that the work that gets valorized and focused on is not the same as the work that makes a day-to-day difference. It can also be hard to be in a field where there is not as much scholarship about the things I need to learn and improve at. While I actually really enjoy thinking about how to collect the necessary data and how to make a system legible in the form of data, that work currently is not considered Machine Learning work.

Writing this piece helped me understand my feelings about ML and why I might be less interested in it. My fundamental attraction to understanding intelligence is still there, but now it feels like Machine Learning is not the field to approach that question. I have been recently reading more about neuroscience and biology, which might approach these questions more directly. While ML is quite impactful and will continue to be, from my experience the most impactful work in industry that is related to ML is not currently considered ML work. I would be excited about research focusing on how do we make systems legible in data and how we can construct some theories about this that are widely applicable. ML is not moving as quickly anymore and that might be why some of my interest has gone into technologies like crypto and quantum computing which feel like they are just at their beginnings.

From these reflections it also has become evident that I have some opinions about what directions ML should move in as a field. The research community should aim to have more diversity and people trying out many different approaches. While NNs have been quite successful it is not clear that they are the only path forward. Also, our understanding of NNs is still so limited. It might be worthwhile to push people towards work understanding NNs rather than improvement. I find a lot of the work from Distill particularly inspiring.

I would love to read other’s opinions about their relationship with the field of ML and where they see the field heading!