Salary no longer determines prosperity

In the past, it used to be that getting paid a high salary ensured an upper-middle class lifestyle. As recently as the 1990s, a salary roughly in the range of 60K pounds, 80K euros or 100K dollars would ensure access to all the trappings of a comfortable lifestyle: a big house, car, restaurants, etc. This is no longer possible. In some cities, like London, New York, San Francisco or even Munich (Germany being traditionally cheap) those numbers are simply the bar for a normal life. The reasons are well-publicised. Some of them are relatively recent, like food inflation – caused by high natural gas/fertiliser prices, plus a contraction in supply from Ukraine – as well as the pandemic, which caused microchips to skyrocket in price, and also made cars more expensive, both new and secondhand. Gas and electric have shot through the roof, although thankfully they are beginning to come down. However, these are short-term shocks, and relatively minor in the grand scheme of things. For example, food prices increased anywhere from 10 to 35%—hardly massive. Whereas housing costs have effectively doubled, tripled or even quadrupled in some areas, compared to the 70s, 80s, and as recently as the 90s. In addition to this, tuition fees and debt has increased substantially. Because house prices and rents have reached such obscene levels, the path to building wealth has diverged markedly from the conventional path: getting a degree, and going to a big city to work, is more likely to make a young professional poor than rich these days. If you want to live like your parents or grandparents did, i.e. house, kids, etc. you need to think outside the box. May I suggest something along the following: 1. Consider a non-conventional degree. Online-only courses and bootcamps can be vastly cheaper. Also consider emigrating to a country where education costs are more reasonable (this is what I did). And don’t assume you need to study for 5 years to master something; a smart and determined individual can learn a trade or profession in 2 years if they put their mind to it. Learn on the employer’s dime! 2. Don’t be afraid to live with your parents into young adulthood to save money. 3. Finally, and most importantly, get a job which can be done remotely, like software engineering, digital marketing and so on. Do not delude yourself into thinking you can buy in London or San Francisco if you *just* try to get that promotion, because it’s not gonna happen. Housing will be your single biggest financial outgoing. If your job can’t be done remotely, look for work in a medium-sized town in the North of England/Scotland, the Midwest of America, etc.

Would the real data scientists please stand up?

There are a number of myths surrounding the field of data science, the jobs available, and the pay. When I was in my salad years and green with judgement (read: before I started my master’s) I was lead to believe that data scientists are brilliant individuals who go on to earn $10,000 a month right out of school—provided that they can pass the rigorous demands of their educational programme.

The reality is much more nuanced. It is true that there are a small number of graduates who earn that much straight out of school, along with a somewhat larger number of experienced individuals (e.g. Head of Data, Senior Machine Learning Engineer, Data Science Lead). However, there are many things they don’t tell you:

1. Those kinds of salaries are rarely found outside of the US, and since most data scientists have at least a master’s degree, that often means debt—as much as $100,000, for which the Federal interest rate is a criminal 6%;
2. They typically go only to graduates of the top schools;
3. They are typically only for on-site roles in some very expensive cities, e.g. San Francisco, New York and maybe London, where rents are sky-high;
4. They are mostly offered by brand-name tech companies, finance firms, and a few startups; the recruitment process tends to be arduous. It often involves a battery of tests in coding, maths, IQ, and personality tests, in addition to multiple interviews. And by “interview”, think live-coding.
5. By “right out of school”, think PhD. A master’s is just a pre-requisite in many firms.

So clearly, data scientists being paid $100,000+ per year are the minority. If you look at job sites like Glassdoor, you’d think that this is the median salary, but these sites suffer from sampling bias. The actual median salary in the UK is little over 50K GBP. Comparing salaries across countries is difficult because:

1. Exchange rates fluctuate, and purchasing power parity needs to be taken into account;
2. In the US, employers pay 7.65% social security taxes; in European countries, it’s usually higher, e.g. 15% in the UK, 25% Belgium.
3. Rents vary.
4. Student loans vary dramatically.

So what are some typical salary bands, responsibilities, and what kind of skills are actually required? If you think that you can get a six figure job just by knowing how to work with sklearn, pandas and matplotlib, you are sadly mistaken. These are simply the basic requirements for an entry-level job. (And not even all the basics, at that.)

In order from highest to lowest:

• $100,000+ per year, or the PPP equivalent in euros and pounds: There are 3 types of people, in my experience.
  • The most “classical” profile is someone with a PhD in machine learning/AI from a top school who has written a few white papers in novel ML algorithms. These people work in really cutting-edge applications.
  • The second kind of person is someone with at least a couple of years of experience who is very proficient at programming, and competent with many technologies: not just Python, but they may also be crack C++, Java or Julia programmers. They know their way around cloud computing, containerisation, distributed computing, and have read a book or two on design patterns. They know SQL and NoSQL. These people productionize ML models. They are basically glorified software engineers.
  • Experienced personnel with deep domain knowledge in medicine, business, finance, or linguistics who also have the technical skills.
• Anywhere from $60K to less than $100K: These are either PhD grads from reputable-but-less-than-top universities (those other than MIT, Oxbridge, Ivy League), or ML engineers with some experience. Fresh PhD grads will start on the low end of the scale.
• From $40K to $60K or PPP equivalent: They know the basics, I guess. Strong Python/R skills and stats. Decent SQL skills. Can wrangle their data, write OOP code, and knows how to setup a Docker container or write a shell script. Fresh grads start on the low end, but salaries increase quickly with a bit of experience.
• Less than $40K: These are typically called data analysts. They might write a Python script or two.
• Unemployed (or not employed as a data scientist): The legions of wannabes from bootcamps, online certificates, and 1 year degrees who write awful code (usually in Jupyter notebooks), don’t understand statistics, don’t know how to query a database, clean data, or understand how to properly evaluate the output of a model.

Some other things to discuss. I hear a lot of people mention domain knowledge; often, these people have a lot of domain knowledge in their field of expertise and are trying to break into data science. I hate to break it to them, but while domain knowledge is valuable, all the domain knowledge in the world won’t help if you don’t the master the basics of programming and statistical analysis. It’s usually the coding part where they struggle.

Is there a shortage of data scientists in the industry? Ten years ago, definitely. Reputable sources still claim that anywhere from 20% to 50% of companies struggle to fill data science roles, but take these numbers with a pinch of salt.

What I can say is that there’s definitely a shortage of talented, experienced data scientists who are willing to work for less than six figures. (The competition for six figure roles is blisteringly intense, with many talented, qualified individuals applying for each role.) The shortage is particularly acute for “full-stack” data scientists who can productionize models and do data engineering. However, there seems to be a big oversupply of bootcamp/online course graduates, as well as degree holders from orthogonal disciplines. The life and social sciences are the worst offenders. A bit of data analysis in SPSS, or rookie Python ability probability won’t net you a job, or a low-paying one at best.

If you are looking to enter the field like I am, my advice is to start by getting a degree in DS, CS or statistics (but make sure it’s a stats course with strong programming requirements). It’s generally easier to become a data scientist by first starting out as a developer or a data engineer (glorified developer). By the way, a position as a data analyst does not prepare you for the reality of data science or software engineering. The stories you hear about data analysts becoming data scientists was from years ago when management didn’t know the difference – or they have the title data scientist but only do analytics work. A data analyst is usually a glorified Excel monkey who maybe knows some Python and SQL. While they might be able to create a simple ML model, they often don't really understand how to interpret the model or tune it, as their statistics knowledge only covers the basics. The learning curve for software engineering is even steeper and such a person can only do their job if they have an army of data engineers and ML engineers working for them.

Although companies like IBM tell us that data is a fast-growing field, the threat of automation should not be discounted. In 10 years, there might not be a market for data scientists, or at least, it won’t employ large numbers of people. The bar could be set even higher as simple, routine tasks become automated.

I do believe that creative jobs are immune from automation, and certainly, there are no AI models that can write stories, so I still plan on being a writer in the long-term. We’ll have to wait and see whether my creative abilities end up being more lucrative than my technical ones.