Think twice before working at a startup

I’m a machine learning professional who has worked for a startup in the past. I’m writing from an EU and UK perspective, but generally speaking, tech startups are very American in the way they operate. This can result in some nasty shocks. This post will give you an honest take on the upsides of working at a startup—and the downsides.

Myth vs reality

Years of media coverage, often written by journalists who have never worked in a startup, have painted a very rosy picture, which I fell for in the early years of my career. The image of the startup is that of a small, Agile company that isn’t afraid to innovate, unlike big bureaucratic corporations. The startup is typically located in a sexy area (like a garage in Silicon Valley); likewise, the work they do is sexy. Your colleagues will become your friends. You can play videogames or ping pong, and drink beers. The startup is diverse, with people of many different nationalities and backgrounds.

And all of this is true, but this is a classic example of lying by omission. Here are the realities:

1. Your colleagues may be from diverse nationalities and/or ethnic origin, but they will almost all be men between the age of 25–40. They are also less likely to be parents. If you’re young, you’re considered too inexperienced to work there, and if you’re old, forget about it.
2. You may drink beers and play video games, but on a Friday afternoon after 5pm, when you should be enjoying your weekend. The other 4 days of the week will be spent frantically working, often with longer hours than well-established players. Early stage startups, and senior employees, might spend time working on weekends and holidays.
3. Sexy areas = high cost of living, and startups are often resistant to remote work policies (despite being “innovative”).

It doesn’t get better, I’m afraid.

Startups and the Agile methodology

Startups like to bill themselves as Agile, which means (roughly translated for non-software folks out there) as doing things interatively and not getting trapped in analysis paralysis. What actually happens is that startups write a lot of untested code which constantly breaks. The requirements change all the time, and not always because of good reasons (like the client wanting something else) but because management is fickle. There is often not much time for a formal design process, which results in poor abstractions; and nobody is responsible for writing documentation or doing QA. Likewise, there was no model review at the startup I worked for.

If you want to learn industry best practices, you are usually better off elsewhere. I would be especially careful about any startup that operates in healthcare, defence or finance. “Move fast and break things” might work for a social media site, but it will be a disaster in these fields.

Are startups innovative?

Some startups genuinely do innovative work that pushes the envelope of software engineering and/or machine learning. But this is the exception rather than the rule. Plenty of startups are building just another travel app, ecommerce site or payments platform – and it’s often a product that nobody wants (which is the reason so many startups fail). In machine learning, another extreme is the startup that tries pie-in-the-sky ideas which are academic projects, not serious commercial ideas.

And there is plenty of anodyne work in startups. You’ll end up doing the same thing as in a big company, but with worse management, benefits, and (especially) job security.

Furthermore, tech startups can often have messianic leaders. I personally find this very annoying. Startups are rarely solving world hunger or climate change, and no, your smartphone app doesn’t count. This particular criticism can be levelled at tech workers in general, not just startups, but I find many techies boring. They talk about tech during dinner parties or drinks, and rarely know much about current affairs or art. At best, I can talk about my gym routine with them.

Management can suck

Of course, management can suck in any company. But startups are often run by people who have no management experience. Furthermore, startups have a tendency to be extremely disorganised.

Job security sucks

Job security sucks for two reasons. One is that startups can go bankrupt quite quickly – they have a high burn rate, and are very dependent on investor capital, often with few clients and not enough revenue to be profitable. The second reason is that startups can be very erratic and unpredictable – and they operate with a short-term mindset. They might not need your skillset in six months or even weeks (!) from now. They don’t have the long-term mindset of keeping employees around so that they can train them.

This mindset results in a litany of problems:

1. Lots of technical debt, with complex systems in production and not many people that know how they work.
2. High turnover is demoralising, and a lot of people decide to leave fearing they will be next.
3. An inability to gain a competitive advantage by building up in-house know-how.

(If you are wondering, yes, I have seen a startup fire an entire department weeks after hiring them.)

From the perspective of the employee, it will be stressful, especially if you have children depending on you to put food on the table.

Is there any reason you should work for a startup?

There are some. Let’s get the bad reason out of the way: while you may get rich, it’s unlikely. What you do get out of working in a startup is lots of experience with a wide variety of technologies, which you won’t get if you’re pigeonholed somewhere in a big company. You will learn faster. And you will often get a stronger grounding in business problems (if you’re a technical worker) or technology (if you’re from a business background). Finally, there will be less red-tape at a startups; you will have greater freedom to try different things.

The compensation varies but I have not found startup salaries to be markedly different from the industry norm. You might want to work for a startup if they give you the best compensation (but keep in mind the downsides).

Startups from an EU perspective

Pundits admonish the EU for having way fewer tech startups than the US. Frankly, I’m not sure that’s such a bad thing. Startups:

1. Can burn millions, yet never become profitable even years after being founded. The startup I worked for wasn’t profitable in 3 years after being founded, and didn’t plan to be profitable until 5 years in the future.
2. Can cost the clients money when their product fails unexpectedly.
3. Most never develop a useful product.

I am leaning more towards the Nordic/German/Benelux model of SMEs. It’s better to start small, serving a niche, and then expand, than to attempt something grandiose.

Would the real data scientists please stand up?

There are a number of myths surrounding the field of data science, the jobs available, and the pay. When I was in my salad years and green with judgement (read: before I started my master’s) I was lead to believe that data scientists are brilliant individuals who go on to earn $10,000 a month right out of school—provided that they can pass the rigorous demands of their educational programme.

The reality is much more nuanced. It is true that there are a small number of graduates who earn that much straight out of school, along with a somewhat larger number of experienced individuals (e.g. Head of Data, Senior Machine Learning Engineer, Data Science Lead). However, there are many things they don’t tell you:

1. Those kinds of salaries are rarely found outside of the US, and since most data scientists have at least a master’s degree, that often means debt—as much as $100,000, for which the Federal interest rate is a criminal 6%;
2. They typically go only to graduates of the top schools;
3. They are typically only for on-site roles in some very expensive cities, e.g. San Francisco, New York and maybe London, where rents are sky-high;
4. They are mostly offered by brand-name tech companies, finance firms, and a few startups; the recruitment process tends to be arduous. It often involves a battery of tests in coding, maths, IQ, and personality tests, in addition to multiple interviews. And by “interview”, think live-coding.
5. By “right out of school”, think PhD. A master’s is just a pre-requisite in many firms.

So clearly, data scientists being paid $100,000+ per year are the minority. If you look at job sites like Glassdoor, you’d think that this is the median salary, but these sites suffer from sampling bias. The actual median salary in the UK is little over 50K GBP. Comparing salaries across countries is difficult because:

1. Exchange rates fluctuate, and purchasing power parity needs to be taken into account;
2. In the US, employers pay 7.65% social security taxes; in European countries, it’s usually higher, e.g. 15% in the UK, 25% Belgium.
3. Rents vary.
4. Student loans vary dramatically.

So what are some typical salary bands, responsibilities, and what kind of skills are actually required? If you think that you can get a six figure job just by knowing how to work with sklearn, pandas and matplotlib, you are sadly mistaken. These are simply the basic requirements for an entry-level job. (And not even all the basics, at that.)

In order from highest to lowest:

• $100,000+ per year, or the PPP equivalent in euros and pounds: There are 3 types of people, in my experience.
  • The most “classical” profile is someone with a PhD in machine learning/AI from a top school who has written a few white papers in novel ML algorithms. These people work in really cutting-edge applications.
  • The second kind of person is someone with at least a couple of years of experience who is very proficient at programming, and competent with many technologies: not just Python, but they may also be crack C++, Java or Julia programmers. They know their way around cloud computing, containerisation, distributed computing, and have read a book or two on design patterns. They know SQL and NoSQL. These people productionize ML models. They are basically glorified software engineers.
  • Experienced personnel with deep domain knowledge in medicine, business, finance, or linguistics who also have the technical skills.
• Anywhere from $60K to less than $100K: These are either PhD grads from reputable-but-less-than-top universities (those other than MIT, Oxbridge, Ivy League), or ML engineers with some experience. Fresh PhD grads will start on the low end of the scale.
• From $40K to $60K or PPP equivalent: They know the basics, I guess. Strong Python/R skills and stats. Decent SQL skills. Can wrangle their data, write OOP code, and knows how to setup a Docker container or write a shell script. Fresh grads start on the low end, but salaries increase quickly with a bit of experience.
• Less than $40K: These are typically called data analysts. They might write a Python script or two.
• Unemployed (or not employed as a data scientist): The legions of wannabes from bootcamps, online certificates, and 1 year degrees who write awful code (usually in Jupyter notebooks), don’t understand statistics, don’t know how to query a database, clean data, or understand how to properly evaluate the output of a model.

Some other things to discuss. I hear a lot of people mention domain knowledge; often, these people have a lot of domain knowledge in their field of expertise and are trying to break into data science. I hate to break it to them, but while domain knowledge is valuable, all the domain knowledge in the world won’t help if you don’t the master the basics of programming and statistical analysis. It’s usually the coding part where they struggle.

Is there a shortage of data scientists in the industry? Ten years ago, definitely. Reputable sources still claim that anywhere from 20% to 50% of companies struggle to fill data science roles, but take these numbers with a pinch of salt.

What I can say is that there’s definitely a shortage of talented, experienced data scientists who are willing to work for less than six figures. (The competition for six figure roles is blisteringly intense, with many talented, qualified individuals applying for each role.) The shortage is particularly acute for “full-stack” data scientists who can productionize models and do data engineering. However, there seems to be a big oversupply of bootcamp/online course graduates, as well as degree holders from orthogonal disciplines. The life and social sciences are the worst offenders. A bit of data analysis in SPSS, or rookie Python ability probability won’t net you a job, or a low-paying one at best.

If you are looking to enter the field like I am, my advice is to start by getting a degree in DS, CS or statistics (but make sure it’s a stats course with strong programming requirements). It’s generally easier to become a data scientist by first starting out as a developer or a data engineer (glorified developer). By the way, a position as a data analyst does not prepare you for the reality of data science or software engineering. The stories you hear about data analysts becoming data scientists was from years ago when management didn’t know the difference – or they have the title data scientist but only do analytics work. A data analyst is usually a glorified Excel monkey who maybe knows some Python and SQL. While they might be able to create a simple ML model, they often don't really understand how to interpret the model or tune it, as their statistics knowledge only covers the basics. The learning curve for software engineering is even steeper and such a person can only do their job if they have an army of data engineers and ML engineers working for them.

Although companies like IBM tell us that data is a fast-growing field, the threat of automation should not be discounted. In 10 years, there might not be a market for data scientists, or at least, it won’t employ large numbers of people. The bar could be set even higher as simple, routine tasks become automated.

I do believe that creative jobs are immune from automation, and certainly, there are no AI models that can write stories, so I still plan on being a writer in the long-term. We’ll have to wait and see whether my creative abilities end up being more lucrative than my technical ones.