Attending a coding bootcamp can be a life-changing, career-launching experience. But not all bootcamps are equally effective. And while many claim near-perfect hiring rates, few are transparent about their outcomes. This puts prospective students in a precarious position: How can you judge the return on investment of a program if you can’t fully understand the results?
At Flatiron School, we’ve been committed to student outcomes from the start, publishing annual independently-verified jobs reports according to strict reporting standards we set with the White House and other leading bootcamps. While some schools have followed our lead and started reporting more data, the industry as a whole has a long way to go. This makes it the students’ responsibility to be skeptical, do their research, and effectively parse coding bootcamp outcomes reports before making this investment in their education.
Unless a bootcamp doesn’t report its outcomes at all – the first big warning sign for prospective students – there are three tiers of outcomes reporting you’ll find, each with different levels of transparency and external scrutiny:
-
Self-reported: Here a school boasts a few bold statistics on their website with no look “under the hood” to show how those numbers were calculated. If you come across self-reported outcomes, it doesn’t necessarily mean they can’t be trusted, but you should be skeptical. It’s been three years since Flatiron released the industry’s first Jobs Report, and some level of reporting is becoming more common across the industry – it’s worth asking why a school isn’t committing to transparency as the rest of the industry moves in that direction.
-
Independent verification: Here a bootcamp has employed a third-party to review the numbers – a first stopgap against data being manipulated. That said, this type of reporting doesn’t always go deep enough and can give a false impression of objectivity to intentionally vague or misleading presentations of data. It’s important to look super closely at how involved the third-party really is and how much they're empowered to qualify the results.
-
Examination by third-party auditors: There are so many potential ways for a bootcamp to slice and dice the numbers to obscure its actual outcomes results – i.e., considering only a select group of students, only a certain location, only a certain time period. To present a truly objective portrait of a bootcamp’s success, outcomes reports must be “examined” by third-party auditors, who are tasked with contacting students directly to verify student salaries, job titles, hire dates, and more. Flatiron’s auditors require a 100% response rate from contacted students and 100% accuracy in the findings. They also hold us to a high standard for transparency and clarity – if we make a claim they believe to be misleading, they won’t verify the report, and we go through revisions with the auditors to create examined reports that they believe to be clear and precise for readers. This isn’t the case for bootcamps who self-report or only have their data “reviewed” by third-parties.
But whether or not a school has their reports independently examined, there are six key data points to look for to interpret a bootcamp’s outcomes. If you’re reading an examined report, it’s easier to find – and trust – this information. If a bootcamp’s data is self-reported or only reviewed by a third party, it’s even more important to ask.
1. Completion rate
Look to make sure a school presents their graduation rate, as well as the unvarnished numbers they use to calculate it – the White House specifies that bootcamps should include the total number of students enrolled and the total number of those who graduate since percentages alone can be misleading and more easily manipulated. (Going beyond statistics applies to each of the following points as well.)
As you explore these numbers, be sure to note a bootcamp’s definition of “graduate,” which could allow them to obscure how many students are stuck in the job-search cycle or have dropped out of the program. Unfortunately, graduating doesn’t mean the same thing across the board for bootcamps, so a school’s true graduation rate can be hard to decipher and compare. For example, a school could say you don’t officially graduate until you get a job (“You don’t graduate until you get a job, therefore 100% of our graduates have jobs!”). This is a marketing tactic to get a perfect employment rate, not an objective way to show a program’s efficacy.
On the other hand, if a reported graduation rate is low, it’s important to understand why. Are students not graduating because they have chosen to quit the program? Have they failed or been forced out?
Key questions to ask a bootcamp:
-
How do you calculate your graduation rate?
-
What are your graduation requirements?
-
What additional factors might cause a student to not graduate or be exempt from your graduation rate calculations?
2. Time to get a job
Schools should present how long it takes their grads to get jobs, and report on the number of students to accept paid work at various intervals after graduating. There are a few different ways we’ve seen this data presented, so look for exactly how a bootcamp measures the time a student takes to get a job. Is it until a student receives their first job offer – regardless of whether he or she accepts it – or is it when he or she actually accepts an offer?
The White House reporting standard we follow goes by the latter, requiring the “total number of students who accept paid work within 30, 60, 90, 120, 120+ days of graduation.” Note: There’s still some ambiguity here. For example, Flatiron counts time to get a job from a “job-seeking start date” rather than “graduation date” since some students take time off between completing the program and beginning their job search. It’s a small percentage of students, and we report this data to our auditors for verification.
Key questions to ask a bootcamp:
-
How do you measure the time it takes your students to get jobs?
-
How do you define a student getting a job in this context?
3. Number of students in “technical roles”
This is distinct from number of students hired. A big discrepancy between number of students to get jobs (any jobs) and number of students in technical roles is a warning sign. Even without a discrepancy, it’s important to look for a school’s definition of “technical role,” which some schools have expanded to include any role in the tech sector. The White House framework, on the other hand, defines “technical roles” as working as part of, or directly with, an engineering team. Consider your goals and be sure to read the fine print. If your goal is to leave your job as a salesperson, you want to be sure that a bootcamp doesn’t consider it a success to help you get a job at a tech company as a salesperson.
Key questions to ask a bootcamp:
-
How do you define a “technical role”?
-
How many of your students end up in technical roles after completing your program?
-
What are the reasons some of your students don’t end up in technical roles? What roles do they take?
4. Average starting salary
Look out for whether a school reports on the average starting salary of accepted offers or of offers received, which can skew their average higher. There are many reasons a student might choose to not accept their highest offer; perhaps there’s a lower offer from their dream company or one that has other perks they’re seeking. It’s also difficult to verify the accuracy of a job offer. As such, the White House requires the “average salary of students, defined as base salary received in first role accepted immediately following graduation.” If a school reports anything but the actual salary received in that initial job, see if you can get that information in order to make a more informed decision.Key questions to ask a bootcamp:
-
How do you calculate your students’ average starting salary?
5. Job-seeking vs. non job-seeking students
A job-seeking student as defined in the White House framework is a student “actively looking for a role in the 90 days following graduation.” It’s important to know how a school defines job-seeking and non job-seeking because, usually, a school’s job placement rate – the stat you’ll find pinned to the top of most bootcamp websites – is calculated off students who are considered actively job-seeking. Understanding why students are reported as non-job seeking helps you be sure a school isn’t over-categorizing people into that category to achieve a better hiring rate. That’s why the White House framework requires transparent reporting of specific post-completion plans for any student reported as non job-seeking – omitting that information can be a red flag.
Digging into why a school might categorize someone as non-job-seeking also gives clarity around your eligibility to be a Job-Seeker there. Do you have to be authorized to work in the US? Are there age or language requirements?
Key questions to ask a bootcamp:
-
How do you define a grad as “Job-seeking”?
-
What are you Career Services eligibility requirements?
-
What are the post-graduation plans for your non job-seeking students?
6. Demographic breakdown
It’s important to remember that salaries can vary significantly across the country. When evaluating average starting salaries across bootcamps, take into consideration where graduates are located and if that may have an impact on the numbers reported. Other demographic statistics can also give you a window into a school’s community and how much they invest in cultivating diversity. Key questions to ask a bootcamp:
-
What’s the geographic breakdown of where your students accept jobs?
-
How would you describe your student community?
A note on job guarantees
The promise of the now-common money-back job guarantee may make some students not scrutinize outcomes as carefully – after all, if you don’t get a job, they’ll give you a refund. But before deciding to join a bootcamp based on their job guarantee, understand exactly how a given school defines a job in their terms and conditions. The White House reporting framework defines a job asa paid full-time role, apprenticeship, or contract role. But not all programs follow that standard. Some bootcamps may consider any job in tech as fulfilling their terms even if you’re not writing code; some even consider their responsibility to you fulfilled by offering freelance work at their own company, and void your refund if you reject the offer. So if your expectation is that a school will help you find a full-time job as a developer, be sure to check if a school has other types of jobs that satisfy their requirements – but might not satisfy yours.
If your expectation is that a school will help you find a full-time job as a developer, be sure to check if a school has other types of jobs that satisfy their requirements – but might not satisfy yours.
Key questions to ask a bootcamp:
-
Exactly what kinds of jobs fulfill your job guarantee terms?
-
What sort of support do students get in their job search?
-
What happens if I don’t like the first job offer I receive?
Now, everything we’ve covered here is actually pretty straightforward and would be simple for bootcamps to report on. These are also all things that many of the leading bootcamps already agreed to report on in the our letter to the White House – however, Flatiron has been the only school to follow this standard from the start, with Wyncode joining us last year. We hope to see more schools follow suit as our industry continues to grow. Until then, however, students must continue to do their research and ask the tough questions about bootcamp outcomes before making an investment.
