Welcome to episode 154 of the Nerd Journey Podcast [@NerdJourney]! We’re John White (@vJourneyman) and Nick Korte (@NetworkNerd_), two Pre-Sales Technical Engineers who are hoping to bring you the IT career advice that we wish we’d been given earlier in our careers. In today’s episode we share part 1 of our interview with Tom Hatch, the creator of Salt and founder of SaltStack. Tom will share how he got into the industry through a love for public speaking, tips for interviewees and interviewers, and share the difficult challenges along the way.

Original Recording Date: 11-12-2021

Topics – Meet Tom Hatch, A Love for Public Speaking, Interviews and Finding the Passionate Few, Learning to Interview Other People, The Intelligence Community, Stepping into Automation (The Birth of Salt), Working Yourself to Death, Difficult Changes

3:50 – Meet Tom Hatch

• Tom Hatch is the original creator of Salt, eventually founding a company called SaltStack.
  • With SaltStack being acquired by VMware in late 2020, Tom works as a Senior Staff Engineer, moving things forward with the SaltStack group.
  • Tom’s background is entrepreneurship, business management, and software development.
• Tom wasn’t raised a nerd and did not really get into technology until he was in his late teens, building computers. He learned to write software in his mid 20s as part of a career shift.
• Tom had started down a business management path. After his first startup failed, he realized how much he liked working with computers.
  • He spent a lot of time learning about software and learning how Linux worked to better understand how computers worked.
• Back in 2007 Tom got a job at Red Hat as an instructor (first real tech job out of college). The company flew him all over the place.
  • Tom was teaching classes on Red Hat Linux and administering the Red Hat Certified Engineer exam. It was a lot of fun.
  • Tom ended up dropping out and not finishing his bachelor’s degree. He heard about a local company that did Red Hat training and sent the company his resume out of the blue to see what happened.
  • Tom was surprised when the company called him in for an interview. He was asked to teach people how Linux permissions worked, and that was the interview.
  • At the end of the interview the guy who owned the company said Tom’s technical skills needed some work but that he could talk, which was really hard to find. They were going to send Tom to a class, and if he passed the test over it he would be hired as an instructor.
  • Tom was thrilled, wanting to get a good job. He was sent to a Red Hat Certified Engineer class.
  • Previous to this, Tom had not been a Systems Administrator but just tinkered now and then.
  • After studying his brains out, he passed the test. Tom didn’t realize at the time that very few people pass the test on their first try.
  • As promised, the company hired Tom.
• His first assignment was to teach a class in St. Louis. At the time he was pretty scared. The night before the class started he wasn’t sure he would be able to talk for a full week on the topics.
  • Once he started teaching, it was magic. He kept talking the whole week.
  • Tom said this was one of those life moments where he realized there was something he was really passionate about and something he was good at doing.
  • At the end of the class, someone put into the review that it was refreshing to have someone with a PhD in the subject (which Tom didn’t have and never said he did).
  • He left that week feeling like "I can do this." It was a transformative experience from just a guy wanting a good job to break into tech. All of a sudden Tom had a certification, was getting great reviews, and was becoming someone who can administer the exam. The level of transformation was mind boggling.

9:26 – A Love for Public Speaking

• Tom has always loved public speaking, even when he was a teenager, and any opportunity to give presentations is something he would seize.
• Tom started a company right after getting married in his early 20s doing media duplication which went under.
  • Once he sold everything, he relocated the family and went to Southern Utah University because they let him in.
  • Tom started taking a class, and the professor stopped showing up. The first time this happened, Thom thought nothing of it. When it happened again, Tom told everyone he would teach the lesson because he had read ahead.
  • The professor only showed up for roughly 1/3 of his classes that term, and Tom taught the remainder of them. He was written in as best Computer Science Professor that semester.
  • Does this mean Tom is "broken that way" or that he has a superpower? Nick feels it is the latter.
• The trick to learning things quickly is to not worry too much about learning them quickly.
  • Tom likes to wander around a subject and dive in deep.
  • One of the things they would do at SaltStack is hire people with a passion for the subject matter.
  • If you have a passion for the subject, if it means a lot to you, it will be something you will dig into deep.

12:47 – Interviews and Finding the Passionate Few

• The interview process followed at SaltStack helped find passionate people.
• This would start with surface level questions. Depending on how well the person answered, the interviewer would dig deeper until a candidate ran out of steam.
• You can tell someone is passionate because they independently went deep. The passion matters a lot more than just learning quickly.
• The passion is something internal and that drives a person emotional. You develop a depth of understanding about the foundations of the technology you’re working with, which is exciting to Tom.
• What kind of answer to a question you don’t know the answer to communicates you have the passion and desire to go find out?
  • It’s about how much they take personal ownership for their answers.
  • "I don’t know is one of my favorite things to hear in an interview." – Tom Hatch
  • Every now and then you interview someone who refuses to say they don’t know (which acknowledges shortcomings). You don’t want to hire them.
  • Tom wants to hear where someone’s knowledge ends but even more than that wants to know that someone will take personal ownership and accountability for who they are. This means they will be an upright person who can be relied upon.
  • You can’t count on someone who can’t own their own self and where they are in life.
• Nick recently saw a Reddit thread where someone admitted to lying about their skillset in interviews and ended up getting the job who was not sure what to do.
• Tom says you can do a lot with confidence in the interview process.
  • He has interviewed a number of people who have been at the VP level who were really good at the interview but didn’t actually know what they were doing. It was their confidence that got them previous roles.
  • In situations like this Tom likes to ask more questions. He’s called up former employers of the super confident that advised against hiring these folks.
  • Watch out for overconfidence, and be willing to call people out when they don’t know how something works. A sniff test may be in order.
    • This isn’t about being rude to the person but finding out what they do and do not know.
    • It’s so much harder to get rid of someone if you have made a hiring mistake than it is to hire a new employee.
• Even though a job requirement may say a certain certification is needed, do not let it keep you from applying.
  • You’re selling yourself short by not giving it a try. You apply anyway, and if they don’t call you back, ok. Don’t worry about it.
  • So many of us are afraid to step up to a challenge put in front of us, and we’re afraid of being rejected.
  • Being rejected is one of the most beautiful things in the world. Tom loves being rejected because there really is a silver lining to it. You learn something and may have dodged a bullet.
  • Most people step back when told no and think "I’m not good enough." But it doesn’t mean that. It means the puzzle piece didn’t fit. Go find some other part of the puzzle.

21:18 – Learning to Interview Other People

• In 2008 Tom was laid off from his job as an instructor. He worked for a company called Applied Signal Technology, a US Intelligence community contractor (to 3-letter government entities) that was later acquired by Raytheon.
• During this time Tom worked with some of the most brilliant engineers. The way they interviewed allowed Tom to begin to understand how to do it.
  • These folks would bring a candidate in and put them through a series of panel interviews to grill the candidate. But it was always done in a respectful way.
  • They were looking for technical competency, but there were still some shortcomings in the way these people interviewed.
  • Most of the engineers Tom worked with were fresh out of college, and they would ask a lot of introductory questions.
  • It’s important to look for a mixture of knowledge and wisdom in a candidate, but the engineers Tom mentioned were extremely knowledge focused.
• At SaltStack Tom started using this model but augmented it, trying to add elements to find passion, experience, and pivoting from technical questions to understanding how someone is going to navigate a difficult task.
  • What you get from this is some insight into how the person will function inside a specific organization or team.
  • A number of times Tom has passed over someone with more technical knowledge and experience only to land on someone with less because he was able to determine that was the person he needed the most instead of the one who impressed him the most.
• There’s a difference between an interviewer being aggressive compared to disrespectful.
• Also remember you’re interviewing them as much as they are interviewing you. This is tech. If you don’t want to work with that person, don’t. There are jobs out there.

25:54 – The Intelligence Community

• Working for the intelligence community was a great job.
• The initial interview was targeted toward technical capabilities, but after that an office manager shared with Tom that the role required a top secret security clearance and what that would entail, stating if any of that was a problem he would not be able to keep the job.
• It took almost 5 months to complete the security clearance and 2 weeks spent filling out all the paperwork before that.
  • In the end, it was incredibly thorough.
  • The checks are often looking for liabilities that could be exploited.
  • Those who have betrayed the intelligence community often times have had money problems and drug addictions (many other issues stem from these).
• Nick has heard this kind of clearance opens a lot of doors for people.
• The work Tom did as part of having that clearance was intense but also deeply satisfying.
• Tom was writing software to automate different things. He was on a team of brilliant people.
  • They were helping with top level terrorist situations happening at the time, and it was exciting and inciteful. Though he could not discuss any of the details even with his family, he felt honored to work on that kind of stuff.

30:04 – Stepping into Automation (The Birth of Salt)

• Tom was working on systems automation in the early days of DevOps.
• After leaving the intelligence community, he worked for a small startup called Beyond Oblivion. At the end of 2011, they failed as a company.
  • Tom was working there as the Infrastructure Architect and used Puppet to automate all of the Linux systems.
• In February 2010, Tom had a brain tumor removed. The left side of his tongue is paralyzed as a result, and the left side of his brain stem has been compressed.
  • His left vocal cord is paralyzed.
  • He could not talk for about 2 years as a result of this and could barely whisper.
• Tom had been playing around with fan out command and control since college, and he had iterated on it a few times.
  • He got approval from the company to make an open source project which contained a fan out technology that was crazy fast.
  • Someone at work mentioned to Tom he could do a better job than Puppet. Tom said no, feeling he would have to write a programming language for this and would also have to grow a beard (listen to the story about why a beard seemed fitting).
  • He figured out a way to do this without authoring a programming language, adding this into Salt for declarative idempotent systems management.
  • The office director came to Tom and stated everyone hates Puppet. This person knew Tom was working on something that might be better. When Tom said it was something he was working on for fun, the person said, "it’s not for fun anymore. Make it work."
  • The next 1.5 months was spent making it work to properly automate systems.
  • That drive is what got Tom over the line to get Salt to function.
  • There’s a lot to be said about having a real user to drive development.
  • There is nothing like the pressure of having to make something work.

34:42 – Working Yourself to Death

• This point Tom was getting up, going to work, coming home, spending time with family for a little while, and working on Salt until the wee hours of the morning…repeating this routine day after day from late 2011 until sometime in 2014 when Tom ended up in the hospital with heart problems.
  • He had to dial things back a little as a result.
  • Nick references the interview with Keiran Shelden on You, Your Health, and the Datacenter.
  • In the tech industry, we don’t always keep health at the forefront.
• It is an insane amount of work maintaining an open source project, but the reality is you can take care of yourself and work really hard.
  • Finding that balance is really important. Balance is hard, and that’s just life.
  • A good life is a balanced life.
  • If you’re not trying to find balance, you’re not owning your own shortcomings, you’re not being honest about the problems in front of you, you’re not finding balance.
  • It’s worth it to dial things back and constantly evaluate.
  • Ask yourself…
    • What’s really going on out there?
    • How does this stuff really work?
  • What is really happening in the world isn’t what is in the news. It’s not what is in the conspiracy theories.
    • It’s a bunch of human nature. People are crazy and do silly things.
  • This world is complicated. Let it be complicated, and try to live a happy life.

38:16 – Difficult Changes

• It was hard for Tom to make changes, and he doesn’t feel like he did it as well as he believes he should have.

• He was the CTO of SaltStack at the time, and there was a great deal of growth involved in being a CTO.

• Tom made a lot of mistakes along the way. Deep down he just wanted to be an engineer.

  • It was incredibly scary and intimidating to have to fire someone, to figure out how to make payroll, and to go into a board meeting and start yelling at people.
  • He didn’t have the fortitude to do what he felt should have been done, which is a big regret.
  • Tom realized he was hiding from a lot of the aspects of running a business by focusing so much on the code.

• Tom had to step out of the code and learn monetization strategies, for example. It was switching gears, realizing it was not just dialing back on the amount but that there were new things he needed to learn. He needed to stop chasing the comfortable path and start chasing the discomfort.

  • Tom had to understand how to get things accomplished without him being the driving force behind it or the only one doing it. He needed to understand what drives sales and monetization of software and where automation fit.

• Preventing working yourself to death is not just about dialing back the hours. If you’re working yourself to death, there is something else driving it.

  • Are you desperate to get out of a situation?
  • Is it because you feel a weight of responsibility you shouldn’t feel?
  • Is it because you’re passionate about the wrong things?
  • It’s more about trying to find the root problem.

• When Tom got out of the hospital, he knew he had to fix things. But he didn’t know what the core thing was he needed to fix.

  • He didn’t get it fixed for another 5 years.
  • The core thing was not code, which is where his obsession was. It was the problems in his own personal life he needed to clean up (how he viewed himself, responsibilities to his family, difficult personal situations, etc.).
  • Tom was working himself to death to escape from other personal problems he didn’t have the guts to look at.
  • Nick compares this to root cause analysis for the full human, perhaps the hard work we do not always want to do.
  • It requires looking at yourself with brutal honesty and recognizing what needs cleanup. But at the same time, you can’t beat yourself up too much.
  • Recognize you may have to do hard things to resolve the problems. Only then will you be able to find your path and find the inner peace that allows you to sleep at night.
  • That stuff is what’s really hard, but often times if you’re being driven to an unhealthy place, there’s something unhealthy driving you there.

• For further reading based on the intro and outro of this episode, check out:

  • Episode 42 and Episode 43 with Ethan Banks and John’s reference to the Dunning Krueger Effect

Contact us if you need help on the journey.