A sysadmin is a guy who makes servers work. These days that means helping management choose equipment and software, getting the equipment and software working together and maintaining it thereafter.
Servers form the core of a business’s information technology. They store all your data, the send your email all over the world and receive it, they keep track of your Internet sessions and phone calls, your credit card transactions and your driver’s license, your medical bills, everything you ever bought or sold, every transaction you made or action you took that there is any kind of electronic record for has touched on servers. To see this website you’ve contacted at least one webserver, and to find it you probably brushed on dozens.
Servers are the computers you never see because they’re locked (literally behind palm-locks and thumb scanners) in deafeningly loud air conditioned humidity-controlled dust-free rooms the size of football fields or tennis courts called ‘Data Centres’ nowadays, humming away and cheerfully helping you perform every part of modern life.
A sysadmin’s daily job typically comprises installing new servers, managing access to servers, backing up all that data, backing all that data up some more, updating and upgrading software on servers, decommissioning old servers and migrating data around. It means figuring out how to get the functionality his business or organisation needs, finding the most reliable and cost effective means of achieving it, convincing the business to adopt it and making it work thereafter.
Good systems administrators in good companies spend the bulk of their time preventing bad things from happening and finding new ways for IT to improve their organisation’s efficiency. Bad systems administrators or good ones in bad companies spend the bulk of their time fixing problems that come up - usually the same problems over and over again.
Here’s where I want to focus today’s article. These are the qualities I believe make a good SA - not things you can learn like experience or qualifications, but the intrinsic qualities you arrive on the scene with. I’ll deal with experience and qualifications in another article - this one will focus on the things you can’t learn - you either got it or you don’t.
All of the best sysadmins I’ve worked with were smart, and all of the bad ones were varying flavours of idiots. It’s that simple - intelligence is the most important trait. Computers are complex and server systems are the most complex of all. If you don’t think you’re smart, now is a good time to give up on this career.
Fact of life #2 is that geeks make the best sysadmins. If you’re not interested in computers then you’re going to quickly fall behind the guys who go home and play with their eight computers there. You aren’t going to learn about new technology as quickly and ou’re not going to work as hard at the weird little technical problems as the nerds, and you just aren’t going to care as much if the backups seem to take longer every day.
Nerds think about computers all the time, and their intense interest means they don’t miss the tiny clues and warning signs that let a good sysadmin stop trouble before it becomes a problem.
Most importantly, being interested in computers means that nerds learn about them more quickly. They’ll go home and play with the new technologies that interest them, they’ll mess around with them in their lunch breaks and if they’re doing personal web surfing it’s more likely to be a technical site than something irrelevant.
I’ve worked with a lot of different sysadmins in my time - and in my experience the ones who had no interest in computers outside of work could obtain competency in the field but no more. Their knowledge and skills always paled in comparison to the Nerds.
I know - this is going to sound weird - but the fact is that computers have absolutely no purpose whatsoever without people. If you don’t understand people and aren’t interested in them at all, you’re going to suck at systems administration. All your brains and nerd knowledge mean nothing if you’re totally incapable of matching the technology to the needs of the people and the organisation around you.
To a degree, you can learn this one. But if you aren’t interested - for the love of god go do something else. Science or something, I don’t know.
Believe it or not, that’s it. If you’ve got those three traits - or you have two and a half and are still building the third - you can one day be an SA. If not, give it a miss - the field is already full of mediocre bozos. One thing you don’t need is a penis - I’ve only worked with one female sysadmin in all that time, but she was one of the best.
So I got pretty well to the top of my field - without going overboard and becoming a super-specialist / cranky old 20 year veteran, before I got promoted out of the field a few weeks ago.
Before the memories fade too far I wanted to write down what led to my success - and mistakes I made that others could avoid - so that aspiring sysadmins could take advantage of my past few years work too. Especially aspiring sysadmins who have what it takes (See part 1 of this series to find out if you have what it takes) because in my new job, I need more of you guys around!
Part 2 - What course to do?
University? I’ll admit I’m probably biased against university. I didn’t go myself - when the time came to make that decision the courses open to me were near totally irrelevant to the business world outside. Old programming languages nobody used, achres of academic drivel, lecturers and tutors with little or no relevant industry experience… Maybe universities have changed since I last gave them a close look - but the feedback I’m getting from recent students says otherwise. As a rule, I think if graduates are still finding the real world bewilderingly unfamiliar after spending years supposedly learning about it, the mark has been missed.
Not only are universities struggling and largely failing to re-acquire relevance to the industry, but their archaic courses (computer science? what the hell is that? when did computer techs last wear lab coats?) are extremely generalist - they touch on Very Neat Programming For Dummies, Unix Administration Strictly By The Book but they don’t make you a sysadmin. True, only experience will do that, but for courses to get you started, there has to be a better option than university.
Depressed about uni now I’ve said all that? Cheer up - completing a degree gives you a lot of points. Sooner or later, not having _any_ degree is probably going to hurt my career. I haven’t hit that barrier yet, and when I do I might very well be faced with three years of uni, or more years ‘part time’ in my 30’s or 40’s - not so cool as going when you’re 18. University is also a great social mecca for young people that I missed out on. It’s got good points - if you’re halfway through a degree, stick with it and finish it. An unfinished degree won’t look so hot on your resume.
Again though - don’t kill yourself if you’ve got a half finished degree. If you’re young, you’ve probably heard a lot of bull from older people like that your life will end if you don’t get the right qualifications. This is crap - the most valued thing in this industry at least is experience - and you can only get that by doing the job. Qualifications are there to help you break down the door and get your first job, or beef up your resume against the other candidates when you’re looking to upgrade.
Cisco? Cisco courses are the bees knees if you want to be a network engineer. If you want to spend the rest of your life messing around with routing, firewalling, load balancing and doing a hell of a lot of planning for not much work - this may be your career of choice. If you’re obsessed with networking, it very probably IS your career of choice. But Cisco courses are not for systems administration. Cisco makes the world’s best routers and arguably hardware firewalls - not the world’s best servers or operating system.
Worse - Cisco courses are old-school. They include written / multiple choice exams, they include a load of old technology nobody uses anymore, and demand you memorise a lot of crap to pass the exams. Uh oh! Doesn’t that mean someone who’s good at memorising stuff but barely understands networks could pass? Well frankly YES for the entry level CCNA - but beyond that maybe not. It gets complicated.
Still, you’re on a better track here - industry certifications are far more valuable on your resume than a university degree, if systems administration is your thing. And don’t get it into your head that sysadmins don’t need networking. You need to understand netmasks and firewalls, basic routing and stuff - and a lot of the problems you face will be network related - so understanding networks will help you a lot.
I’m not saying don’t get a CCNA. I’m saying don’t rely on it to get you job as an SA. I’ve interviewed a few guys who knew nothing about anything but had a CCNA. They did not get the job. Every git and his goat seems to have a CCNA nowadays.
Microsoft? Warmer… Microsoft Certified Reboot Engineer isn’t the 60-minute-cram-the-night-before-pass-the-exam-still-clueless joke it used to be - and if Windows is your thing (Windows Servers aren’t the joke they used to be either) there is no other choice. Better yet, Windows is a perfectly valid back door into Unix administration - many positions involve a little of both so you can make the crossover. But it sure isn’t my pick - I’m really not into shrugging my shoulders when the boss asks me why the servers are screwing up. I like to have real answers. Preferably before the servers screw up in the first place - but that’s a rant for another day.
To me, looking at someone’s Resume, an MCSE is meaningful but no cigar. They’re still not that difficult to get, and they’re still marred by the old stigma of the laughably-easy… we’ve all met an ‘MCSE’ long ago who couldn’t figure out that DNS wasn’t configured on a desktop… ‘Why doesn’t The Internet work?’
RedHat? RedHat are my choice. If you take their pre-assessment exam
https://www.redhat.com/apps/training/assess/, you will see they have a range of courses for everyone from the total noob to the hardenened Unix veteran. My personal experience of RedHat courses is that they’re up to date, keenly relevant, completely practical, well taught, and the exams are right on the money - favouring practicality and “does it work?” above all else. Oh and I know how they got there too - there’s an exit survey on your way out of the course. They’ve been asking the best people in this field about how to improve their courses for years. Smart - as usual.
The upper echelons of RedHat certification - RHCT, RHCE, RHCA are no joke to obtain. I found RHCE to be extremely challenging, for all the rightreasons, after years of linux administration. If I see RHCE on someone’s resume - even if their experience isn’t quite up to what we’re asking - I’m likely to want to interview them. You just can’t get that certification without knowing what you’re doing - and that makes it valuable - and rare.
Other Linux? Two problems spring to mind here. Firstly, RedHat dominate the field of commercial linux. They do it better than anyone else, and both their product and their service kicks butt. Why do you want anyone else’s name on your resume? Second: Who? There’s a couple of really common Unix certs you can get, none of which are well known or understood. As a general rule, avoid all lesser-known certifications! What is the use of it, if nobody at the company you’re applying to has any clue what it is, or what it means you know?
Whichever way you cut it, doing courses will cut into your time and money - if you’re at the beginning of your career placing you in debt in some way. In my view, RedHat training is the best value for money, and the best value for time. Remember, your first 12 months on the job are more valuable than any qualification you’ll ever get. If you’re new, you want a qualification to GET you those first twelve months - if you want one at all.
If you’re an aspiring SysAdmin, you need to look at courses as a stepping stone to get you in the door of your first sysadmin job. Once you’re in that door, if you succeed, you’re made - because the most valued thing on your resume is experience.
Breaking down the door is the subject of our next exciting episode.
Breaking Down the Door
So you’re faced with a paradox.
For a Systems Administrator, the most valuable thing on his or her resume is experience. People without experience simply cannot compete for a job against people who have it. This field is completely hands-on and practical - most of the skills and knowledge you use, you will learn on the job.
So how the hell do you break in?
Before you even begin
Are you into Linux? Get a box, install linux on it, and start messing around with daemons. Don’t waste your time with stuff nobody uses - get CentOS (free recompile of RedHat)
http://www.centos.org/ and install it on something. Play with databases, webservers, email servers, webmail, samba - get a bunch of functionality working. Create a database in MySQL without phpmyadmin - do it with the command line. Build some tables, learn basic SQL. Move onto PostgreSQL, do the same thing. Learn to dump out databases and back them up. Learn how to use rsync, learn the command line options to tar. Put “bash tutorial” into google and learn basic bash scripting. Learn how to use find, watch, grep, cut, and sort. Get used to reading man pages and googling stuff when you get stuck. Learn to paste error messages into google.
If you’re into windows get a copy of 2000 server and 2k3 server and start playing with them. I don’t know how you’re going to afford that - buy educational copies or something. Get Exchange, get IIS, get them working. Get outlook talking to exchange. Get webmail working. Get an AD server going and learn how to manage groups.
Ideally, you want to have done most things a business will ask you to do before you start. More importantly, you absolutely MUST have learned HOW to learn things quickly. Where to look for information.
Second, focus on this one fact: You must get into a role where you’re performing Systems Administration work, and you must be willing to make any sacrifice necessary to do so. It’s worth almost anything to get 12 months of ‘Systems Administrator’ on your resume. Even a three month contract puts you 100% ahead of a guy with no experience.
Strategy One - the Direct Approach
I’ll include this for completeness sake.
- Get a big qualification (say all the way up to RHCE, or a degree or something)
- Apply to every “junior admin” role you see advertised. Sometimes people advertise ‘graduate’ positions.
- Keep getting interviews until you land a role.
I don’t know how well this one works. It certainly isn’t the approach I used. I imagine university graduates who scored really well soak up these handful of jobs, and best of luck to them.
I didn’t get any qualifications at all until I was several years in and had experience under my belt. But - if you can manage it - starting out with RHCE or MCSE if you’re into windows would help you a lot, and give you a chance to go straight in the front door like this.
Strategy Two - Small Business
Several ways to approach this one.
- Find a small computer/support business and apply for a job. If you can’t get a job, offer your services as a casual, or even apply for work experience. So long as you have some opportunity to interact with SERVERS. Start accruing months of experience working with servers. Apply your knowledge. If you can’t find a place with one of these businesses keep looking.
- Start servicing a bunch of small businesses yourself. Everyone has data, almost everyone has a computer. You’re going to have to deal with a lot of desktops at first, and your own business is a nasty proposition on so many levels, but this is better than being purely unemployed. Just make sure that within a few months you’re working on some servers - any servers. Run your own servers and supply services to small businesses with them. Host their damn backups remotely or something. Just RUN SERVERS.
Experience in small business isn’t the most valuable out there - but experience working in small business is priceless compared to no experience at all. A few years, maybe even a few months, working for small business and you can move across to a more exciting and lucrative area. But I’ll discuss that in another article.
Strategy Three - Back DoorsThis is the last approach. I list is last because out of all these approaches it’s the least direct and the most likely to trap you along the way.
Back doors are ways you can get into businesses that have systems administrators, and eventually maybe possibly perhaps become one yourself.
- Get a job in a network operations centre. You will no doubt have to man phones and monitor hideously boring things, rotate tapes and all kinds of other menial crap. BUT, big companies that have a NOC also have sysadmins - and odds are there’ll be at least one unix box you can log into here. This is the closest you can get to systems administration without being an SA. Prove yourself here and make sure everybody knows your aspirations, make every effort you can to learn and grow towards them. Don’t get stuck here for years. Be looking both outside and inside the company all the time for opportunities to move into that junior sysadmin role.
- Desktop support. Desktop support roles generally give you more of a chance to move into windows systems administration. If you’re smart, have an MCSE, play with windows servers at home etc, you may get the chance after a few months or a year or two. Again - make sure everyone knows your aspirations, keep learning, keep pushing yourself and proving every day that you’re the smartest desktop admin there - ready at any moment to move over into a junior admin role. Again, look constantly within and without your company for that opportunity.
- Helpdesk / L1 support - avoid this if you can. You wont be working anywhere near the sysadmins and you’re two steps removed in most companies. From here you probably need to push for desktop support or NOC style roles. Similar things apply - keep studying, fund more qualifications, keep playing with servers at home and make sure everyone knows what you want to be. Except here you probably want everyone to know you want to be a desktop support or NOC guy next - and a sysadmin later.
The road can be tough - but remember this: Varied experience helps, and nothing turns people on more when they read your resume than a progression up through the ranks. People who learned desktop support before they became systems administrators have valuable knowledge that the guy who jumped from uni to a junior admin role didn’t get. You learn all kinds of things answering phones to pissed off customers and teaching total dumbasses how to turn their computer on too. Things like patience and people skills that you’ll still be using as a senior SA.
Finally - don’t look on these options as hot coals you need to walk over to get what you want. Most of these jobs are fun, all of them pay money, all of them will give you experience and give you opportunities to learn and grow. Not to mention meet new people. Pick whichever of this option sounds like it will work for you - or sounds the most fun - and go for it. If it doesn’t work, try another. If you’ve got what it takes to be a Systems Administrator (see part one), you belong in the role and deserve to get there - so keep working on it. We need you.
One last piece of advice - if you’re in a small town with not much of an economy, move to the big city nearby. Don’t make this harder than it already is. Once you’ve broken down the door and got some experience as an SA, you can come back to your small town, if you still want to, and have a much better chance of getting one of those rarer SA roles there.
The Ultimate Systems Administrator
The Ultimate Systems Administrator doesn’t exist. There’s always someone better than you at something - or some aspect of it. But don’t ever let that stop you trying.
I wont let it stop me from trying to paint a portrait of this fictional server-ninja.
I’ll take the best traits from the best SA’s I’ve ever worked with, from myself if I have to, and where that falls short… I’ll put down what I wish we had - with the full benefit of hindsight. Strap in - we’re gonna do a little genetic engineering, a little personality engineering, a little training and a lot of building the ultimate career.
The Ultimate Systems Administrator - a Genetic Experiment Gone Horribly Right
Imagine you are the ultimate systems administrator.
You are highly intelligent. You love computers, and have been messing with their inner workings since childhood. You understand them backwards now.
You are extremely strong with your chosen operating system and software platforms. You know them inside out. You respond to mailing list queries on them, and maybe you write articles. You are a recognised authority in these circles.
Yet, you will willingly attack any task or problem asked of you. You never say “no, not my field”, but instead always “I don’t know it but I will after this!”. You can always be relied upon to give it a go - you are our backup person in a lot of areas.
You are brutally pragmatic with technology. If something better than your favourite tool comes along, you are the first to want to adopt it and let go of the old. You are sceptical and will prove it does the job first, but you will jump at the chance to improve your arsenal. You ask: What works? What’s the best? What fits our business the best?
You are obsessed with learning new technologies, new software, new hardware, new techniques. You are fearless in saying “I don’t know but I’ll find out” and tireless in pursuing that knowledge.
Yet, you also understand people. You like people and are a helpful person - always seeking to ensure that your customers both internal and external to our business are satisfied. You understand business too, and you like business. You accept the business’s priorities and recognise that together we stand, or together we fall. You know you are the meeting point between computers, people and business. You’re constantly on the lookout for new ways to apply technology and make things better.
You’re an expert in networking. Not necessarily hands-on router configuration, but you understand how tcp and udp work, you know what an arp packet is and you can picture how a network has to function without trying. You can see the packets.
You are highly trained. You hold qualifications from industry leading organisations, and you made the absolute most of that training - becoming an expert at the top of your field.
You have an almost uncanny ability to sense and work around problems; a trained intuition built on years of experience with every kind of issue, fault and artefact. You’ve seen it all, and when you see something new you learn immediately - continually refining your spooky abilities. You are direct when it works, indirect when it doesn’t. You let nothing stop you from getting the system working and keeping it working. Then you go back and you find out why, and you stop the problem coming back.
When there is an outage, you take it personally. You lose sleep because you have an idea on how to prevent the next one at 2am that night. You don’t lose sleep worrying about it.
You can handle the pressure. Any pressure. When you feel your human limits approaching, you adjust your expectations and your language to pace yourself. Not “I need to complete this by Friday” but “I want to complete this by Friday”. When you’re at 100% capacity, you recognise that you’re at capacity, let people know and accept it. Deadlines will slip - operational work will disrupt your projects. But the tasks will all, eventually, get done. You will stay at 100% until then. You don’t try for 110%, or 105%, or 101% and burn out. 100%, sustainable indefinitely.
You have excellent work and time management habits. Whenever you agree to do something you enter it into the ticketing system, or put it on your to-do list, or you scribble it down until you do. Nothing ever slips through the cracks. Every task is noted, every task is completed.
In your mind is a huge priority tree, encompassing every system and every part of our business. Project work, operational work. Service Level Agreements, internal needs. Loss of income vs loss of reputation. If thirty priorities conflict in a given day, you sort them out methodically and consistently on your priority tree and get them done in the right order.
You can draw our business’s entire Information System, including the parts you never work on, at a high level on a whiteboard from memory and understanding without pause, explaining to an audience how it all connects and works, and why as you draw. You can put names and faces next to each component you’ve ever touched upon - and explain what they know, when to go to them and why they are important.
Finally, you have a downright powerful moral compass. Your ethics are unquestionable. If you’re ever asked to do something you’re uncomfortable with you’ll politely refuse, explain why you can’t, and offer alternatives. Millions of dollars run under your fingers every day - you have to know right from wrong with absolute, and personal, certainty.
When you’re all of that, drop me a line - I’ll find room for you wherever I am, somehow.
Seriously though - no one person can do all of that, perfectly, every day. However, you can try; and why not? Why settle for what you are now when you can continue to grow towards The Ultimate?
