Scripting (part 1) - it’s important
This is not a particular practice, more a whole approach.
Adopt, learn and use a scripting language. JFDI.
1) Decide on a scripting language for your IT dept
2) Learn it
3) Use it
Why is this important?
Scripting reduces repetitive work, and increases speed and accuracy.
If you have procedures in your documentation which have numerous small sequential steps, or require the administrator to check complex details, this is a real danger sign that you are using people to do things that computers do better (and more cheaply).
OK, if a procedure will only be followed once or twice, you can argue it’s not worthwhile to automate. But consider that a script has more advantages than just convenience:
- If you have to stop your prod system during a change, you can reduce downtime by kicking off one script vs many manual steps.
- A script gives you reliable results, i.e. more reproducible. Even if you only run it once in prod, as you test your change in your test environments you can refine your script to incorporate all the lessons learned. All that testing will be wasted if your execution in prod is spoiled by a typo or a missed step in a long procedure.
- A script also tends to be more up-to-date and reliable at tracking lessons learned than a document. When executing a script, if there’s a problem, it’s easy to incorporate an extra check right there. If a note has to be added to a separate doc in an unrelated system, many people will just jot it down in a notebook (or somewhere even less permanent) with the sincere intention of updating the doc later. If there is a problem in the script that prevents it executing properly, that will tend to get fixed urgently, whereas a typo in a doc often becomes something that everyone just remembers to correct as they work.
- If a problem does occur after a change, reviewing the script will show you exactly what was executed (as opposed to what the administrator remembers executing). Also, detailed logging in a script saves the person having to note every step, or save their session logs in case of later questions.
Some guidelines:
- Ensure that your scripts fail-safe, i.e. if something goes wrong, they exit rather than blindly carrying on. A standard error-handling function in the template can help with this, or a function that wraps the execution of each external command with some failsafe checks.
- Try to make your script re-executable. If it errors out midway, it should be safe to re-run it - the earlier steps should be safe to repeat, or should be intelligently skipped. It can also have arguments that allow execution to begin directly from particular points in the flow.
Note that with a small procedure-replacement script, I do not believe the full force of “Agile development” should be applied.
- Test-Driven Development isn’t easy to imagine (unless your test installation is extremely easy to return to base state).
- Comments are helpful, not clutter: bear in mind that scripting languages, particularly when calling external commands, can be extremely high-level in that one line can have large effects. Comments also explain why it does something - e.g. “this file is copied across to apply patch X; check first in case patch Y is already there”. Far from becoming misleading crud, this is essential.
- You should definitely have a version-control system for your scripts - you may find it convenient to use the same as your developers, but it’s important to keep it easy - something as simple as RCS may be sufficient.
The important principle with scripts is to make them time-effective - particularly for the first person to create the first prototype. If the startup time to create a script is too high, you’ll always have long sets of instructions lying around.
- Don’t insist on industrial levels of testing - if this makes you nervous, remember that a person executing a sequence of instructions is always untested; they could make a mistake at any point, however often they have done it before.
- Create templates for scripts that include logging and error-handling functions, automatic version info (e.g.
$Id$with RCS), so the basic framework is created instantly. - Create reusable modules for your environment, e.g. to get info from your LDAP or send an alert to your monitoring.
It can feel that creating a script is more work than human execution because you have to put in more error checking etc. Firstly, templates for failsafe execution functions as previously described can help with this. Secondly, you don’t have to check for everything - but because you get in the habit, you end up checking for all errors routinely, which is something a person following a procedure may not do. Suppose your procedure says “edit file XX and add line YY”, then has another instruction afterwards: “check the line is added with grep YY XX“. Will the person actually bother with the check, when they “know” they’ve done the edit right? In a script you can assume that the edit is successful (it has been automated after all), or you can enforce the check by simply adding the grep into the script.
Also…
Most of the above focuses on replacing procedures. Scripting can also be used for larger-scale work, which can easily overlap with “development”. This might include some of the tools described in this blog, all the glue that keeps your installation running, small web interfaces for your team to use or for your customers, etc.
This sort of thing is good - scripting in the same high-level language for all these uses can produce results far more quickly than launching a full IDE and associated development practices. However, as script development tends towards full-scale application development, you should introduce more of the practices around testing, version control, coding standards, etc. Consider how critical the system will be - a user interface with “helpful tip of the day” may not matter if it has outages, but admin scripts can quickly become more critical than you originally expected.