Why learn the command-line interface?
This blog post explains some good reasons for learning to use the terminal / command-line for data scientists.
When I first started programming as part of my job, it was around 20 years ago. I was a biologist, gradually acquiring hundreds of Excel spreadsheets from my lab results, and needed to compare results from one gene/spreadsheet to another. So I needed a tool to help with data analysis (which led me to R). But for each of the genes I was working with, I wanted to obtain some metadata about it - gene symbols, weights, where did it live in the cell, what processes was it involved in…
For this kind of stuff, I would look at a variety of websites (some lost to history) and copy annotations into my spreadsheets. It was long-winded and I knew it could be speeded up - the websites were just text, after all. So I started learning Perl. Perl was a very common language in Bioinformatics in the 2000s. It is exceptionally good at text processing and had some really good training resources/books. Nowadays, I’d recommend Python for the type of text-file processing work that I was using Perl for (though I still think Perl is great).
Configuring scripts ‘from the outside’
Why the long-winded intro to Perl and my ancient projects?
Well, my programming setup looked like this at the time:
- Windows computer with ActivePerl and R installed
- R running in the RGui (before RStudio existed)
- Perl/R scripts written in Notepad++
Initially, I would run Perl scripts by double clicking on a filename in the file explorer. That would start the Perl interpreter and run all the commands in the file. It also meant that if I wanted to run the processing script on a different set of inputs, I’d have to modify inside the source code of the script. So at the time, I didn’t have a way to configure scripts from the outside, to redirect their inputs and outputs.
The thing that made a huge difference was learning about the command line interface (CLI).
All the operating systems provide a CLI (possibly named ’terminal’, the ‘command prompt’ or the shell) with different commands and syntax. The CLI is a low-level way to interact with your computer/operating system. Your only interaction with the CLI is through typed commands, which feels like a very restricted way to interact with the computer but is actually really powerful - you can access and move about through all of the file system, you can look at memory/processor usage and other low-level details, you can start and stop scripts/programs, and so on. Many programs, like Git, were initially written to be used from the CLI and so interacting with these via the CLI, rather than a visual interface, provides access to the fullest suite of the available commands.
To configure a Perl script from the CLI, the syntax looks like this:
perl [perl-options] script.pl [script-arguments]
So, you can pass in arguments (either by position or name) that define input/output filepaths, making your script configurable and more reusable.
# Examples
perl my-script.pl input.csv output.csv
perl other-script.pl -- --input input.csv --output output.csv
You can similarly configure the running of a Python or R script:
python my_script.py arg1 arg2
python -m runnableModule arg1 arg2
Rscript the-script.R arg1 arg2
# or R <options> -f the-script.R arg1 arg2
How user-supplied arguments are handled inside your scripts varies across different programming languages. You can typically access these arguments as an array or list, though external packages may make working with arguments easier. For example, in Python you might use the Click package, and in R you might use {optparse} or {argparse}.
Environment management
Through necessity, Python users tend to be more comfortable with the CLI than R users. This is because installing Python packages and managing virtual environments (with pip, poetry, uv or similar) is done outside of a Python session, and usually from the CLI.
Once you have R and RStudio or Positron installed, very little of the management of R projects needs
to be done from the CLI.
In my current workflow, I tend to use the CLI tool
rig to switch between different R versions and then start RStudio
or Positron from the command line, so that on a given project I’m always using the correct version
of R - some clients use specific versions of R and sometimes it’s useful to check your code works
correctly on upcoming or older versions of R.
Project coordination
At Jumping Rivers, we use Linux on all of our programming laptops. We also use multiple programming languages across our projects. In the multi-language setting, the CLI becomes essential. Because it sits ‘outside’ of your project code, and can call R, python, quarto and so on, the CLI (and languages like bash on Linux, which make the CLI work) is a sensible place to coordinate the running of a project.
Tools like ‘make’ can be used to coordinate running different parts of a project. The book Data Science at the Command Line by Jeroen Janssens (available free online) contains a chapter that explains how to use ‘make’ to coordinate a multi-step project. There are alternative project-coordination/data-pipeline tools, some written to be more data science focussed than ‘make’ (which has been used to build software for decades).
Text-analysis tools
On Linux / Unix, the CLI makes a range of tools available that you can use to summarise datasets / files. I use the following commands extremely often:
| Tool | Purpose |
|---|---|
grep | Search text for lines matching a pattern (plain text or regex) |
sed | Stream editor — find/replace or otherwise transform text line-by-line |
awk | Pattern-scanning and processing language — good for column-based/field-based data extraction and simple calculations |
sort | Sort lines of text (alphabetically, numerically, by field, etc.) |
uniq | Remove or count duplicate adjacent lines (usually paired with sort first) |
cut | Extract specific columns/fields from each line (by delimiter or character position) |
wc | Count lines, words, or characters/bytes in a file |
The commands can be chained together using the | pipe character.
Summary
In this short blog post, we’ve covered the basics of why learning how to use the command line interface can benefit you as a data scientist. You’ve seen that working from the CLI allows you to write more general purpose scripts, to coordinate projects, and to manage the environments within which projects run. We’ve also touched on some linux tools that help summarise files and mentioned how tools like Git provide the fullest set of their functionality to CLI users.
