Command line tips
Working through the command line, i.e. working with a text-only interface from a terminal, is a very powerful way to do precise and flexible data manipulation. Unfortunately, the command line has fallen out of use for a wide variety of reasons, non of which having to do with it being in any way inferior or more difficult. As a matter of fact, it is easier to do certain operations, in particular producing automated and reproducible processes, and it is the preferred method for a large group of scientists (not only programmers) and even just regular “power users”.
In the following section, I will gather some tricks that make it easier for you to work with command line tools like CQP. A terminal only handles text in- and output so mouse functionality is limited. Some keys and key combinations also migh show unexpected behavior.
The shell
The shell is what is running in your terminal and interprets your commands. It is a bit like the counterpart of the background process of Windows, Linux or macOS that draws windows. You can move through your directories, create and delete files, run programs and much more. For example, instead of moving your mouse over your browser icon (let’s say Firefox) and double clicking it, you would type the name of the program firefox and hit enter. cqp for example is a program that has a text-based interface, so if you type cqp in your terminal, you are opening in a different program. Some shell commands run programs that only have text output and no interface, such as wc. If you type wc -l and a filename wc -l myfile.txt, you run a program called wc which stands for ‘word count’ and you run it with the option -w which stands for ‘word’ and ask it to run on myfile.txt. It outputs the number of lines your file has. If you skip -l, you get the word count by default. In the end a shell is just a way to run programs like a graphical desktop is. The difference is that it is an interface which has full scripting ability.
Let’s say I have got a frequency list freqs.txt, and I need to know how many types of words are in there.
This is not an in-built function inside cqp, but (or rather because) you can easily do that in your shell like so: wc -l freq.txt
Your freq list has one word per line so the line count is your type frequency.
What if you left some words in there, that you should have excluded?
You could either rerun your cqp-queries and get a new data set or simply use shell commands to clean up your data.
If you want to remove every word ending in ing you could do this: grep -v ".*ing" myfile.txt | wc -l.
Here, you use the program grep to search for a regular expression (those should look familiar) with the -v option for invert to get everything excluding your search term. Then we ‘pipe’ it into wc -l from before. That means we pass the result of grep into wc. This time we should get our type frequency without words ending in ing.
Copy & Paste
Depending on your terminal program you might want to try the following things.
-
ctrl + shift + c(copy) andctrl + shift + v(paste) - middle mouse button (in Linux and macOS)
- Windows only: right click, (both copy and paste)
ctrl + c and ctrl + v have different meanings in most terminals. ctrl + c e.g. cancels the current process.
Colors
It might sound weird, but the default color scheme in a terminal is probably one of the main reasons people find working in the command line scary. Black on white or white on black (blue for Powershell) are ugly, hurt your eyes, and can make focussing for longer time periods difficult. My suggestion is, therefore, to switch this to a low contrast color scheme. Light gray text on a darker gray background is what I personally can work with best.
- Windows Powershell: right click on the top bar → properties → colors
- Windows Terminal: See here
- Putty: color menu is right on the left on the start screen
- macOS: right click on “Terminal” in the panel → properties →pr
There is a whole parallel universe of people for whom terminal color schemes are an art form (check out this subreddit). Since CQP does not produce colored output (yet), it is unfortunately not too useful for us.