Stream editor and exercises

Stream editor sed

sed is a powerful tool to filter and transform text. We see some examples of sed commands:

  • sed '1d' filename : deletes first line;
  • sed '1,3d' filename : deletes first three lines;
  • sed -n '1p' filename : prints first line (-n tells sed not to print lines unless they match the following commands, in this case 1p that prints the first line);
  • sed 's/dino/mino/' filename : substitutes the first occurrence of dino with mino for each line. Adding g at the end (global) has the effect of applying substitution to any occurrences of dino. Option i makes the search case insensitive.
    A number indicates which occurrence should be replaced. For example, sed 's/ciao/hello/i2' is case insensitive and replaces the second occurrence of ciao with hello for each line;
  • Notice that it is possible to use a custom separator with substitution command. For example, sed 's:dino:mino:' filename
  • It is possible to indicate which rows should be examined as in sed '6,7s:dino:mino:' filename which only operates from row 6 to row 7;
  • In order to only print rows that match a string dino it is possible to use sed -n '/dino/p' filename.
    Analogously it is possible to use d to delete rows that match (in this case without -n);
  • sed 'y/abc/ABC/' replaces each occurrence of a,b and c with A, B and C, respectively.

Regular expressions in sed

sed supports regular expressions. For substitutions, it is useful to refer to the matched text. This can be done in different ways:

  • Using & it is possible to perform find-and-replace: & is substituted with the matched string. For example, in order to add hello before world we can use sed 's/world/hello &/g';
  • It is possible to refer to matched substring using back references and brackets. For example, to extract the name from a letter we can use sed 's/Dear \([^ ]*\) .*$/Name = \1/g'. Notice that the pattern we refer to is surrounded by \( and \), while the reference to it is \1. If we have more than one substring we can use \2, \3, and so on.

Exercise 1

Given a list of telephone numbers of the form 123456789 use sed to rewrite them as (123)456-789.

$ cat numeri.txt
123456789
392948291
321582923
321904984
Not a number
hello
$ cat numeri.txt | sed ...
(123)456-789
(392)948-291
(321)582-923
(321)904-984
Not a number
hello

Exercise 2

Use sed to select and convert all file names with suffix .html given as output by ls into capital letters with suffix .HTM. Check out command y in sed man page. An example of the expected output follows:

$ ls
document.pdf
myPage.html
test.html
$ ls | sed ...
MYPAGE.HTM
TEST.HTM
$

Exercise 3

Use sed to extract full user names from /etc/passwd/. For example:

$ sed ... /etc/passwd
Riccardo Focardi
MySQL Server
NVIDIA Persistence Daemon
...
$

Exercise 4

Try to solve as many levels as possible of the bandit wargame.

References

  1. http://www.tutorialspoint.com/unix/unix-regular-expressions.htm

NOTE: This page has been converted from asciidoc html to wordpress html using:
sed '1,/<body/d; /<\/body/,$d; s/<\/*body[^>]*>//g; s/<\/*div[^>]*>//g; s/<\/*p>//g; s/<pre><code>/<pre lang="bash">/g; s/<\/code><\/pre>/<\/pre>/g; s/^Last updated .*//g; /^\s*$/d' shell-english.html > shell.wordpress.html