1-DAV-202 Data Management 2023/24
Previously 2-INF-185 Data Source Integration

Materials · Introduction · Rules · Contact
· Grades from marked homeworks are on the server in file /grades/userid.txt
· Dates of project submission and oral exams:
Early: submit project May 24 9:00am, oral exams May 27 1:00pm (limit 5 students).
Otherwise submit project June 11, 9:00am, oral exams June 18 and 21 (estimated 9:00am-1:00pm, schedule will be published before exam).
Sign up for one the exam days in AIS before June 11.
Remedial exams will take place in the last week of the exam period. Beware, there will not be much time to prepare a better project. Projects should be submitted as homeworks to /submit/project.
· Cloud homework is due on May 20 9:00am.


Difference between revisions of "Command-line basics"

From MAD
Jump to navigation Jump to search
Line 4: Line 4:
 
* A folder can also contain other folders, forming a tree structure
 
* A folder can also contain other folders, forming a tree structure
  
===Moving around folders (ls, cd) ===
+
==Moving around folders (ls, cd) ==
 
* One folder is always selected as the current one; it is shown on the command line
 
* One folder is always selected as the current one; it is shown on the command line
 
* The list of files and folders in the current folder can be obtained with the <tt>ls</tt> command
 
* The list of files and folders in the current folder can be obtained with the <tt>ls</tt> command
Line 27: Line 27:
 
</pre>
 
</pre>
  
===Absolute a relative paths===
+
==Absolute a relative paths==
 
* '''Absolute path''' determines how to get to a given file or folder from the '''root''' of the whole filesystem
 
* '''Absolute path''' determines how to get to a given file or folder from the '''root''' of the whole filesystem
 
* For example <tt>/tasks/perl/</tt>, <tt>/tasks/perl/series.tsv</tt>,  <tt>/home/username</tt> etc.  
 
* For example <tt>/tasks/perl/</tt>, <tt>/tasks/perl/series.tsv</tt>,  <tt>/home/username</tt> etc.  
Line 42: Line 42:
 
Commands <tt>ls</tt>, <tt>cd</tt> and others accept both relative and absolute paths.
 
Commands <tt>ls</tt>, <tt>cd</tt> and others accept both relative and absolute paths.
  
===Important folders===
+
==Important folders==
 
* '''Root''' is the folder with absolute path <tt>/</tt>, the starting point of the tree structure of folders
 
* '''Root''' is the folder with absolute path <tt>/</tt>, the starting point of the tree structure of folders
 
* '''Home directory''' with absolute path <tt>/home/username</tt> is set as the current folder after login
 
* '''Home directory''' with absolute path <tt>/home/username</tt> is set as the current folder after login
Line 48: Line 48:
 
** Tilde <tt>~</tt> is an abbreviation for your home directory. For example <tt>cd ~</tt> will place you there
 
** Tilde <tt>~</tt> is an abbreviation for your home directory. For example <tt>cd ~</tt> will place you there
  
===Wildcards===
+
==Wildcards==
 
* We can use wildcards to work with only selected files in a folder
 
* We can use wildcards to work with only selected files in a folder
 
* For example, all files starting with letter x in the current folder can be printed as <tt>ls x*</tt>
 
* For example, all files starting with letter x in the current folder can be printed as <tt>ls x*</tt>
Line 54: Line 54:
 
* All files containing letter x anywhere in the name can be printed as <tt>ls *x*</tt>
 
* All files containing letter x anywhere in the name can be printed as <tt>ls *x*</tt>
  
===Examining file content (less)===
+
==Examining file content (less)==
 
* Type <tt>less filename</tt>
 
* Type <tt>less filename</tt>
 
* This will print the first page of the file on your screen. Then you can move around the file using space or keys Page up and Page down. You can quit the command pressing letter q (abbreviation of quit). Help with additional keys is accessed by pressing h (abbreviation of help)
 
* This will print the first page of the file on your screen. Then you can move around the file using space or keys Page up and Page down. You can quit the command pressing letter q (abbreviation of quit). Help with additional keys is accessed by pressing h (abbreviation of help)
Line 64: Line 64:
 
</pre>
 
</pre>
  
===Kopírovanie súborov (cp)===
+
==Creating new folders (mkdir)==
* <tt>cp odkiaľ kam</tt>
+
* To create a new folder (directory), use a command of the form <tt>mkdir new_folder_path</tt>
* Skopíruje súbor <tt>odkiaľ</tt> na miesto <tt>kam</tt>
+
* The path to the new folder can be relative or absolute
* Môžeme použiť absolútne alebo relatívne cesty.
+
* For example, assume that we are in the home folder, the following two commands both create a new folder called test and folder test2 within it.
* Cieľové miesto <tt>kam</tt> môže byť adresár alebo celé meno súboru
 
 
 
'''Príklad:''' ak aktuálny adresár je /projects/data-ppb/, nasledujúce tri príkazy všetky kopírujú súbor README do poadresára events:
 
 
<pre>
 
<pre>
# relatívne cesty
+
mkdir test
cp README events/
+
# change the next command according to your username
# absolútne cesty
+
mkdir /home/username/test/test2 
cp /projects/data-ppb/README /projects/data-ppb/events/
 
# celé meno súboru
 
cp README events/README
 
 
 
# tento príkaz kopíruje do /projects/data-ppb/events/README2
 
cp README events/README2
 
# ak sa presunieme do adresára events, môžeme kopírovať súbor do aktuálneho adresára .
 
cd events
 
cp ../README .
 
 
</pre>
 
</pre>
  
===Kopírovanie súborov zo servera/na server (scp)===
+
==Copying files (cp)==
* Kríženec medzi ssh a cp (Secure CoPy)
+
* To copy files, use a command of the form <tt>cp source destination</tt>
* Na server (spustite na vašom linuxovom počítači napr. v učebni): <tt>scp subor meno@vyuka.compbio.fmph.uniba.sk:nove_meno_suboru</tt>
+
* This will copy file specified as source to the destination.
* Zo servera: <tt>scp meno@vyuka.compbio.fmph.uniba.sk:subor nove_meno_suboru</tt>
+
* Both source and destination can be specified via absolute or relative paths.
* Ak chcete kopírovať súbory medzi serverom a Windowsovým počítačom, nainštalujte si program [[Softvér na stiahnutie|WinSCP]]
+
* The destination can be a folder into which the file is copied or an entire name of the copied file
 +
* We can also copy several files to the same folder as <tt>cp file1 file2 file3 destination</tt>
  
 +
'''Example:''' Let us assume that the current directory is /home/username and that directories test and test2 were created as above. The following will copy file /tasks/perl/reads-small.fastq to the new directory test and afterwards also to the current folder (whic is th ehome directory). In the third step it will be copied again to the current folder under a new name test.fastq. In the final steps it will be copied to the test directory under this new name as well.
 
<pre>
 
<pre>
#skopíruje súbor README2 do adresára /projects/data-ppb/ na serveri
+
# change the next command according to your username
scp README2 hrasko37@vyuka.compbio.fmph.uniba.sk:/projects/data-ppb/
+
# this copies an existing file to the home directory using absolute paths
 +
cp /tasks/perl/reads-small.fastq /home/username/
 +
# now we use relative paths to copy the file from home to the new folder called test
 +
cp reads-small.fastq test/
 +
# now the file is copied within current folder under a new filename test.fastq
 +
cp reads-small.fastq test.fastq
 +
# change directory to test
 +
cd test
 +
# copy the file again from the home directory to the test directory under name test.fastq
 +
cp ../test.fastq .
 +
# we can copy several files to a different folder
 +
cp test.fastq reads-small.fastq test2/
 +
</pre>
  
#skopíruje súbor README2 do domovského adresára užívateľa hrasko37 na serveri
+
==Other file-related commands (cp -r, mv, rm, rmdir) ==
scp README2 hrasko37@vyuka.compbio.fmph.uniba.sk:
+
* Copying whole folders can be done via <tt>cp -r source destination</tt>
 
+
* While using <tt>cp</tt>, it is good to add -i option which will warn us in case we are going to overwrite some existing file. For example:
#skopíruje súbor README2 do domovského adresára užívateľa hrasko37 pod menom README3
+
<pre>
scp README2 hrasko37@vyuka.compbio.fmph.uniba.sk:README3
+
cd ~
 
+
cp -i reads-small.fastq test/
#skopíruje súbor README2 z domovského adresára užívateľa hrasko37 na serveri do aktuálneho adresára
 
scp hrasko37@vyuka.compbio.fmph.uniba.sk:README2 .
 
 
</pre>
 
</pre>
 +
* To move files to a new folder or rename them, you can use <tt>mv</tt> command, which works similarly to <tt>cp</tt>, i.e. you specify first source, then destination. Option -i can be used here as well.
 +
* Command <tt>rm</tt> will delete specified files, <tt>cp -r</tt> whole folders
 +
* An empty folder can be deleted using <tt>rmdir</tt>
  
==Poznámky==
+
===Beware: be very carefull on the command-line===
===Upozornenie: dvakrát meraj, raz rež===
+
* The command-line will execute whatever you type, it generaly does not ask for confirmation, even for dangerous actions
* Príkazový riadok spraví, čo napíšete, nepýta sa, či to myslíte naozaj
+
* You can very easily remove or overwrite some important file by mistake
* Príkazy cp a scp môžu prepísať už existujúce súbory
+
* There is no undo
* Neexistuje undo
+
* Therefore always check your command before pressing Enter. Use -i switch for cp, mv, even rm.
* Preto si vždy dobre premyslite, čo chcete spraviť a skontrolujte príkaz pred tým, ako dáte Enter
 
 
 
===Zjednodušenie práce===
 
Užitočné pomôcky na príkazovom riadku:
 
* kláves Tab
 
** ak je len jeden súbor alebo adresár, ktorý pasuje na rozpísaný začiatok slova, doplní ho automaticky
 
** ak je súborov alebo adresárov viac, doplní, čo majú spoločné,po opakovanom stlačení ponúkne možnosti
 
* šípky hore/dole
 
** prechádzanie históriou spustených príkazov
 
* copy&paste myšou
 
** ľavým tlačidlom a ťahaním po texte označíme
 
** kliknutím stredného tlačidla (kolieska) vložíme kam potrebujeme
 
** ak nemáme stredné tlačidlo, klikneme naraz pravým aj ľavým
 
  
 
===See also===
 
===See also===
 
* [http://tldp.org/LDP/gs/node5.html Linux tutorial]
 
* [http://tldp.org/LDP/gs/node5.html Linux tutorial]

Revision as of 17:17, 12 February 2023

Files and folders

  • Images, texts, data, etc. are stored in files
  • Files are grouped in folders (directories) for better organization
  • A folder can also contain other folders, forming a tree structure

Moving around folders (ls, cd)

  • One folder is always selected as the current one; it is shown on the command line
  • The list of files and folders in the current folder can be obtained with the ls command
  • The list of files in some other folder can be obtained with the command ls other_folder
  • The command cd new_folder changes the current folder to the specified new folder
  • Notes: ls is an abbreviation of "list", cd is an abbreviation of "change directory"

Example:

  • When we login to the server, we are in the folder /home/username.
  • We then execute several commands listed below
  • Using cd command, we move to folder /tasks/perl/ (the computer does not print anything, only changes the current folder).
  • Using ls command, we print all files in the /tasks/perl/ folder.
  • Finally we use ls /tasks command to print the folders in /tasks
username@vyuka:~$ cd /tasks/perl/
username@vyuka:/tasks/perl$ ls
fastq-lengths.pl  reads-small.fastq  reads-tiny-trim1.fastq  series.tsv
protocol.txt      reads-tiny.fasta   reads-tiny-trim2.fastq
reads.fastq       reads-tiny.fastq   series-small.tsv
username@vyuka:/tasks/perl$ ls /tasks
bash  bioinf1  bioinf2  bioinf3  cloud  flask  make  perl  python  r1  r2

Absolute a relative paths

  • Absolute path determines how to get to a given file or folder from the root of the whole filesystem
  • For example /tasks/perl/, /tasks/perl/series.tsv, /home/username etc.
  • Individual folders are separated by a slash / in the path
  • Absolute paths start with a slash
  • Relative path determines how to get to a given file or folder from the current folder
  • For example, if the current folder is /tasks/perl/, the relative path to file /tasks/perl/series.tsv is simply series.tsv
  • If the current folder is /tasks/, the relative path to file /tasks/perl/series.tsv is perl/series.tsv
  • Relative paths do not start with a slash
  • A relative path can also go "upwards" to the containing folder using ..
  • For example, if the current folder is /tasks/perl/, the relative path .. will give us the same as /tasks and ../../home will give us /home

Commands ls, cd and others accept both relative and absolute paths.

Important folders

  • Root is the folder with absolute path /, the starting point of the tree structure of folders
  • Home directory with absolute path /home/username is set as the current folder after login
    • Users typically store most of their files within their home directory and its subfolders, if there is no good reason to place them elsewhere
    • Tilde ~ is an abbreviation for your home directory. For example cd ~ will place you there

Wildcards

  • We can use wildcards to work with only selected files in a folder
  • For example, all files starting with letter x in the current folder can be printed as ls x*
  • The star represents any number of characters in the filename
  • All files containing letter x anywhere in the name can be printed as ls *x*

Examining file content (less)

  • Type less filename
  • This will print the first page of the file on your screen. Then you can move around the file using space or keys Page up and Page down. You can quit the command pressing letter q (abbreviation of quit). Help with additional keys is accessed by pressing h (abbreviation of help)
  • If you have a short file and to just print it all on your screen, use cat filename
  • Try for example the following commands:
less /tasks/perl/reads-small.fastq  # move around the file, then press q
cat /tasks/perl/reads-tiny.fasta    # see the whole file on the screen

Creating new folders (mkdir)

  • To create a new folder (directory), use a command of the form mkdir new_folder_path
  • The path to the new folder can be relative or absolute
  • For example, assume that we are in the home folder, the following two commands both create a new folder called test and folder test2 within it.
mkdir test
# change the next command according to your username
mkdir /home/username/test/test2  

Copying files (cp)

  • To copy files, use a command of the form cp source destination
  • This will copy file specified as source to the destination.
  • Both source and destination can be specified via absolute or relative paths.
  • The destination can be a folder into which the file is copied or an entire name of the copied file
  • We can also copy several files to the same folder as cp file1 file2 file3 destination

Example: Let us assume that the current directory is /home/username and that directories test and test2 were created as above. The following will copy file /tasks/perl/reads-small.fastq to the new directory test and afterwards also to the current folder (whic is th ehome directory). In the third step it will be copied again to the current folder under a new name test.fastq. In the final steps it will be copied to the test directory under this new name as well.

# change the next command according to your username
# this copies an existing file to the home directory using absolute paths
cp /tasks/perl/reads-small.fastq /home/username/
# now we use relative paths to copy the file from home to the new folder called test
cp reads-small.fastq test/
# now the file is copied within current folder under a new filename test.fastq
cp reads-small.fastq test.fastq
# change directory to test
cd test
# copy the file again from the home directory to the test directory under name test.fastq
cp ../test.fastq .
# we can copy several files to a different folder
cp test.fastq reads-small.fastq test2/

Other file-related commands (cp -r, mv, rm, rmdir)

  • Copying whole folders can be done via cp -r source destination
  • While using cp, it is good to add -i option which will warn us in case we are going to overwrite some existing file. For example:
cd ~
cp -i reads-small.fastq test/
  • To move files to a new folder or rename them, you can use mv command, which works similarly to cp, i.e. you specify first source, then destination. Option -i can be used here as well.
  • Command rm will delete specified files, cp -r whole folders
  • An empty folder can be deleted using rmdir

Beware: be very carefull on the command-line

  • The command-line will execute whatever you type, it generaly does not ask for confirmation, even for dangerous actions
  • You can very easily remove or overwrite some important file by mistake
  • There is no undo
  • Therefore always check your command before pressing Enter. Use -i switch for cp, mv, even rm.

See also