1-DAV-202 Data Management 2023/24
Previously 2-INF-185 Data Source Integration

Materials · Introduction · Rules · Contact
· Grades from marked homeworks are on the server in file /grades/userid.txt
· Dates of project submission and oral exams:
Early: submit project May 24 9:00am, oral exams May 27 1:00pm (limit 5 students).
Otherwise submit project June 11, 9:00am, oral exams June 18 and 21 (estimated 9:00am-1:00pm, schedule will be published before exam).
Sign up for one the exam days in AIS before June 11.
Remedial exams will take place in the last week of the exam period. Beware, there will not be much time to prepare a better project. Projects should be submitted as homeworks to /submit/project.
· Cloud homework is due on May 20 9:00am.


Difference between revisions of "HWcloud"

From MAD
Jump to navigation Jump to search
Line 13: Line 13:
  
 
Count the number of occurrences of each 4-mer in the provided data.
 
Count the number of occurrences of each 4-mer in the provided data.
 +
 +
Provided data are in Fastq format. They contain reads from some genomic sequencing.
 +
By a read we mean part of some DNA. In Fastq format each read is on 4 lines.
 +
The first line starts with @ and contains the read name.
 +
The second line contains the actual read (this is the important part for you).
 +
The third line contains + and the read name again.
 +
The fourth line contains a quality score for each base (you should ignore this).
 +
 +
By k-mer we mean any consecutive substring in a read.
 +
For example, in a read "ACGGCTA" the 4-mers are: "ACGG", "CGGC", "GGCT", "GCTA".
 +
  
 
===Task C===
 
===Task C===
  
−
Count the number of pairs of reads which overlap in exactly 30 bases (end of one read overlaps beginning of the second read). You can ignore reverse complement.
+
Count the number of pairs of reads that overlap in exactly 30 bases (the end of one read overlaps the beginning of the second read). For bioinformaticians: You can ignore the reverse complement.
  
 
Hints:  
 
Hints:  

Revision as of 12:44, 9 May 2024

See also the lecture

For both tasks, submit your source code and the result, when run on whole dataset (gs://fmph-mad-2024-public/). The code is expected to use the Apache beam framework and Dataflow presented in the lecture. Submit directory is /submit/cloud/

Task A for bonus points

Document (with screenshots) your login journey until step `gcloud projects list`.

Task B

Count the number of occurrences of each 4-mer in the provided data.

Provided data are in Fastq format. They contain reads from some genomic sequencing. By a read we mean part of some DNA. In Fastq format each read is on 4 lines. The first line starts with @ and contains the read name. The second line contains the actual read (this is the important part for you). The third line contains + and the read name again. The fourth line contains a quality score for each base (you should ignore this).

By k-mer we mean any consecutive substring in a read. For example, in a read "ACGGCTA" the 4-mers are: "ACGG", "CGGC", "GGCT", "GCTA".


Task C

Count the number of pairs of reads that overlap in exactly 30 bases (the end of one read overlaps the beginning of the second read). For bioinformaticians: You can ignore the reverse complement.

Hints:

  • Try counting pairs for each 30-mer first.
  • You can yield something structured from Map/ParDo operatation (e.g. tuple).
  • You can have another Map/ParDo after CombinePerKey.
  • Run code locally on small data to quickly iterate and test :)
Retrieved from "https://compbio.fmph.uniba.sk/vyuka/mad/index.php?title=HWcloud&oldid=2062"