1-DAV-202 Data Management 2023/24
Previously 2-INF-185 Data Source Integration

Materials · Introduction · Rules · Contact
· Grades from marked homeworks are on the server in file /grades/userid.txt
· Please submit project proposals until Friday April 12. Topics from potential bachelor topic supervisors can be found in /tasks/temy.txt (in Slovak).
· Due to Easter holidays, Web and Bioinf1 homeworks are due on April 4, 9:00am.


HWcloud

From MAD
Jump to navigation Jump to search

See also the lecture

For both tasks, submit your source code and the result, when run on whole dataset (gs://fmph-mad-2023-public/). The code is expected to use the Apache beam framework and Dataflow presented in the lecture. Submit directory is /submit/cloud/

Task A for bonus points

Document (with screenshots) your login journey until step `gcloud projects list`.

Task B

Count the number of occurrences of each 4-mer in the provided data.

Task C

Count the number of pairs of reads which overlap in exactly 30 bases (end of one read overlaps beginning of the second read). You can ignore reverse complement.

Hints:

  • Try counting pairs for each 30-mer first.
  • You can yield something structured from Map/ParDo operatation (e.g. tuple).
  • You can have another Map/ParDo after CombinePerKey.
  • Run code locally on small data to quickly iterate and test :)