{"id":2144,"date":"2013-03-25T07:45:14","date_gmt":"2013-03-25T07:45:14","guid":{"rendered":"http:\/\/www.odbms.org\/blog\/?p=2144"},"modified":"2013-03-25T07:45:14","modified_gmt":"2013-03-25T07:45:14","slug":"big-data-for-genomic-sequencing-interview-with-thibault-de-malliard","status":"publish","type":"post","link":"https:\/\/www.odbms.org\/blog\/2013\/03\/big-data-for-genomic-sequencing-interview-with-thibault-de-malliard\/","title":{"rendered":"Big Data for Genomic Sequencing. Interview with Thibault de Malliard."},"content":{"rendered":"

“Working with empirical genomic data and modern computational models, the laboratory addresses questions relevant to how genetics and the environment influence the frequency and severity of diseases in human populations”<\/em> –Thibault de Malliard.<\/strong><\/p>\n

Big Data for Genomic Sequencing<\/em>. On this subject, I have interviewed Thibault de Malliard,<\/strong> researcher at the University of Montreal\u2019s Philip Awadalla Laboratory<\/a>, who is working on bioinformatics solutions for next-generation genomic sequencing.<\/p>\n

RVZ<\/p>\n

Q1. What are the main research activities of the University of Montreal\u2019s Philip Awadalla Laboratory?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> The Philip Awadalla Laboratory is the Medical and Population Genomics Laboratory at the University of Montreal<\/a>. Working with empirical genomic data and modern computational models, the laboratory addresses questions relevant to how genetics and the environment influence the frequency and severity of diseases in human populations. Its research includes work relevant to all types of human diseases: genetic, immunological, infectious, chronic and cancer.
\nUsing genomic data from single-nucleotide polymorphisms <\/a>(SNP), next-generation re-sequencing<\/a>, and gene expression<\/a>, along with modern statistical tools, the lab is able to locate genome regions that are associated with disease pathology and virulence as well as study the mechanisms that cause the mutations.<\/p>\n

Q2. What is the lab’s medical and population genomics research database?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> The lab’s database is regrouping all the mutations<\/a> (SNPs) found by DNA genotyping<\/a>, DNA sequencing<\/a> and RNA sequencing<\/a> for each samples. There is also annotation data from public databases.<\/p>\n

Q3. Why is data management important for the genomic research lab?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> All the data we have is in text csv files<\/a>. This is what our software takes as input, and it will output other text csv files. So we use a lot of Bash<\/a> and Perl <\/a>to extract the information we need and to do some stats. As time goes, we multiply the number of files by sample, by experiment, and finally we get statistics based on the whole data that need recalculating each time we perform a new sequencing\/genotyping (mutation frequency, mutations per gene, etc).<\/p>\n

With this database, we are also preparing for the lab\u2019s future:
\n\u2022\tAs the amount of data increases, one day the memory will not fit an associative array.
\n\u2022\tLooking to a 200 GB file to find one specific mutation will not be a good option.
\n\u2022\tAdding new data to the current files will take more and more time\/space.
\n\u2022\tWe need to be able to select the data according to every parameter we have, i.e., grouping by type of mutation and\/or by chromosome, and\/or by sample information by gender, ethnicity, age, or pathology.
\n\u2022\tWe then need to export a file, or count \/ sum \/ average it.<\/p>\n

Q4. Could you give us a description of what kind of data is in the lab’s genomic research database storing and processing? And for what applications?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> We are storing single nucleotide polymorphisms<\/a> (SNPs), which are the most common form of genetic mutations among people, from sequencing and genotyping. When an SNP is found for a sample, we also look at what we have at the same position for the other samples:
\n\u2022\tThere is no SNP but data for the sample, so we know this sample does not have the SNP.
\nOR
\n\u2022\tThere is no data for the sample, so we cannot assess whether or not there is an SNP for this sample at this position.<\/p>\n

We gather between 1.8 and 2.5 million nucleotides (at least one sample has it) per sample, depending on the experiment technique. We store them in the database along with some information:
\n\u2022\thow damaged the SNP can be for the function of the gene
\n\u2022\tits frequency in different populations (African, European, French Canadian…).<\/p>\n

The database also contains information about each sample, such as gender, ethnicity, pathology. This will keep growing with our needs. So, basically, we have a sample table<\/em>, a mutations table<\/em> with their information, an experiment table<\/em> and a big table linking<\/em> the 3 previous tables with relations one to many.<\/p>\n

Here is a very slightly simplified example of a single record in our database:<\/p>\n

\n\r\n\r\n
Single record in our database<\/caption>\n
Type of data<\/th>\nData<\/th>\nTable<\/th>\n<\/strong><\/th>\n<\/tr>\n<\/thead>\n
SNP<\/td>\n\tT<\/td>\n\tBegin Mutation information table<\/td>\n<\/em><\/td>\n<\/tr>\n\r\n
Chromosome<\/td>\n\t1<\/td>\n<\/td>\n<\/tr>\n\r\n
Position<\/td>\n\t100099771<\/td>\n<\/td>\n<\/tr>\n\r\n
gene<\/td>\n\tNZT<\/td>\n<\/td>\n<\/tr>\n\r\n
Damaging for gene function?\t<\/td>\nsynonymous<\/td>\n<\/td>\n<\/tr>\n\r\n
Present in known database?<\/td>\n\tyes<\/td>\n End Mutation information table<\/td>\n<\/em><\/td>\n<\/tr>\n\r\n
Sequencing quality\t<\/td>\n 26<\/td>\n\tBegin Table linking other tables together<\/em><\/td>\n containing information about 1 mutation for 1 sample from 1 sequencing<\/td>\n<\/em><\/td>\n<\/tr>\n\r\n
Sequencing coverage<\/td>\n 15<\/td>\n<\/td>\n<\/tr>\n\r\n
Validated by another experiment?<\/td>\n\tno<\/td>\n End Table linking other tables<\/td>\n<\/em><\/td>\n<\/tr>\n\r\n
Sample<\/td>\n\t345<\/td>\n\tBegin Sample table<\/em><\/td>\n<\/td>\n<\/tr>\n\r\n
Research project<\/td>\n\tProject_1<\/td>\n<\/td>\n<\/tr>\n\r\n
Gender<\/td>\n\tMale<\/td>\n<\/td>\n<\/tr>\n\r\n
Ethnicity<\/td>\n\tFrench<\/td>\n<\/td>\n<\/tr>\n\r\n
family<\/td>\n\t10<\/td>\n End Sample table<\/td>\n<\/em><\/td>\n<\/tr>\n\r\n
Sequencing information<\/td>\n\tIllumina Hiseq 2500<\/td>\n\tBegin Sequencing table<\/td>\n<\/em><\/td>\n<\/tr>\n\r\n
Sequencing type (DNA RNA…)<\/td>\n\tRNAseq<\/td>\n<\/td>\n<\/tr>\n\r\n
Analysis pipeline info<\/td>\n\tNo PCR duplicates only Properly paired<\/td>\n End Sequencing table<\/em><\/td>\n<\/td>\n<\/tr>\n<\/tbody><\/table><\/div>
\nThe applications are multiple, but here are some which come to my mind:
\n\u2022\textract subset of data to use with our tools
\n\u2022\tdoing stats, counts
\n\u2022\tfind specific data
\n\u2022\tannotate our data with public databases<\/p>\n

Q5. Why did you decide to deploy TokuDB database storage engine to optimize the lab’s medical and population genomics research database?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> We knew that the data could not be managed with MySQL<\/a> and MyISAM<\/a>. One big issue is the insert rate, and TokuDB offered a solution up to 50 times faster. Furthermore, TokuDB<\/a> allows us to manipulate the structure of the database without blocking access to it. As a research team, we always have new information to add, which means column additions.<\/p>\n

Q6. Did you look\/consider other vendor alternatives? If yes, which ones?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> None. This is much too time consuming.<\/p>\n

Q7. What are you specifically using TokuDB for?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> We only store genetic data with information related to this genetic.<\/p>\n

Q8. How many databases do you use? What are the data requirements?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> I had planned to use three databases:
\n1. Database for RNA\/DNA sequencing and from DNA genotyping (described before);
\n2. Database for data from well-known reference databases (dbsnp<\/a>, 1000genome);
\n3. A last one to store analyzed data from database 1 and 2.<\/p>\n

The data stored is manly the nucleotide (a character: A, C, G, T) with integer information like quality, position, and Boolean flags. I avoid using any string to keep the table as small as possible.<\/p>\n

Q9. Especially, what are the requirements for data ingest of records and retrieve of data?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> As a research team, we do not have high requirements like real-time insertion from logs. But I would say, at most, the import should take less than a night. The update of the database 1 is critical with the addition of a new sequencing or genotyping experiment: a batch of 50M records (can be more than 3 times higher!) has to be inserted. This has been happening monthly, but it should increase this year.<\/p>\n

We have a huge amount of data, and we need to get query results as fast as possible, We have been used to one or two days (a weekend) of query time \u2013 having 10 seconds is much more preferable!<\/p>\n

Q10. Could you give some examples of what are the typical research requests that need data ingestion and retrieval <\/strong><\/p>\n

Thibault de Malliard:<\/strong> We have a table with all the SNPs for 1000 samples. This is currently a 100GB table.
\nA typical query could be to get the sample that got a mutation different from the 999 others. We also have some samples that are families: a child with its parents. We want to find the SNPs present in this child, but not present in the other family member.
\nWe may want to find mutations common to one group of sample given the gender, a disease state, ethnicity.<\/p>\n

Q11. What kind of scalability problems did you have?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> The problem is managing this huge amount of data. The number of connections should be very low. Most of the time, there is only one user. So I had to choose the data types carefully and the relationships between the tables. Lately, I ran into a very slow join with a range so I decided to split the position based tables by chromosome. Now there are 26 tables and some procedures to launch queries through the chromosomes. The gain of time is not quantifiable.<\/p>\n

Q12. Do you have any benchmarking measures to sustain the claim that Tokutek\u2019s TokuDB has improved scalability of your system?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> I populated the database with two billion records in the main table and then did queries. While I did not see improvements with my particular workload for queries, I did see significant insertion performance gains. When I tried to add an extra 1M record (Load data infile), it took 51 minutes for MyISAM to load the data, but less than one minute with TokuDB. I extend this amount of data to an RNA sequencing experiment: it should take 2.5 days for MyISAM but one hour for TokuDB.<\/p>\n

Q13. What are the lessons learned so far in using TokuDB database storage engine in your application domain?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> We are still developing it and adding data. But inserting data into the two main tables (0.9G records, 2.3G records) was done in a fairly good time, less than one day. Adding columns to fulfill the needs of the team is also a very easy feature: it takes one second to create the column. Updating it is another story, but the table is still accessible during this process.
\nAnother great feature, like the one I use with each query, is to be able to follow the state of the query.
\nYou can follow in the process list the number of rows which were queried. So if you have a good estimation of the number of records expected, you know exactly the time of the query. I cannot count the number of process I killed because the query time expected was not acceptable.<\/p>\n

Qx. Anything you wish to add?<\/strong><\/p>\n

Thibault de Malliard:<\/strong> The sequencing\/genotyping technologies evolve very fast. Evolving means more data from the machines. I expect our data to grow at least three times each year. We are glad to have TokuDB in place to handle the challenge.<\/p>\n

————-
\nSince 2010, Thibault de Malliard<\/strong> has worked in the University of Montreal\u2019s Philip Awadalla Laboratory<\/a> where he provides bioinformatics support to the lab crew and develops bioinformatics solutions for next-generation genomic sequencing. Previously, he worked for the French National Institute for Agricultural Research (INRA) with the MIG laboratory (Mathematics, Informatics and Genomics) where, as part of the European Nanomubiop project, he was tasked with developing software to produce probes for a HPV chip. He holds a masters degree in bioinformatics (France).<\/p>\n

Related Posts<\/strong><\/p>\n

– Big Data: Improving Hadoop for Petascale Processing at Quantcast. March 13, 2013<\/a><\/strong><\/p>\n

– On Big Data Analytics \u2013Interview with David Smith. February 27, 2013<\/a> <\/strong><\/p>\n

– Big Data Analytics at Netflix. Interview with Christos Kalantzis and Jason Brown. February 18, 2013<\/a><\/strong><\/p>\n

– MySQL-State of the Union. Interview with Tomas Ulin. February 11, 2013<\/a><\/strong><\/p>\n

– Scaling MySQL and MariaDB to TBs: Interview with Mart\u00edn Farach-Colton. October 8, 2012<\/a><\/strong><\/p>\n

Related Resources<\/strong><\/p>\n

– Big Data for Good.<\/strong> by Roger Barca, Laura Haas, Alon Halevy, Paul Miller, Roberto V. Zicari<\/em>. June 5, 2012:
\nA distinguished panel of experts discuss how Big Data can be used to create Social Capital. <\/em>
\nBlog Panel | Intermediate | English | DOWNLOAD (PDF)<\/strong><\/a>| June 2012|<\/p>\n

– ODBMS.org resources on Relational Databases, NewSQL, XML Databases, RDF Data Stores<\/a>.
\n<\/strong><\/p>\n

Follow ODBMS.org on Twitter: @odbmsorg<\/a><\/strong>
\n##<\/p>\n","protected":false},"excerpt":{"rendered":"

“Working with empirical genomic data and modern computational models, the laboratory addresses questions relevant to how genetics and the environment influence the frequency and severity of diseases in human populations” –Thibault de Malliard. Big Data for Genomic Sequencing. On this subject, I have interviewed Thibault de Malliard, researcher at the University of Montreal\u2019s Philip Awadalla […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[35,66,154,214,395,414,572,579,593],"_links":{"self":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/2144"}],"collection":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/comments?post=2144"}],"version-history":[{"count":0,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/posts\/2144\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/media?parent=2144"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/categories?post=2144"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.odbms.org\/blog\/wp-json\/wp\/v2\/tags?post=2144"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}