An assembly of reads, contigs and scaffolds

A blog on all things newbler and beyond

Running newbler: de novo transcriptome assembly I

Posted by lexnederbragt on August 31, 2010

RNA (source:

Since version 2.3, newbler has a -cdna option for de novo transcriptome assembly. In this post, I’ll explain the principles and setting up the transcriptome assembly. The next post will discuss the output of a transcriptome assembly.

1) Principles of transcriptome assembly

As with other assembly projects, the first steps for transcriptome assembly are identical, and newbler builds a contig graph, see this post. Ideally, the reads coming from the transcript of a certain gene should result in a single contig. However, because of splice-variants (and other sequence particularities), there may be several contigs for each transcript, which themselves form a small contig graph. Splice-variants will result in reads that , relative to other reads have an insert (representing an additional exon in the transcript), thereby breaking the contig graph, see the figure.

Relationship between exons, contigs and isotigs

So, for transcriptome projects, there will be numerous subgraphs each potentially representing one gene. Each of these subgraphs are called an isogroup. Next, newbler will traverse the contigs in the subgraphs of each isogroup to generate transcript variants, which are called isotigs, again, see the figure. There are certain rules for this traversing step, for example, for starting the path and for ending it. Another rule, for complex graphs, is a cutoff such that no more than a maximum number of isotigs are generated per isogroup (by default set to 100 isotigs). If fully traversing the graph will result in more isotigs than this maximum, the contigs of this isogroup are reported in the output instead of the isotigs. Read the rest of this entry »

Posted in How it works, Using newbler | Tagged: , , , , , , | 4 Comments »

Running newbler: more de novo assembly parameters (and a hidden one)

Posted by lexnederbragt on July 16, 2010

Trimming (reads) by running newbler

There is a long list of options/flags/parameters for a newbler assembly, some of which have been treated in the previous post. In this post I will describe some more parameters. At the end, as a bonus, I will share a parameter that is not mentioned in the current documentation…

-ss -sl -sc -ais -ads
These parameters control read overlap detection (there are two more, -mi and -ml, which I described in the previous post). More on seeds and overlap detection is described in the post explaining how newbler works. I never change these parameters as I assume 454 has done a good job optimizing them. But I would love to hear from people that have tried the effect of adjusting these parameters… Read the rest of this entry »

Posted in Using newbler | 8 Comments »

Running newbler: de novo assembly

Posted by lexnederbragt on June 10, 2010

This post is about how to start up newbler for de novo assembly projects. I will describe setting up newbler using the command line. Most of the options I will mention are also available through the GUI version, but I will not describe how to use them here.

For a description of the progress that newbler reports during assembly, please check this post. For a description of the different output files, these are described in a series of previous posts.

1) default newbler on one or more files

runAssembly /data/sff/EYV886410.sff

This is the most simple way of running newbler: just provide it with one sff file. It will generate a folder called along the lines of P_yyyy_mm_dd_hh_min_sec_runAssembly and put all output in there. If you want to have control over the name of this folder, use

runAssembly -o project1 /data/sff/EYV886410.sff

-o describes the name of the folder newbler will provide all output in, in this case ‘project1’

Read the rest of this entry »

Posted in Using newbler | Tagged: , , , , , | 65 Comments »

Newbler output VI: the ‘status’ files (454TrimStatus.txt, 454ReadStatus.txt, 454PairStatus.txt) and the 454AlignmentInfo.tsv file

Posted by lexnederbragt on May 20, 2010

The files that are the topic of this post are all tables, i.e. tab separated text files. The ‘status’ files describe what happened with all the reads and the paired end halves, while the AlignmentInfo file summarizes the contig alignments.

The fact that these files are tabular makes for easy parsing using by perl/python or, my favorite, awk.

1) 454TrimStatus.txt

Accno   Trimpoints Used Used Trimmed Length     Orig Trimpoints Orig Trimmed Length     Raw Length
ERGMJHS01CYVHW  5-78    74      5-98    94      100
ERGMJHS01D6IHL  5-116   112     5-116   112     161
ERGMJHS01DYTX5  5-127   123     5-127   123     173
ERGMJHS01DYDH0  5-78    74      5-78    74      124
ERGMJHS01ECEGM  5-256   252     5-256   252     271
ERGMJHS01CRQ8D  5-272   268     5-272   268     273
ERGMJHS01ECMVT  5-260   256     5-260   256     270
ERGMJHS01EZ7VU  5-41    37      5-61    57      62
ERGMJHS01ERDXB  5-207   203     5-207   203     252

This file describes what (trimmed) part of the read was considered for alignment. The columns describe:

Posted in Newbler output | Tagged: , , , , , , | 14 Comments »

Newbler output III: the 454ContigGraph.txt file

Posted by lexnederbragt on April 13, 2010

The single file I’ll discuss today has in fact almost the entire assembly in it, besides the actual sequences (although even some of these are also included, see below). As explained in my first post, newbler (as many other assembly programs) builds a contig graph. Contigs are the nodes, and reads spanning between them (starting in one contig and continuing or ending in another) indicate the edges. All the information on this graph, except the actual read alignments and consensus contigs, is in the 454ContigGraph.txt file.

The file is divided into several sections, for each one the lines start with a capital letter, except for the first section.

Putting together an assembly...

Section 1) Contig statistics

Read the rest of this entry »

Posted in Newbler output | Tagged: , , , , , , | 33 Comments »

Newbler output II: contigs and scaffolds sequence files, and the 454Scaffolds.txt file

Posted by lexnederbragt on March 22, 2010

The files most people are after when they do an assembly must be these: the actual contig and scaffold sequences. The contigs are in the files 454AllContigs and 454LargeContigs. ‘All’ indicates by default contigs of at least 100 bp, while ‘Large’ contigs are at least 500 bp. These lower limits can be set during assembly.
The ‘fna’ files contain the sequences (bases) in fasta format (I actually do not why this extension was chosen over ‘fasta’ or ‘fa’ which are most often used). The ‘qual’ files contain phred-like quality scores (see previous post). The contigs are in the same order between fna and qual files, and the quality scores are in the same order as the bases:

Read the rest of this entry »

Posted in Newbler output | Tagged: , , , , , , , | 19 Comments »

Newbler output I: the 454NewblerMetrics.txt file

Posted by lexnederbragt on March 11, 2010

With this post, I’ll start going through the output files newbler generates. Some of these will be described in detail as they contain a lot of important information.

For today’s post, we’ll start with the 454NewblerMetrics.txt file. This file contains a lot of details on the reads used during the assembly, as well as the resulting contigs and, in the case of paired end reads, scaffolds.

The file starts with some metadata, such as the date of the assembly, where is is located, and what version of newbler was used. For this post, I used a file of as assembly generated with version 2.3 and both shotgun and paired end read files. Note that the output will be slightly different for a mapping project (to be described in a later post) than for an assembly project.

Section 1: runData

path = "/your/path/yourfile1.sff";

numberOfReads = ######, ######;
numberOfBases = #########, #########;

For each input file, the numbers mentioned are reads and bases in the file, reads and bases after trimming.

Read the rest of this entry »

Posted in Newbler output | Tagged: , , , , | 36 Comments »

How newbler works

Posted by lexnederbragt on February 9, 2010

I thought to start by explaining briefly how newbler works. I’ll do this by following the output newbler generates during the assembly process. This information is displayed during assembly, and can also be found in the 454NewblerProgress.txt file. It is a good thing anyways to have a look at this file, as it sometimes displays certain warnings (see below).

This example assembly is based on a read dataset consisting of both shotgun reads, and paired end reads (for more on 454 paired end reads, have a look here).

The first thing you’ll see is a message stating that the assembly computation started, and which version of newbler you used.

Then, you’ll see messages for each input file saying Indexing XXXXXXX.sff…, and a counter. During indexing, newbler scans the input file, performs some checks and trims the reads (sometimes more than the base-calling software already did). One of the checks is for possible 3′ and 5′ primers: if a certain percentage of reads contains the same sequence on either the 3′ or 5′ end, this is mentioned. I’ve had some surprises here, such as finding out that reads I got from another group contained an adaptor sequence, which caused problems during the assembly. More on primer removal later…

If an input sff file contains paired end reads, this will be mentioned, as well as the number of reads that contained the paired end linker sequence, for example:

224024 reads, 58599257 bases, 112080 paired reads.


Setting up long overlap detection…
XXXXX reads to align
Building a tree for YYYYYY seeds…
Computing long overlap alignments…

The first phase of assembly is finding overlap between reads. Newbler splits this phase into one for long reads (this goes very fast) and shorter reads (can take quite some time). As aligning all reads against each other would take too long time, newbler (and many other programs) actually make seeds, 16-mers of each read, where each seed starts 12 bases upstream of the previous one. These seed length and step sizes can be changed if you want (I’ve never tried this, though). When two different reads have identical seeds the program tries to extend the overlap between the reads until the minimum overlap (default 40 bp) with the minimum alignment percentage default 90%) has been reached. These settings can also be changed and influence the alignment stringency, this I will come back to in a later post. Read the rest of this entry »

Posted in How it works | Tagged: , , , , , , | 57 Comments »


Posted by lexnederbragt on February 9, 2010

With this blog I intend to share some of my experiences with the newbler assembly program from 454, also known as gsAssembler, or gsMapper. It is the software suite developed by 454 Life Sciences to be used with the sequence data coming from the GS FLX sequencing instrument.

Reads: those are the fragments, small bits and pieces that I write about in the blog.

Contigs: together, these fragments make a larger pieces of information on the subject.

Scaffolds: by building bridges between the subjects I hope to reach a complete overview of the newbler program. Granted, there will be gaps, as there are also gaps in my knowledge, but in the end, the information should be useful guide to newbler.

I learned a lot about newbler when working with data from bacterial genome assemblies, and the cod genome project, for which I am one of the bioinformaticists. I am also connected to the 454 node of the Norwegian High-Throughput Sequencing Centre (NSC), where many of our users are relying on newbler for their projects.

The first post will describe step-by-step how newbler generates contigs and scaffold from reads. So, let’s start the assembly!

Posted in Miscellaneous | Tagged: | Leave a Comment »