Accuracy
JCVI will conduct a sampling of Pathema genes to test for accuracy. This will be done by searching the BRC genes against 100 randomly chosen TIGRFAM equivalog HMM to provide the set of genes that should have the same function. A human curator will inspect a small set of results and make a subjective assessment of the correctness of the functional name assignment. The statistic will be reported as a percentage of "correct" functional name annotations.
Completeness
JCVI will perform an exhaustive search of Pathema genes against TIGRFAM equivalog HMMs to identify sets of genes that have the same function. The TIGRFAMs members that contain functional names, genetic names, GO ids, and EC#'s will serve as the source of datatypes that are expected to appear in the BRC genes. Completeness will be measured by counting the number of functional names, genetic names, GO ids, and EC#'s that have been assigned. Note that this metric does not attempt to assess the correctness of the annotations, only that an annotation is provided. The completeness statistic will be reported as a percentage of possible annotations, based on the metric: (number of actual annotations) / (number of expected annotations).
Consistency
JCVI will perform an exhaustive search of Pathema genes against TIGRFAM equivalog HMMs to identify sets of genes that have the same function. Each set of genes will be expected to have consistent functional names. Consistency will be measured for functional name assignments within Pathema. The functional names from Pathema will only be compared to each other, the names will not be compared against the TIGRFam name. The consistency statistic will be reported as the likelihood of any 2 genes having the same annotated text string.
