- ID: 2018-tuos-adb-ug-10a
- Student: Zer J Eng zjeng1@sheffield.ac.uk
- Cohorts: CS/Math/SE (other after approval by supervisor)
- Keywords: source code control, software security, vulnerability, CVSS, CWE, patch analysis, source code analysis, Open Source, FOSS, repository mining, github
Not all Free/Libre and Open Source (FLOSS) Projects publish fixed software vulnerabilities in an easy to consume manner (e.g., as CVEs). Moreover, even if they do, it is often not easy to identify the actual code commit fixing a security vulnerability.
As for users of FLOSS components, it is important to understand which vulnerabilities are known and when/how they were fixed, it is important to have an in-depth understanding of vulnerabilities in FLOSS components (of course, also for an attacker/hacker, this information is of value).
In this project, a repository mining tool should be developed that is, e.g., able to detect
- silent patches/fixed, i.e., commits that fix security vulnerabilities that are not yet known
- commits that fix known vulnerabilities
The project can be extended to learn/derive from the identified commits configuration for security testing tools that help to decide if an application using the component is affected by the vulnerability or not.
- Good programming skills
- Good understating of source code control systems (e.g. git)
- An interest in application security
- OWASP Top 10: https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
- https://github.com/TQRG/secbench-mining-tool
Zer Jun is very much engaged with the project and works very goal-oriented. His time management is excellent.
A significant amount of work, in form of a high-quality literature survey, has been completed. As a next step, it is important that the actual implementation is started.
The analysis shows a deep understanding of the problem at hand, including a clear definition of the scope of the project.
The time plan is nicely broken down into several phases and covers all important aspects.
The survey and background sections is clearly structured and covers all important works in the necessary details.
The presentation is of high quality (both, in terms of actual content and in terms of language/layout).
Overall, an excellent analysis and survey stage report.
A thorough literature review has been undertaken and a clear plan has been put in place together with initial basic implementations.
The aims of the project are described accurately and a clearly defined plan is in place. The project though seems very ambitious given the amount of time available. Several challenges (risks) have been identified. I am not sure that plans have been put in place to face such contingencies if they occur.
As said above, the time plan is very ambitious.
The literature survey is very broad and to the point. It is well written with very rare English language mistakes. Usually in the use of singular/plural.
The presentation is of high quality. The only flaw is that acronyms should be explained the first time they are introduced. The abstract was incomprehensible to me the first time I read it.
The student has clearly worked hard towards the production of this report. As a non-expert it is unclear how likely the risks are and whether they may compromise seriously the outcomes of the project.
Zer Jun's engagement with the project, since start of semester 1, was outstanding. The overall results of this project surpass the expectations both in quality and quantity.
The developed tool is of very high quality and will be used in future research. In particular, the tool is extensible, it has a clear and well-defined (command-line) interface, and the code is well structured, allowing for future extensions.
A clear and well structured process has been followed. The work was nicely distributed across the whole academic year.
The evaluation, covering both unit tests and the analysis of large real world repositories, has been done very thoroughly. The results are impressive and provide interesting insights for future research.
An outstanding amount of work has been implemented.
Both the poster presentation and the report are well structured. In particular, the poster presentation also covered edge cases that could not be found using the developed approach.
The project is an extremely well done project and Zer Jun showed an extraordinary understanding of the problem and was always eager to improve the approach further.
The promised repository mining tool has been developed as promised in the interim report. A true positive rate of 44% is reported.Also false positive rates were promised, bit I could not spot any data about this.
The introduction of the report has been updated to reflect changes since the interim report.
On one hand no justification is provided on whether the size of the used sample is large enough for the rate to be considered accurate. On the other hand no comment is made related to how good or how bad a 44% true positive rate is.
A remarkable amount of work has been completed.
The presentation quality has improved over the already very good quality of the interim report. Previous feedback has been implemented which is a measure to see.
A well exectuted project.