Possibility of including text properties and JAR libraries in CodeQL databases

[luchua-bc]

Currently CodeQL databases include XML files and Java source code, which are great for analysis. However, many configuration items are in Java properties files in the format of "key=value" key/value pairs.

Also the CodeQL library contains the class JarFile but JAR libraries are not included in CodeQL databases. Typically the META-INF directory suffices for additional analysis therefore compiled Java classes don't need to be included.

Is it worth considering including text properties files and/or JAR libraries in CodeQL databases? Thanks.

[Marcono1234]

However, many configuration items are in Java properties files in the format of "key=value" key/value pairs.

There is the ConfigFiles module which includes Properties files. Though I have not used it myself yet, nor do I know which files the extractor is considering.

Typically the META-INF directory suffices for additional analysis

About which kind of JAR files are you talking, the ones on the classpath during building or the ones which are bundled (in some way) with the application? For JARs bundled with the application I think it highly depends on the used build tool, plugins and libraries how and where such JARs are included.

[luchua-bc]

Thanks @Marcono1234 for the response.

I will try the ConfigFiles module.

For JAR files, I'm talking about the ones on the classpath during building and it's easier for the CLI tool to create the database since it could only extract the META-INF directory to be included in the database.

I'm thinking about this because this will facilitate some queries e.g. queries to check obsolete APIs being used. Obsolete libraries with security issues fixed in newer versions are still widely used in many projects, especially with those having libraries in application's own classpath like lib (not using Maven). There is a MAINFEST.MF file under that directory, which contains contents like:

Manifest-Version: 1.0
Export-Package: org.apache.commons.logging;version="1.2",org.apache.commons.logging.impl;version="1.2"
Implementation-Title: Apache Commons Logging
Implementation-Vendor: The Apache Software Foundation
Implementation-Vendor-Id: org.apache
Specification-Title: Apache Commons Logging
Implementation-Version: 1.2
Build-Jdk: 1.7.0_55
Bundle-Version: 1.2.0

Queries can be developed to extract the bundle version of libraries bundled within the app with a specific build tool like Ant or Gradle then fed to a bot of GitHub. I think this could help a lot of projects.

[Marcono1234]

Were you able to find any .properties files with the ConfigFiles module?
It looks like by default CodeQL CLI might not index .properties files, maybe you can index them manually as described here: #4500 (comment)

[sbilello]

@luchua-bc #5321 (reply in thread)

[Marcono1234]

There is a MAINFEST.MF file under that directory, which contains contents like

@luchua-bc, it looks like reading MANIFEST entries (including OSGi ones) is already possible using the predicates of JarFile. However, I am not sure which JarFiles the database includes; I would assume only the ones which are somehow referenced by the project, but I have not tested it extensively. Additionally it appears there is no easy way to differentiate between JARs used as regular and as test dependency.

[luchua-bc]

Thanks @Marcono1234. I will give it a try then report back on how well it meets my needs.