retaining ScanResult for lifetime of application

[jechlin]

We've been using classgraph successfully for several years, thanks for all your work on it.

I'm not sure if it's OK to ask such a question here, apologies if it's not, I was just looking for some advice.

We scan the classpath once on startup, in order just to generate a list of classnames… these then indexed in such a way that users can find them by typing camel-cased partial names, eg FoBQ would find FooBarQuz, like an IDE.

In some cases we need to further filter the name matches to check if the class is-a Something… which we do by loading the class and using reflection. This is impractical if the user has only entered a single character as search term, or no characters at all.

Obviously it's much faster to use ClassGraph to answer this question, and many others that we need to answer, but we don't know what info we need from the ScanResult up front. So my question is, is it inadvisable to leave the ScanResult hanging around for a long time? It's a server application so that could be months or longer.

I'm aware that there is a memory consideration, and temporary files created etc, but would they cause a problem do you think? I haven't looked at the retained size of the ScanResult recently but as far as I remember, when I looked years ago, it was not significant (approx 60k classes). ClassGraph options are enableAnnotationInfo, and nothing else. We may use methodInfo in the future to avoid more reflection but not now.

Another alternative is to index the ScanResult using Lucene then close it as usual, which would be fine except that's much harder than it should be due to the environment we operate in.

I'd appreciate your thoughts if you have time.
cheers, jamie

[lukehutch]

@jechlin great, I'm glad you have been getting some good usage out of ClassGraph!

To answer your general question:

You can keep a ScanResult open as long as you want to, including for the whole lifetime of a program, as long as you don't mind keeping some resources open. This will typically include only file handles, but under some circumstances (or on some platforms), this may also include some memory-mapped files (memory mapping can be disabled). The reason for keeping files open is that the user might request the content of some resource from the ScanResult, and it would be inefficient to have to re-open jarfiles (potentially including jars-within-jars) to access each resource requested by the user.

Probably ClassGraph should have another flag to enable access to Resources (files) within jarfiles. That way, if this flag were not enabled, the ScanResult would be closed immediately after scanning is complete.

Another possibility would be to close the ScanResult after scanning is complete, and then reopen jarfiles for every resource that is requested (if any).

Right now I think that when the ScanResult is closed, there are a bunch of fields that are nullified, since a ScanResult should not be used after it is closed. But the classfile information should probably still be accessible even after a ScanResult is closed. The only problem is that you won't be able to load a class via ClassInfo#loadClass() after the ScanResult is closed.

In the next major version of ClassGraph, if I can find the time, I want to factor out all the optimized and multithreaded zipfile parsing and slicing code into a separate virtual filesystem library. Once that is done, a lot of the above dilemmas will become simpler to manage, since it will be easier to open or close resources as needed via the VFS API.

Basically this would mean that in ClassGraph-5.x, to scan the classpath, you'd first fetch the classpath, then open all elements on the classpath, producing a list of resources by hierarchically scanning paths only. Then in a separate step, all classfiles from this resource list could be scanned, producing a list of ClassInfo objects. These ClassInfo objects would not be tied to the lifecycle of the classpath resources.

Suggested solution for now:

Because of the implements Closeable nature of ScanResult as it currently exists, and because the above changes would be significant, I suggest continuing ahead with your current plans.

It sounds like you already call ClassInfo#getName() for each class discovered during the scan, then you index these classes, then you close the ScanResult. Rather than using reflection at runtime to read additional properties of classes (which is a bad idea, because class initialization is time consuming and can even have side effects!), you should simply read all the other properties you want for each class (e.g. ClassInfo#isAnnotation()) before closing the ScanResult. Then figure out what sort of data structure you need to store all this metadata in. I don't think it requires Lucene, you can just use a Map<String, ClassInfoLite> mapping a class name to a ClassInfoLite that just stores the values of the properties you care about, including maybe a list of method names.

[jechlin]

Thanks for your answer @lukehutch , I really admire your commitment to this project and its users.

I stupidly asked this question at a point I need to work on something else for a while so I don't have time to test some theories and get some metrics that I wanted, based on your answer.

Probably ClassGraph should have another flag to enable access to Resources (files) within jarfiles

That sounds good, I can't see that we would ever need to get resources, or call ClassInfo#loadClass(). Whilst we load classes I don't see that we'd do it via a ClassInfo.

It sounds like you already call ClassInfo#getName() for each class discovered during the scan, then you index these classes, then you close the ScanResult.

Yes, the FQCNs are stored in a Patricia trie so that you can quickly look up classes by components of the class simple name, as mentioned. So my understanding of your suggestion is that we store whatever info that we may need later alongside... we need to know whether it's an annotation yes, but also all superclasses/interfaces. My first thought is just to store a List of those names, but maybe it would be more efficient to keep them in a DAG or something. Anyway, that's not your problem!

I don't think it requires Lucene

Probably not, the advantage there is that is that it could be easily written to disk and not be stored in memory, and also the partial name matching might be faster or require less code to implement. My app is a plugin to a larger framework, the part that requires ClassGraph may not be used for days/weeks at a time, then it might be required for a few hours... so my thinking was not to use that memory when not needed. But we already store the trie in a SoftReference and recreate as necessary so maybe that's enough.

I'm a bit loathe to mark this as Answered right now in case I have any follow-up, if you wouldn't mind can I leave it open for 10 days or so, hopefully I'll be able to try a prototype.

The VFS stuff sounds interesting, looking forward to trying it at some point.

Thanks again

[lukehutch]

The trie model you have already built to be able to do partial name matches should be the only complex data structure you will need. Once a class name is fully resolved as a String, it can simply be used as a key to index a HashMap consisting of metadata objects about the class (I called the metadata class ClassInfoLite in my last message). Maybe I don't understand the nuances of what you're trying to do, but I think you're overcomplicating things by trying to figure out what indexing engine you should use to index the other class metadata.

You're welcome to keep this discussion open as long as you would like, I don't track todo items based on open conversations. However your questions caused me to think deeply about some of the fundamental usecases of ClassGraph, and I can see a much better way to handle lifecycle now. I'll open a bug to track some of these plans for ClassGraph-5.x.

[jechlin-adaptavist]

You're almost certainly right... I'll build the most simple thing possible that does what I want then add complexity if necessary depending on measurement of memory and performance etc.

(I called the metadata class ClassInfoLite in my last message

Yep, I understood, no worries. There is probably some cognitive bias involving a reluctance to accept the simplest and easiest solution is the right one, and I am over-complicating this.

I'll try it and update here.

Thanks!