Integration and light repository

[russelljjarvis]

Hey @JustasB, I need to integrate more closely with your work, as it puts the optimizer results in perspective.

I wonder if I can use the source code in the cellmodel repository, to regenerate your plot?
https://github.com/scrook/neuroml-db/blob/master/Import%20Scripts/model-importer/cellmodel.py

Also I am starting to write optimizer outputs to *csv, what should be included in csv files?
https://github.com/russelljjarvis/neuronunit/blob/opt_cluster/neuronunit/optimization/optimization_management.py#L244-L247

[russelljjarvis]

Hey @JustasB,

I have since done more work on this integration issue.

Here I evaluate the Druckmann tests, on an optimized model. In code I have sourced this model from the Open Science Framework, if it doesn't exist locally. So that part should be machine independent.

https://github.com/russelljjarvis/neuronunit/blob/opt_cluster/neuronunit/examples/PCA_JB.ipynb

I am having trouble getting NeuroML DB to work due to package management conflicts.

I wonder if there is a way to structure the data frame appropriately without building NeuroMLDB?
https://github.com/russelljjarvis/neuronunit/blob/opt_cluster/neuronunit/examples/Clustering_GetEphyzFromDBRJ.ipynb

[JustasB]

@russelljjarvis That looks great.

To skip the need to connect to the NMLDB, read this CSV file into a dataframe.

The columns are just the Druckmann test classes' names without "Test" (you have them listed in your "tests" variable).

You can remove all the rows if you want, or keep them if you want to compare to the other NML DB models.

The values in the dataframe are raw values from the Druckmann tests that have been NA filled and log-transformed. You can NA fill & transform the values from the druckmann tests in response to your model by performing steps described in 3 notebook cells of this file. The code of the first cell starts with: "df['AP1Amplitude'].fillna(0, inplace=True)"

Let me know if this helps.

[russelljjarvis]

This is good. I have made the data frame, still working on the rest.

[russelljjarvis]

I wonder if it's possible to have a branch were data and source code are separated?

I think a lot of your source code would be very useful, and I should integrate it, but it's a barrier that it is behind a 13GB download. An alternative approach I fork the repository on GH, and then manually delete data files cannot really work, as there are too many data files to manually delete.

[JustasB]

I can see the reasons why you would want to split the repo. I can also see reasons for keeping it as one. Spitting it would def be a non-trivial change.

I think you could avoid having to download the whole repo with sparse checkout. Take a look at git sparse checkout. This will let you check out just the sub-folders you want, without having to download the whole repo.

Let me know if you think this will work.

[russelljjarvis]

Thanks for the link to git sparse checkout. As a step to more closely integrating our pipelines I created this repository: https://github.com/russelljjarvis/NeuroML-DB-thinclient

I used the approach of piecemeal adding files to satisfy dependencies, instead of git spare commit, as I had already committed a lot of time to that approach.

I might need to do some manipulations, to register the repository as a fork of NeuroML-db, so I can easily integrate any code updates you make there.

[JustasB]

Ok sounds good