mgr/dashboard: monitoring: replace Grafana JSON with Grafonnet based code

[aaSharma14]

This PR intends to add grafonnet to generate grafana JSON files

Fixes: https://tracker.ceph.com/issues/45184
Signed-off-by: Aashish Sharma aasharma@redhat.com

Before:

After:

Show available Jenkins commands

• jenkins retest this please
• jenkins test classic perf
• jenkins test crimson perf
• jenkins test signed
• jenkins test make check
• jenkins test make check arm64
• jenkins test submodules
• jenkins test dashboard
• jenkins test api
• jenkins test docs
• jenkins render docs
• jenkins test ceph-volume all
• jenkins test ceph-volume tox

[p-se]

It already looks like we'll be saving a lot of boilerplate code with this approach. Some of it is due to the data being written differently. In jsonnet there are some attributes on a single line, in JSON they aren't. But, assuming that radosgw-sync-overview.jsonnet in this PR is complete, it reduces the LOC from 440 in radosgw-sync-voerview.json to 66 in jsonnet. Which I think is significant. And we might even be able to reduce that a little bit more by outsourcing grafana.annotation.datasource.

[p-se]

I've added a bunch of comments of things that I noticed and might have done differently. Those are not necessarily required to be changed, it's just my opinion.

But there a few things that we need to have a look at:

• the jsondiff tools seems to be a different one on my system (Ubuntu 20.04) and is not compatible with these changes
• I'm not sure if it is safe to remove "iteration", but it might be. I hope it is, as it doesn't seem to be possible to keep it, is that correct?
• a refresh time interval has likely been forgotten to be included
• the unit field and its value has been removed in two panels. It might be okay, but we'll need to check.

Please consider the rest not as requested changes but as benevolent suggestions.

[aaSharma14]

I've added a bunch of comments of things that I noticed and might have done differently. Those are not necessarily required to be changed, it's just my opinion.

But there a few things that we need to have a look at:

• the jsondiff tools seems to be a different one on my system (Ubuntu 20.04) and is not compatible with these changes
• I'm not sure if it is safe to remove "iteration", but it might be. I hope it is, as it doesn't seem to be possible to keep it, is that correct?
• a refresh time interval has likely been forgotten to be included
• the unit field and its value has been removed in two panels. It might be okay, but we'll need to check.

Please consider the rest not as requested changes but as benevolent suggestions.

Thanks @p-se . I have addressed most of your comments. But as of now I don't see any side-effects of removing iteration and unit properties and also I couldn't yet find a way to add them as well. I have attached the screenshots for the RGW SYNC OVERVIEW grafana panel, before and after this PR. I can do more research on the iteration and unit properties and will update the PR accordingly.

[p-se]

I've added a bunch of comments of things that I noticed and might have done differently. Those are not necessarily required to be changed, it's just my opinion.
But there a few things that we need to have a look at:

• the jsondiff tools seems to be a different one on my system (Ubuntu 20.04) and is not compatible with these changes
• I'm not sure if it is safe to remove "iteration", but it might be. I hope it is, as it doesn't seem to be possible to keep it, is that correct?
• a refresh time interval has likely been forgotten to be included
• the unit field and its value has been removed in two panels. It might be okay, but we'll need to check.

Please consider the rest not as requested changes but as benevolent suggestions.

Thanks @p-se . I have addressed most of your comments. But as of now I don't see any side-effects of removing iteration and unit properties and also I couldn't yet find a way to add them as well. I have attached the screenshots for the RGW SYNC OVERVIEW grafana panel, before and after this PR. I can do more research on the iteration and unit properties and will update the PR accordingly.

Yeah, might be the case that we don't need to worry about those.

Have you found anything out about the jsondiff issue? Is that due to different versions of jsondiff or do I happen to have a different application installed? I'm asking because I'm not even how to resolve that issue for me, but I also wonder if we need to add another dependency to install-dep.sh or not.

[aaSharma14]

I've added a bunch of comments of things that I noticed and might have done differently. Those are not necessarily required to be changed, it's just my opinion.
But there a few things that we need to have a look at:

• the jsondiff tools seems to be a different one on my system (Ubuntu 20.04) and is not compatible with these changes
• I'm not sure if it is safe to remove "iteration", but it might be. I hope it is, as it doesn't seem to be possible to keep it, is that correct?
• a refresh time interval has likely been forgotten to be included
• the unit field and its value has been removed in two panels. It might be okay, but we'll need to check.

Please consider the rest not as requested changes but as benevolent suggestions.

Thanks @p-se . I have addressed most of your comments. But as of now I don't see any side-effects of removing iteration and unit properties and also I couldn't yet find a way to add them as well. I have attached the screenshots for the RGW SYNC OVERVIEW grafana panel, before and after this PR. I can do more research on the iteration and unit properties and will update the PR accordingly.

Yeah, might be the case that we don't need to worry about those.

Have you found anything out about the jsondiff issue? Is that due to different versions of jsondiff or do I happen to have a different application installed? I'm asking because I'm not even how to resolve that issue for me, but I also wonder if we need to add another dependency to install-dep.sh or not.

I have updated the jsondiff command and removed -s and -i flags which didn't seem to be compatible for some distros. Are you still facing the same issue with jsondiff?

I've added a bunch of comments of things that I noticed and might have done differently. Those are not necessarily required to be changed, it's just my opinion.
But there a few things that we need to have a look at:

• the jsondiff tools seems to be a different one on my system (Ubuntu 20.04) and is not compatible with these changes
• I'm not sure if it is safe to remove "iteration", but it might be. I hope it is, as it doesn't seem to be possible to keep it, is that correct?
• a refresh time interval has likely been forgotten to be included
• the unit field and its value has been removed in two panels. It might be okay, but we'll need to check.

Please consider the rest not as requested changes but as benevolent suggestions.

Thanks @p-se . I have addressed most of your comments. But as of now I don't see any side-effects of removing iteration and unit properties and also I couldn't yet find a way to add them as well. I have attached the screenshots for the RGW SYNC OVERVIEW grafana panel, before and after this PR. I can do more research on the iteration and unit properties and will update the PR accordingly.

Yeah, might be the case that we don't need to worry about those.

Have you found anything out about the jsondiff issue? Is that due to different versions of jsondiff or do I happen to have a different application installed? I'm asking because I'm not even how to resolve that issue for me, but I also wonder if we need to add another dependency to install-dep.sh or not.

@p-se I have added the jsondiff dependency in requirements.txt and install-deps.sh file. Please let me know if it works fine for you now.

[ideepika]

#42765

[epuertat]

@epuertat @aaSharma14 this change broke the build

Processing files: ceph-grafana-dashboards-17.0.0-6884.g034de1f5.el8.noarch
error: Directory not found: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/etc/grafana/dashboards/ceph-dashboard
error: File not found: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/etc/grafana/dashboards/ceph-dashboard/*
Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.EAiLqt
+ umask 022
+ cd /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILD
+ cd ceph-17.0.0-6884-g034de1f5
+ DOCDIR=/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/usr/share/doc/ceph-grafana-dashboards
+ export LC_ALL=C
+ LC_ALL=C
+ export DOCDIR
+ /usr/bin/mkdir -p /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/usr/share/doc/ceph-grafana-dashboards
+ cp -pr monitoring/grafana/dashboards/README /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/usr/share/doc/ceph-grafana-dashboards
+ cp -pr monitoring/grafana/README.md /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/usr/share/doc/ceph-grafana-dashboards
+ exit 0


RPM build errors:
    Directory not found: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/etc/grafana/dashboards/ceph-dashboard
    File not found: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-6884-g034de1f5/rpm/el8/BUILDROOT/ceph-17.0.0-6884.g034de1f5.el8.x86_64/etc/grafana/dashboards/ceph-dashboard/*
+ rm -fr /tmp/install-deps.891037
Build step 'Execute shell' marked build as failure

see jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/48127//consoleFull

@tchaikov @ideepika as discussed in the chat, @aaSharma14 is reverting a small subset of changes (installing Grafana json files in the CMakeLists.txt) to see if that's the reason for the failure.

[smithfarm]

@epuertat @aaSharma14 This PR causes the build to reach out onto the Internet and attempt to download a git repo from github. In environments like the openSUSE Build Service where the build machine does not have a network connection, that completely breaks the build.

Since we already use the git submodule feature for including code from other git repos, I wanted to ask whether this external git project could be brought in that way, instead of trying (and failing) to download it at build time?

[smithfarm]

Also, whatever the fix is, it will need to be backported to pacific, because this PR has already been backported to (and merged into) pacific via #42812 :-/

[p-se]

Since the compiled Grafana dashboards are included in the repository anyway and the part of the build process that requires a working internet connection seems to be a test only (provided that is correct), would it be viable to disable WITH_GRAFANA by default? Changes to those files would still be tested by make check when added to the repository but wouldn't require a working internet connection to be build.

[smithfarm]

would it be viable to disable WITH_GRAFANA by default

Hm, wouldn't that break the build of the ceph-grafana-dashboards subpackage, though? See 5debf36b378b

[smithfarm]

In general, I believe any cmake code that introduces ExternalProject_Add should be guarded appropriately to avoid breaking the standard DEB, RPM builds on machines that aren't connected to the Internet.

There are two ways to download code from the Internet for use in Ceph:

1. git submodule
2. make-dist

The latter (make-dist) is a script in the top-level directory of the Ceph git repo. The purpose of this script is to generate the tarball from which the DEB,RPM packages are built.

[p-se]

would it be viable to disable WITH_GRAFANA by default

Hm, wouldn't that break the build of the ceph-grafana-dashboards subpackage, though? See 5debf36

Indeed, it does look so: https://github.com/ceph/ceph/blob/master/CMakeLists.txt#L685-L688

It is quite interesting that there's a switch that seemingly (unless I do not fully understand it) does not work properly when turned off.

[ 3098s] Processing files: ceph-grafana-dashboards-16.2.5-371.g601a4407a67.noarch
[ 3098s] error: Directory not found: /home/abuild/rpmbuild/BUILDROOT/ceph-16.2.5-371.g601a4407a67.x86_64/etc/grafana
[ 3098s] error: Directory not found: /home/abuild/rpmbuild/BUILDROOT/ceph-16.2.5-371.g601a4407a67.x86_64/etc/grafana/dashboards
[ 3098s] error: Directory not found: /home/abuild/rpmbuild/BUILDROOT/ceph-16.2.5-371.g601a4407a67.x86_64/etc/grafana/dashboards/ceph-dashboard
[ 3098s] error: File not found: /home/abuild/rpmbuild/BUILDROOT/ceph-16.2.5-371.g601a4407a67.x86_64/etc/grafana/dashboards/ceph-dashboard/*

Next idea would be to change WITH_GRAFANA (only the occurrences of it in this PR) to WITH_GRAFANA_TESTS or similar and only use that for the make check but, by default, leave it out for building.

The other options, I think, are the onces you've already mentioned, @smithfarm.

[smithfarm]

Next idea would be to change WITH_GRAFANA (only the occurrences of it in this PR) to
WITH_GRAFANA_TESTS or similar and only use that for the make check but, by default,
leave it out for building.

Yes. Or, just in general, guard any cmake ExternalProject_Add directives appropriately to avoid breaking the standard DEB, RPM builds on machines that aren't connected to the Internet.