Skip to content

Commit

Permalink
[wip] Highlight code with rewrite-clj, not highlight.js
Browse files Browse the repository at this point in the history
The highlight.js integration has been costly to maintain, both for
day8 and for our users. See:

#376

Now, we use rewrite-clj and reagent for minimalistic
highlighting. This is similar to borkdude's approach:

https://blog.michielborkent.nl/writing-clojure-highlighter.html

rewrite-clj labels forms by their type and purpose. It provides a
zipper api for expressive traversal, including line & char
numbers.

Now, we simply transform rewrite-clj's node tree into hiccup. We do
two post-order traversals, which isn't ideal, but it works. We could
consolidate the traversal with some careful refactoring.

TODO: ::highlighted-form-bounds is mostly regex math which
reverse-engineers the output of re-frame-debux. Now, ::highlighted?
adds another layer of reverse-engineering. It seems like we could
delete all this calculation, and delete zprint, if re-frame-debux
could simply provide the node tree.

re-frame-debux could also provide some analysis data from clj-kondo,
making it possible to color locals differently from globals, for
instance.
  • Loading branch information
kimo-k committed Jul 20, 2023
1 parent 4ab5910 commit 029b996
Show file tree
Hide file tree
Showing 7 changed files with 152 additions and 36 deletions.
7 changes: 7 additions & 0 deletions CHANGELOG.md
Expand Up @@ -3,6 +3,13 @@ All notable changes to this project will be documented in this file. This change

## Unreleased

#### Added

- Added rewrite-clj 1.1.47 dependency

#### Changed

- Removed highlight.js dependency
- Upgrade clojure & clojurescript to 1.11
- Upgrade zprint to 1.2.7
- Upgrade malli to 0.11.0
Expand Down
2 changes: 1 addition & 1 deletion examples/todomvc/shadow-cljs.edn
Expand Up @@ -5,7 +5,7 @@
[day8.re-frame/tracing "0.6.2"]
[com.yahoo.platform.yui/yuicompressor "2.4.8"
:exclusions [rhino/js]]
[superstructor/re-highlight "2.0.2"]
[rewrite-clj/rewrite-clj "1.1.47"]
[secretary "1.2.3"]
[binaryage/devtools "1.0.6"]
[metosin/malli "0.11.0"]]
Expand Down
8 changes: 2 additions & 6 deletions project.clj
Expand Up @@ -11,12 +11,8 @@
[com.yahoo.platform.yui/yuicompressor "2.4.8"
:exclusions [rhino/js]]
[zprint "1.2.7"]
[superstructor/re-highlight "2.0.2"]
;; re-highlight only has a transitive dependency on highlight.js for
;; shadow-cljs builds, so we need to declare a dependency on cljsjs/highlight
;; for 10x to support other build systems.
[cljsjs/highlight "10.3.1-0"]
[org.clojure/tools.logging "1.2.4"]]
[org.clojure/tools.logging "1.2.4"]
[rewrite-clj/rewrite-clj "1.1.47"]]

:plugins [[day8/lein-git-inject "0.0.15"]
[lein-less "RELEASE"]]
Expand Down
20 changes: 18 additions & 2 deletions src/day8/re_frame_10x/panels/event/subs.cljs
Expand Up @@ -152,12 +152,28 @@
(rf/reg-sub
::highlighted-form-bounds
:<- [::highlighted-form]
:<- [::form-for-epoch]
:<- [::zprint-form-for-epoch]
(fn [[highlighted-form form] _]
(find-bounds (str form)
(find-bounds form
(:form highlighted-form)
(:num-seen highlighted-form))))

(rf/reg-sub
::highlighted?
:<- [::zprint-form-for-epoch]
:<- [::highlighted-form-bounds]
(fn [[zp [left-bound _]] [_ [line char]]]
(when (pos? left-bound)
(let [line (dec line)
char (dec char)
line-counts (map (comp inc count)
(clojure.string/split-lines zp))]
(->> line-counts
(take line)
(apply +)
(+ char)
(= left-bound))))))

(rf/reg-sub
::show-all-code?
:<- [::root]
Expand Down
30 changes: 3 additions & 27 deletions src/day8/re_frame_10x/panels/event/views.cljs
Expand Up @@ -2,7 +2,6 @@
(:require-macros
[day8.re-frame-10x.components.re-com :refer [handler-fn]])
(:require
[re-highlight.core :as re-highlight]
[day8.re-frame-10x.inlined-deps.garden.v1v3v10.garden.units :refer [px ms]]
[day8.re-frame-10x.inlined-deps.spade.git-sha-93ef290.core :refer [defclass]]
[day8.re-frame-10x.inlined-deps.reagent.v1v0v0.reagent.core :as r]
Expand All @@ -21,7 +20,7 @@
[day8.re-frame-10x.panels.settings.subs :as settings.subs]
[day8.re-frame-10x.fx.clipboard :as clipboard]
[day8.re-frame-10x.tools.pretty-print-condensed :as pp]
[day8.re-frame-10x.tools.datafy :as tools.datafy]))
[day8.re-frame-10x.tools.highlight-hiccup :refer [str->hiccup]]))

;; Terminology:
;; Form: a single Clojure form (may have nested children)
Expand Down Expand Up @@ -79,34 +78,11 @@
before (subs form-str 0 start-idx)
highlight (subs form-str start-idx end-idx)
after (subs form-str end-idx)]
; DC: We get lots of React errors if we don't force a creation of a new element when the highlight changes. Not really sure why...
;; Possibly relevant? https://stackoverflow.com/questions/21926083/failed-to-execute-removechild-on-node
^{:key (gensym)}
;; At some point around March 2021 Highlight.js changed their API significantly (e.g. highlightBlock -> highlightElement).
;; re-highlight, the wrapper around Highlight.js, depends on a modern version of Highlight.js with highlightElement.
;; Prior to the below check being added, a transitive dependency or direct project dependency that overrides
;; the version of Highlight.js with an older release than that reqeusted by re-highlight would cause 10x to crash.
;; Therefore, we added this check to ensure that prior to attempting to render source code with Highlight.js we check
;; that a compatible dependency has been loaded.
(if (re-highlight/hljs-compatible?)
[rc/box
[rc/box
:class (code-style ambiance syntax-color-scheme show-all-code?)
:attr {:on-double-click (handler-fn (rf/dispatch [::event.events/set-show-all-code? (not show-all-code?)]))}
:child (if (some? highlighted-form)
[#_re-highlight/highlight :span {:language "clojure"}
(list ^{:key "before"} before
^{:key "hl"} [:span.code-listing--highlighted highlight]
^{:key "after"} after)]
[#_re-highlight/highlight :span {:language "clojure"}
form-str])]
[rc/v-box
:class (hljs-error-style ambiance syntax-color-scheme)
:children [[rc/p
"re-frame-10x found a version mismatch between the Highlight.js loaded and the one that it expects to use."]
[rc/p
"As a result, it can't display the source code for this function."]
[rc/p
"To fix this, please examine this application's dependency tree to see how an old version of Highlight.js is being pulled in (probably transitively) and perhaps then use an appropriate exclusion for that dependency."]]])))})))
:child [str->hiccup form-str]]))})))

(defclass clipboard-notification-style
[_]
Expand Down
18 changes: 18 additions & 0 deletions src/day8/re_frame_10x/styles.cljs
Expand Up @@ -553,3 +553,21 @@
[:5% {:margin-left "0px"
:opacity "1"}]
[:90% {:opacity "1"}])]))

(defclass clj-symbol [] {:color nord10})

(defclass clj-core-macro [] {:color nord11})

(defclass clj-keyword [] {:color nord15})

(defclass clj-number [] {:color nord12})

(defclass clj-string [] {:color nord14})

(defclass clj-nil [] {:color nord3})

(defclass clj-boolean [] {:color nord3})

(defclass clj-highlighted [] {:background-color nord13})

(defclass clj-seq [] {})
103 changes: 103 additions & 0 deletions src/day8/re_frame_10x/tools/highlight_hiccup.cljs
@@ -0,0 +1,103 @@
;; TODO: make this a standalone library

(ns day8.re-frame-10x.tools.highlight-hiccup
(:require [clojure.walk :as walk]
[rewrite-clj.zip :as rz]
[rewrite-clj.node.token :refer [SymbolNode TokenNode]]
[rewrite-clj.node.whitespace :refer [WhitespaceNode NewlineNode CommaNode]]
[rewrite-clj.node.keyword :refer [KeywordNode]]
[rewrite-clj.node.stringz :refer [StringNode]]
[rewrite-clj.node.seq :refer [SeqNode]]
[day8.re-frame-10x.styles :as styles]
[day8.re-frame-10x.inlined-deps.re-frame.v1v1v2.re-frame.core :as rf]
[day8.re-frame-10x.panels.event.subs :as event.subs]))



(def clj-core-macros #{'and 'binding 'case 'catch 'comment 'cond 'cond-> 'cond->> 'condp 'def
'defmacro 'defn 'defn- 'defmulti 'defmethod 'defonce 'defprotocol 'deftype
'do 'dotimes 'doseq 'dosync 'fn 'for 'future 'if 'if-let 'if-not 'import 'let
'letfn 'locking 'loop 'ns 'or 'proxy 'quote 'recur 'set! 'struct-map 'sync 'throw
'try 'when 'when-first 'when-let 'when-not 'when-some 'while})

(defn selected-style [{:keys [position]}]
(when @(rf/subscribe [::event.subs/highlighted? position])
(styles/clj-highlighted)))

(defmulti form type)

(defmethod form :default [node] [:span.clj-unknown {:data-node (type node)} (str (type node))])

(defmulti tf2 (comp type :value))

(defmethod tf2 (type true) [{:keys [string-value]}]
[:code.clj__boolean {:class (styles/clj-boolean)}
string-value])

(defmethod tf2 (type 0) [{:keys [string-value]}]
[:code.clj_number {:class (styles/clj-number)}
string-value])

(defmethod tf2 (type nil) [{:keys [string-value]}]
[:code.clj__nil {:class (styles/clj-nil)} string-value])

(defmethod tf2 :default [{:keys [string-value] :as node}]
[:span.clj__token (str (keys node)) string-value])

(defmethod form TokenNode [node]
[tf2 node])

(defmethod form (type []) [node] node)

(defmethod form CommaNode [node] [:span.clj__comma ","])

(defmulti seq-form :tag)

(defmethod seq-form :default [{:keys [tag]}]
[:code.clj__unknown tag])

(defmethod seq-form :list [node]
(into [:code.seq {:class [(styles/clj-seq)
(selected-style node)]}]
(concat [ "("] (:children node) [")"])))

(defmethod seq-form :vector [node]
(into [:code.clj__seq] (concat ["["] (:children node) ["]"])))

(defmethod seq-form :map [node]
(into [:code.clj__map {:class [(selected-style node)]}]
(concat ["{"] (:children node) ["}"])))

(defmethod form SeqNode [node]
(seq-form node))

(defmethod form SymbolNode [{:keys [value string-value] :as node}]
[:code.clj__symbol {:class [(if (clj-core-macros value)
(styles/clj-core-macro)
(styles/clj-symbol))
(selected-style node)]}
string-value])

(defmethod form WhitespaceNode [node]
[:code.clj__whitespace {:style {:white-space "pre"}}
(:whitespace node)])

(defmethod form NewlineNode [_] [:br])

(defmethod form KeywordNode [{:keys [k] :as node}]
[:code.clj__keyword {:class [(styles/clj-keyword)
(selected-style node)]}
(str k)])

(defmethod form StringNode [{:keys [lines]}]
[:code.clj__string {:class (styles/clj-string)}
\" (apply str lines) \"])

(defn str->hiccup [s]
(let [positional-ast
(-> s
(rz/of-string {:track-position? true})
(rz/postwalk #(rz/edit* % assoc
:position (rz/position %)))
rz/node)]
(walk/postwalk #(cond-> % (record? %) form) positional-ast)))

0 comments on commit 029b996

Please sign in to comment.