Can't install Huginn on ARM device - libv8 tries to build with x64 arch #1586

wasafiri · 2016-07-16T17:40:12Z

Installing on Debian Jessie on a Pine64 device.

When running the command:
sudo -u huginn -H bundle install --deployment --without development test

I get error with the libv8 gem -- all other gems install fine. Seems it tries to compile for x64 and not arm64.

Installing libv8 3.16.14.13 with native extensions

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/ext/libv8
/usr/local/bin/ruby -r ./siteconf20160715-30828-5e4d0u.rb extconf.rb
creating Makefile
Compiling v8 for x64
Using python 2.7.9
Using compiler: /usr/bin/c++ (GCC version 4.9)
In file included from ../src/allocation.h:31:0,
                 from ../src/allocation.cc:28:
../src/globals.h:90:2: error: #error Host architecture was not detected as supported by v8
 #error Host architecture was not detected as supported by v8
  ^
../src/globals.h:116:2: error: #error Target architecture x64 is only supported on x64 host
 #error Target architecture x64 is only supported on x64 host
  ^
make[1]: *** [/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/vendor/v8/out/x64.release/obj.target/preparser_lib/src/allocation.o] Error 1
make: *** [x64.release] Error 2
/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/ext/libv8/location.rb:36:in `block in verify_installation!': libv8 did not install properly, expected binary v8 archive '/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/vendor/v8/out/x64.release/obj.target/tools/gyp/libv8_base.a'to exist, but it was not found (Libv8::Location::Vendor::ArchiveNotFound)
    from /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/ext/libv8/location.rb:35:in `each'
    from /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/ext/libv8/location.rb:35:in `verify_installation!'
    from /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/ext/libv8/location.rb:26:in `install!'
    from extconf.rb:7:in `<main>'
GYP_GENERATORS=make \
build/gyp/gyp --generator-output="out" build/all.gyp \
              -Ibuild/standalone.gypi --depth=. \
              -Dv8_target_arch=x64 \
              -S.x64  -Dv8_enable_backtrace=1 -Dv8_can_use_vfp2_instructions=true -Darm_fpu=vfpv2 -Dv8_can_use_vfp3_instructions=true -Darm_fpu=vfpv3 -Dwerror=''
make[1]: Entering directory '/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/vendor/v8/out'
  CXX(target) /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/vendor/v8/out/x64.release/obj.target/preparser_lib/src/allocation.o
tools/gyp/preparser_lib.target.x64.mk:111: recipe for target '/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/vendor/v8/out/x64.release/obj.target/preparser_lib/src/allocation.o' failed
make[1]: Leaving directory '/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/vendor/v8/out'
Makefile:195: recipe for target 'x64.release' failed

extconf failed, exit code 1

Gem files will remain installed in /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13 for inspection.
Results logged to /home/huginn/huginn/vendor/bundle/ruby/2.3.0/extensions/aarch64-linux/2.3.0-static/libv8-3.16.14.13/gem_make.out

The text was updated successfully, but these errors were encountered:

cantino · 2016-07-16T18:47:45Z

Sorry, I'm not sure how to install on ARM. This sounds like a v8 issue.

wasafiri · 2016-07-16T20:52:45Z

Is therubyracer gem necessary or can I get by just with nodejs on the system?

On Jul 16, 2016, at 2:48 PM, Andrew Cantino notifications@github.com wrote:

Sorry, I'm not sure how to install on ARM. This sounds like a v8 issue.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

cantino · 2016-07-16T21:29:53Z

It's needed for the JavaScriptAgent. If you don't need that, you can
probably use node for asset compilation.

dsander · 2016-07-18T13:11:59Z

@wasafiri Can you try to install libv8 via the package manager and build the gem with gem install libv8 -v '3.16.14.3' -- --with-system-v8? If the gem installed successfully bundler should use the already installed version and not try to build it again.

wasafiri · 2016-07-28T18:46:59Z

bahar@pine64:/home/huginn/huginn$ sudo apt-get install libv8-3.14-dbg
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  libv8-3.14-dbg:armhf
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 14.0 MB of archives.
After this operation, 59.2 MB of additional disk space will be used.
Get:1 http://ftp.us.debian.org/debian/ jessie/main libv8-3.14-dbg armhf 3.14.5.8-8.1 [14.0 MB]
Fetched 14.0 MB in 2s (5,085 kB/s)               
Selecting previously unselected package libv8-3.14-dbg.
(Reading database ... 35425 files and directories currently installed.)
Preparing to unpack .../libv8-3.14-dbg_3.14.5.8-8.1_armhf.deb ...
Unpacking libv8-3.14-dbg (3.14.5.8-8.1) ...
Setting up libv8-3.14-dbg (3.14.5.8-8.1) ...
bahar@pine64:/home/huginn/huginn$ gem install libv8 -v '3.16.14.3' -- --with-system-v8
Fetching: libv8-3.16.14.3.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/local/lib/ruby/gems/2.3.0 directory.
bahar@pine64:/home/huginn/huginn$ sudo -u huginn -H bundle install --deployment --without development test
Fetching gem metadata from https://rubygems.org/
Fetching version metadata from https://rubygems.org/
Fetching dependency metadata from https://rubygems.org/

...

An error occurred while installing libv8 (3.16.14.13), and Bundler cannot continue.
Make sure that `gem install libv8 -v '3.16.14.13'` succeeds before bundling.
bahar@pine64:/home/huginn/huginn$ sudo -u huginn -H gem install libv8 -v '3.16.14.3' -- --with-system-v8
Fetching: libv8-3.16.14.3.gem (100%)
ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions for the /usr/local/lib/ruby/gems/2.3.0 directory

wasafiri · 2016-07-28T23:33:46Z

For what it's worth, I don't get that permissions error when running

sudo -u huginn -H bundle install --deployment --without development test

dsander · 2016-07-29T13:44:42Z

Ah, sorry the huginn user does not have the permissions to install a gem globally. Does this work?

sudo -u huginn -H bundle config build.libv8 --with-system-v8 
sudo -u huginn -H bundle config build.therubyracer --with-system-v8
sudo -u huginn -H bundle install --deployment --without development test

wasafiri · 2016-08-05T03:49:44Z

Different error-- progress!

Installing therubyracer 0.12.2 with native extensions

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/therubyracer-0.12.2/ext/v8
/usr/local/bin/ruby -r ./siteconf20160730-4274-1htkhgs.rb extconf.rb --with-system-v8
checking for main() in -lpthread... yes
checking for v8.h... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/usr/local/bin/$(RUBY_BASE_NAME)
    --with-pthreadlib
    --without-pthreadlib
    --enable-debug
    --disable-debug
    --with-v8-dir
    --without-v8-dir
    --with-v8-include
    --without-v8-include=${v8-dir}/include
    --with-v8-lib
    --without-v8-lib=${v8-dir}/lib
/home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/ext/libv8/location.rb:50:in `configure': You have chosen to use the version of V8 found on your system (Libv8::Location::System::NotFoundError)
and *not* the one that is bundle with the libv8 rubygem. However,
it could not be located. please make sure you have a version of
v8 that is compatible with 3.16.14.13 installed. You may
need to special --with-v8-dir options if it is in a non-standard
location

thanks,
The Mgmt

    from /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/libv8-3.16.14.13/lib/libv8.rb:7:in `configure_makefile'
    from extconf.rb:32:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /home/huginn/huginn/vendor/bundle/ruby/2.3.0/extensions/aarch64-linux/2.3.0-static/therubyracer-0.12.2/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /home/huginn/huginn/vendor/bundle/ruby/2.3.0/gems/therubyracer-0.12.2 for inspection.
Results logged to /home/huginn/huginn/vendor/bundle/ruby/2.3.0/extensions/aarch64-linux/2.3.0-static/therubyracer-0.12.2/gem_make.out

not sure why it can't find the system v8 for therubyracer when it can find it for libv8...

the contents of mkmf.log are:

have_library: checking for main() in -lpthread... -------------------- yes

"gcc -o conftest -I/usr/local/include/ruby-2.3.0/aarch64-linux -I/usr/local/include/rub$
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

"gcc -o conftest -I/usr/local/include/ruby-2.3.0/aarch64-linux -I/usr/local/include/rub$
checked program was:
/* begin */
 1: #include "ruby.h"
 2:
 3: /*top*/
 4: extern int t(void);
 5: int main(int argc, char **argv)
 6: {
 7:   if (argc > 1000000) {
 8:     printf("%p", &t);
 9:   }
10:
11:   return 0;
12: }
13: int t(void) { void ((*volatile p)()); p = (void ((*)()))main; return !p; }
/* end */
--------------------

dsander · 2016-08-06T10:17:12Z

I think this is an issue with libv8 on ARM 64bit, bundle install worked fine without additional switches on a Raspberry 3.

-Dv8_target_arch=x64 \ my guess is that libv8 either does not support the architecture or fails to detect it.

What does ruby -e "puts Gem::Platform.local.cpu" output?

wasafiri · 2016-08-07T18:04:51Z

The output is

aarch64

dsander · 2016-08-07T19:51:24Z

Odd that arch should be supported, you might need to open an issue on therubyracer and/or libv8, I could not find any hints on how to build it on 64bit arm.

dsander · 2016-08-09T20:12:38Z

@wasafiri I had a similar issue on OSX today and it turned out the therubyracer flag I wrote previously was wrong. On OSX this worked:

sudo gem install therubyracer -v '0.12.2' -- --with-v8-dir=/usr/local/opt/v8-315

On debian the path will be different, but maybe it helps.

ghost · 2016-08-23T15:24:53Z

Has any progress been made on this?

souljedi · 2016-09-04T19:59:09Z

I'm stuck at the same issue with an Odroid C2 with Armbian Ubuntu xenial server. @wasafiri or @zone22 any progress on your ends?

cantino · 2016-09-06T17:29:13Z

I had a libv8 issue on OS X and this PR fixes it: #1671

knu · 2016-09-07T00:25:23Z

Looking at https://rubygems.org/gems/libv8/versions, 3.16.14.7 has a precompiled binary gem for the arm-linux architecture, so try locking at the version if manual build fails.

souljedi · 2016-09-27T07:17:06Z

@knu: any suggestions how i could lock to a libv8 version? I already tried several ways without success. Even though i can install several libv8 gems for my platform, therubyracer always wants to do a manual build and then fails with a wrong platform.

knu · 2016-09-27T11:39:47Z

@souljedi You could go ahead and manually edit Gemfile.lock for now. Locate the line that reads libv8 (3.16.14.15) and change it to libv8 (3.16.14.7), then run bundle.

ghost · 2016-11-06T07:23:22Z

Any news? as #1671 has been merged.

sunbinyuan · 2016-11-27T09:51:06Z

# TODO fix false positive on 64-bit ARM

def x64?
  if rubinius?
    x86_64_from_build_cpu || x86_64_from_arch_flag
  else
    x86_64_from_byte_length
  end
end

This was in the code so I guess they haven't implemented support for aarch64. Gonna adding it manually and see if it compiles.

wasafiri · 2016-12-31T14:41:16Z

Hey sunbinyuan, did it work?

sunbinyuan · 2016-12-31T16:03:56Z

Nope, I notice that the build was too old and a the build procedures for arm64 wasn't implemented yet. You can try the newer build of the gem if you manage to install v8 manually.

dsander · 2017-04-10T14:44:56Z

Hi @wasafiri, are you still interested in running Huginn on your Pine64? Could you try to run gem install mini_racer without any additional flag?

wasafiri · 2017-04-10T14:55:44Z

I will give that a shot this week!

…

Sent from my iPhone

On Apr 10, 2017, at 10:45 AM, Dominik Sander ***@***.***> wrote: Hi @wasafiri, are you still interested in running Huginn on your Pine64? Could you try to run gem install mini_racer without any additional flag? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

wasafiri · 2017-05-04T02:37:24Z

result:

sudo gem install mini_racer
Fetching: libv8-5.3.332.38.5.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing mini_racer:
	ERROR: Failed to build gem native extension.

    current directory: /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/ext/libv8
/usr/local/bin/ruby -r ./siteconf20170504-2493-bzvr5d.rb extconf.rb
creating Makefile
Error: Command '/usr/bin/python v8/build/linux/sysroot_scripts/install-sysroot.py --running-as-hook' returned non-zero exit status 1 in /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "managed": False,
    "name": "v8",
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "custom_deps": {},
    "deps_file": "DEPS",
    "safesync_url": "",
  },
]
'
Running: gclient sync --with_branch_heads
Traceback (most recent call last):
  File "/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor/depot_tools/fetch.py", line 353, in <module>
    sys.exit(main())
  File "/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor/depot_tools/fetch.py", line 348, in main
    return run(options, spec, root)
  File "/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor/depot_tools/fetch.py", line 342, in run
    return checkout.init()
  File "/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor/depot_tools/fetch.py", line 142, in init
    self.run_gclient(*sync_cmd)
  File "/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor/depot_tools/fetch.py", line 76, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/vendor/depot_tools/fetch.py", line 66, in run
    return subprocess.check_output(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('gclient', 'sync', '--with_branch_heads')' returned non-zero exit status 2
/usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/ext/libv8/builder.rb:109:in `block in setup_build_deps!': unable to fetch v8 source (RuntimeError)
	from /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/ext/libv8/builder.rb:107:in `chdir'
	from /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/ext/libv8/builder.rb:107:in `setup_build_deps!'
	from /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/ext/libv8/builder.rb:63:in `build_libv8!'
	from /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5/ext/libv8/location.rb:24:in `install!'
	from extconf.rb:7:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /usr/local/lib/ruby/gems/2.3.0/gems/libv8-5.3.332.38.5 for inspection.
Results logged to /usr/local/lib/ruby/gems/2.3.0/extensions/aarch64-linux/2.3.0-static/libv8-5.3.332.38.5/gem_make.out

dsander · 2017-05-24T20:31:54Z

Thanks, that a bummer I hoped mini_racer compiles on arm64.

LexiconCode · 2018-05-23T17:35:34Z

Is there update on this?

dsander · 2018-05-24T08:15:03Z

@LexiconCode None that I know of. We can switch to mini_racer soon, but I don't know if it will work on arm64

LexiconCode · 2018-05-29T17:04:52Z

Let me know you switch to mini_racer and I will test.

The latest version of mini_racer works on arm64 now. I will do some more tests but supporting Huginn on Raspberries (and other arm(64) boards/servers) would be really nice.

https://www.bountysource.com/issues/43955706-replace-therubyracer-with-mini_racer

LorenzoAncora · 2018-09-20T13:01:59Z

@dsander @cantino @knu
Is there any possibility of installing Huginn on a Raspberry Pi 3 with 64-bit architecture enabled?
I have a Raspberry Pi 3 B with Debian aarch64 (ARM64) installed and connected to the network.
I'm willing to be a tester (in case tag me) if it can help you get compatibility with aarch64.

LexiconCode · 2018-09-20T13:37:26Z

This is a relevant pull request to replace rubyracer with mini_racer
#1961

dsander · 2018-09-21T08:55:59Z

Some people has success on raspberries in the past, but I don't know about the 3. Recent libv8 versions seem not to build on ARM/ARM64 so even mini_racer will not help us.

sfischer13 · 2018-09-22T11:52:48Z

I made it work on Raspberry Pi 3 by applying this PR. I also had to fix the versions of some dependencies because they have not been updated for the ARM architecture.

By hindsight, it was not worth the effort. There were regular crashes, probably due to heavy usage of swap memory, which then wore out the SD cards. This is only a guess, but I never had issues again after migrating to a server with more memory (not ARM though).

I don't want to discourage anyone – the Raspberry Pi is a great platform – but you may waste a lot of time tinkering and then realize that Huginn needs more power as your number of agents increases.

LorenzoAncora · 2018-09-22T12:45:59Z

^{I made it work on Raspberry Pi 3 by applying this PR. I also had to fix the versions of some dependencies because they have not been updated for the ARM architecture. By hindsight, it was not worth the effort. There were regular crashes, probably due to heavy usage of swap memory, which then wore out the SD cards. This is only a guess, but I never had issues again after migrating to a server with more memory (not ARM though).}^{I don't want to discourage anyone – the Raspberry Pi is a great platform – but you may waste a lot of time tinkering and then realize that Huginn needs more power as your number of agents increases.}

@sfischer13 how many agents did you keep active? If possible, could you also tell us which ones (not the finality, just the name)?

sfischer13 · 2018-09-22T14:50:34Z

I think it was more than 50 agents, but most of them were lightweight (hourly WebsiteAgent, DigestAgent, EmailAgent, ...). Problems started when I tried CsvAgent and JavaScriptAgent. In my setup, these agents created too many events (happens fast if CsvAgent creates one event per line) and then the crashes began.

LorenzoAncora · 2018-09-22T17:03:43Z

_{I think it was more than 50 agents, but most of them were lightweight (hourly `WebsiteAgent`, `DigestAgent`, `EmailAgent`, ...). Problems started when I tried `CsvAgent` and `JavaScriptAgent`. In my setup, these agents created too many events (happens fast if `CsvAgent` creates one event per line) and then the crashes began.}

@sfischer13
Thank you these are really useful informations.
The use of the swap can not be the direct cause of the crashes, I have a better explanation: the subprocesses initiated by the 50 Huginn agents saturated the memory and were killed by the kernel OOM killer. When the GNU/Linux kernel kills a process this way and the CPU is already working at full capacity the victim processes do not have time to finish properly and thus can cause the entire process to crash. This also implies that the processes terminated abruptly have no way to free the memory and so in the long run the RAM is saturated and the system starts to swap everything, causing a chain reaction of OOM killing.

In the next 2 days I will try to install Huginn with the changes you suggested on a Raspberry Pi model 3 B (that is already executing a web server and other heavy daemons). A few hours of experimentation will clarify whether it is advantageous to run Huginn on a board in the long run. ;-)

LexiconCode · 2018-09-22T17:12:46Z

Understand their other much more powerful ARM based single board computers. For example Odroid series. So don't let the raspberry pi be the final determining factor :).

sfischer13 · 2018-09-23T12:14:42Z

@LorenzoAncora, thanks for the information about Linux process management. Your explanation makes more sense than mine and I guess my corrupted SD was just a coincidence.
Good luck with your project!

ch40s · 2019-12-27T17:25:10Z

Any update on this? There are a few powerful ARM based SBCs these days that can easily run Huginn. In my opinion this issue limits and slows down the adoption of huginn.

ch40s · 2019-12-30T15:10:45Z

It would be nice to have a docker image compatible with arm, if you agree please provide some feedback: #2648
cc: @sfischer13 @LorenzoAncora @LexiconCode @wasafiri

LorenzoAncora · 2020-01-01T11:05:24Z

It would be nice to have a docker image compatible with arm, if you agree please provide some feedback: #2648
cc: @sfischer13 @LorenzoAncora @LexiconCode @wasafiri

@ch40s I agree, currently¹ there are only AMD64 docker images.
An official ARM64\ARMhf docker image of Huginn would be very appropriate. 👍

_{1: the contents of that page may change at any time. You'll find ARM images there if and when authors add them.}

unixfox · 2022-01-11T15:27:28Z

I just created a Docker image of Huginn single process compatible with ARMv7/armhf, ARMv8/aarch64 and intel x86. This image is automatically updated for every new commit on https://github.com/huginn/huginn.
You can find it here: https://quay.io/repository/unixfox/huginn-single-process
The source code of my Dockerfile is here: https://github.com/unixfox/periodic-build-with-github-actions/blob/master/huginn/single-process/Dockerfile

I have no plans to support the multiple processes docker image of Huginn because I don't use it myself, and it is more difficult to build.

mhalano · 2022-05-01T21:26:58Z

Do we have any news about the official support for that?

ghost mentioned this issue Nov 21, 2016

DietPi-Software | Add Huginn: Performs automated tasks online MichaIng/DietPi#250

Closed

dsander mentioned this issue Apr 10, 2017

Replace therubyracer with mini_racer #1961

Merged

dsander mentioned this issue Sep 16, 2018

Installation fails on Raspberry Pi 3: libv8 not supported (therubyracer) #2372

Closed

Can't install Huginn on ARM device - libv8 tries to build with x64 arch #1586

Can't install Huginn on ARM device - libv8 tries to build with x64 arch #1586

Comments

wasafiri commented Jul 16, 2016 • edited

cantino commented Jul 16, 2016

wasafiri commented Jul 16, 2016

cantino commented Jul 16, 2016 • edited

dsander commented Jul 18, 2016

wasafiri commented Jul 28, 2016 • edited

wasafiri commented Jul 28, 2016

dsander commented Jul 29, 2016

wasafiri commented Aug 5, 2016

dsander commented Aug 6, 2016

wasafiri commented Aug 7, 2016

dsander commented Aug 7, 2016

dsander commented Aug 9, 2016

ghost commented Aug 23, 2016

souljedi commented Sep 4, 2016

cantino commented Sep 6, 2016

knu commented Sep 7, 2016

souljedi commented Sep 27, 2016

knu commented Sep 27, 2016

ghost commented Nov 6, 2016 • edited by ghost

sunbinyuan commented Nov 27, 2016 • edited

wasafiri commented Dec 31, 2016

sunbinyuan commented Dec 31, 2016

dsander commented Apr 10, 2017

wasafiri commented Apr 10, 2017 via email

wasafiri commented May 4, 2017

dsander commented May 24, 2017

LexiconCode commented May 23, 2018

dsander commented May 24, 2018

LexiconCode commented May 29, 2018 • edited

LorenzoAncora commented Sep 20, 2018

LexiconCode commented Sep 20, 2018 • edited

dsander commented Sep 21, 2018

sfischer13 commented Sep 22, 2018

LorenzoAncora commented Sep 22, 2018

sfischer13 commented Sep 22, 2018

LorenzoAncora commented Sep 22, 2018

LexiconCode commented Sep 22, 2018 • edited

sfischer13 commented Sep 23, 2018

ch40s commented Dec 27, 2019

ch40s commented Dec 30, 2019

LorenzoAncora commented Jan 1, 2020

unixfox commented Jan 11, 2022 • edited

mhalano commented May 1, 2022

wasafiri commented Jul 16, 2016 •

edited

cantino commented Jul 16, 2016 •

edited

wasafiri commented Jul 28, 2016 •

edited

ghost commented Nov 6, 2016 •

edited by ghost

sunbinyuan commented Nov 27, 2016 •

edited

LexiconCode commented May 29, 2018 •

edited

LexiconCode commented Sep 20, 2018 •

edited

LexiconCode commented Sep 22, 2018 •

edited

unixfox commented Jan 11, 2022 •

edited