Captain Definition file does not exist (but it does!) #109

drmrbrewer · 2018-01-09T16:29:23Z

I successfully deployed the node example from here.

But when it comes to deploying my own node app, I have a problem. I'm deploying the app from within the git repo folder which I know is working (I deploy it to heroku all the time). But when I try with captain, I get:

$ captainduckduck deploy

Deploying to captain-01

Saving tar file to:
/home/ubuntu/workspace/temporary-captain-to-deploy.tar

Pushing last commit on master: 05ab1ab6da7356c570e947b646eb40b235e57f94
Uploading file to https://captain.server-01.example.com
 
This might take several minutes. PLEASE BE PATIENT...
Building started: my-app-test
 
------------------------- Tue Jan 09 2018 16:19:07 GMT+0000 (UTC)
Build started for my-app-test
----------------------
Deploy failed!
Error: Captain Definition file does not exist!
 
Something bad happened. Cannot deploy "my-app-test"

BUT I have put a captain-definition file in the folder, alongside package.json at the top level, in the folder from which I'm deploying. It has the content:

{
    "schemaVersion" :1 ,
    "templateId" :"node/8.9.3"
}

The node version above matches what I have in package.json. The deploy process seems to be going OK until the above point... I can't understand the error because I do have a definition file... what else can I try next?

The text was updated successfully, but these errors were encountered:

gnuns · 2018-01-09T17:04:36Z

@drmrbrewer, did you add and commit the captain-definition file before deploy?

githubsaturn · 2018-01-09T17:31:01Z

Thanks @gnuns again! Either you didn't commit your captain-definition file or it's in a wrong place. You can double check by running

git archive --format tar --output ./test.tar master

and then open test.tar to make sure you can see captain-definition file. The project structure that works is:

inside tar file:
- .git
-  package.json
-  captain-definition
-  somedirectory
----- some file #1
----- some file #2
-  someotherdirectory
----- some other file #1

drmrbrewer · 2018-01-09T17:55:28Z

OK thanks, that must be it. I haven't yet committed the definition file. Perhaps it's worth making that clear in the readme? Maybe I was just being dumb, but it seems like an easy mistake to make, if you're just trying out captain briefly on your own project, by dumping a definition file and deploying.

drmrbrewer · 2018-01-09T17:59:42Z

And I note that the CLI gives the following info when deploying:

Note that uncommited files and files in gitignore (if any) will not be pushed to server

I suppose I passed over that because I didn't realise that the definition file itself needs to be "pushed to the server"... my assumption was that this is purely a local config file and it doesn't matter that it hasn't been committed.

drmrbrewer · 2018-01-09T18:07:04Z

Now it proceeds further, but it chokes with the following errors (which doesn't happen when deploying to heroku)... any ideas?

npm
 WARN deprecated gulp-util@3.0.8: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5

> node-expat@2.3.16 install /usr/src/app/node_modules/node-expat
> node-gyp rebuild
gyp
 
ERR!
 
configure error
 

gyp
 ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at PythonFinder.failNoPython (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:483:19)
gyp ERR! stack
	 at PythonFinder.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:397:16)

gyp
 ERR!
 stack     at F (/usr/local/lib/node_modules/npm/node_modules/which/which.js:68:16)

gyp
 
ERR!
 
stack     at E (/usr/local/lib/node_modules/npm/node_modules/which/which.js:80:29)
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/which.js:89:16
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /usr/local/lib/node_modules/npm/node_modules/which/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqWrap.oncomplete (fs.js:152:21)

gyp
 
ERR!
 
System
 Linux 4.4.0-98-generic

gyp
 
ERR!
 
command
 "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"

gyp ERR! cwd /usr/src/app/node_modules/node-expat
gyp ERR! node -v v8.9.3
gyp ERR! node-gyp -v v3.6.2

gyp
 
ERR!
 
not ok
 

npm
 ERR! code ELIFECYCLE

npm ERR! errno 1

npm
 
ERR!
 node-expat@2.3.16 install: `node-gyp rebuild`
npm ERR! Exit status 1

npm ERR! 
npm ERR! Failed at the node-expat@2.3.16 install script.
npm 
ERR!
 This is probably not a problem with npm. There is likely additional logging output above.


npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-01-09T18_02_49_470Z-debug.log

githubsaturn · 2018-01-09T18:14:16Z

It's NodeJS Docker alpine image issue, see: nodejs/docker-node#384 (comment)

Change your captain-definition file to this:

 {
  "schemaVersion" :1 ,
  "dockerfileLines" :[
                        "FROM node:8.7.0",
                        "RUN mkdir -p /usr/src/app",
                        "WORKDIR /usr/src/app",
                        "COPY ./src/package.json /usr/src/app/",
                        "RUN npm install && npm cache clean --force",
                        "COPY ./src /usr/src/app",
                        "ENV NODE_ENV production",
                        "ENV PORT 80",
                        "EXPOSE 80",
                        "CMD [ \"npm\", \"start\" ]"
                    ]
 }

drmrbrewer · 2018-01-09T18:26:01Z

Thanks, @githubsaturn, I'll try that. Hope you don't mind another quick question... it's not captain-specific I suppose, it's more that I'm a noob to docker... my node app needs phantomjs installed, which in heroku I achieve via a buildpack. With captain/docker, my thinking is that I can use something like this... how do I add this to the captain deploy procedure?

drmrbrewer · 2018-01-09T18:45:02Z

And I'm now getting a rather uninformative build failure error:

Build started for my-test-app
Step 1/10 : FROM node:8.9.3
---> 2eeae8debf3d
Step 2/10 : RUN mkdir -p /usr/src/app
---> Using cache
---> 7eea214316b0
Step 3/10 : WORKDIR /usr/src/app
---> Using cache
---> e472ed5f7bce
Step 4/10 : COPY ./src/package.json /usr/src/app/
---> Using cache
---> ccda5d1a6259
Step 5/10 : RUN npm install && npm cache clean --force
---> Running in e42189f8cf43
npm
 
WARN deprecated gulp-util@3.0.8: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5

> node-expat@2.3.16 install /usr/src/app/node_modules/node-expat
> node-gyp rebuild


{"code":1,"message":"The command '/bin/sh -c npm install && npm cache clean --force' returned a non-zero code: 1"}
The command '/bin/sh -c npm install && npm cache clean --force' returned a non-zero code: 1
----------------------
Deploy failed!
Error: {"code":1,"message":"The command '/bin/sh -c npm install && npm cache clean --force' returned a non-zero code: 1"}
The command '/bin/sh -c npm install && npm cache clean --force' returned a non-zero code: 1

Something bad happened. Cannot deploy "my-test-app"

githubsaturn · 2018-01-09T19:03:14Z

Please include your package.json

drmrbrewer · 2018-01-09T19:15:03Z

{
  "name": "myapp",
  "version": "2.0.6",
  "description": "My app",
  "main": "nodeserver.js",
  "engines": {
	"node": "8.9.3"
  },
  "dependencies": {
	"archiver": "^2.1.0",
	"async": "^2.6.0",
	"body-parser": "^1.18.2",
	"compression": "^1.7.1",
	"connect-timeout": "^1.9.0",
	"cors": "^2.8.4",
	"express": "^4.16.2",
	"filesize": "^3.5.11",
	"finalhandler": "^1.1.0",
	"fs-extra": "^5.0.0",
	"glossy": "^0.1.7",
	"gm": "^1.23.1",
	"heroku-client": "^3.0.6",
	"into-stream": "^3.1.0",
	"jsdap": "^8.1.0",
	"json-stable-stringify": "^1.0.1",
	"json2xls": "^0.1.2",
	"lodash": "^4.17.4",
	"mathjs": "^3.19.0",
	"moment-timezone": "^0.5.14",
	"ndarray": "^1.0.18",
	"ndarray-linear-interpolate": "^1.0.0",
	"nodemailer": "^4.4.1",
	"pg": "^7.4.1",
	"ping-monitor": "^0.1.6",
	"pngquant": "^1.3.0",
	"pngquant-bin": "^3.1.1",
	"puppeteer": "^0.13.0",
	"request": "^2.83.0",
	"whois": "^2.5.0",
	"xml2json": "^0.11.0"
  },
  "devDependencies": {
	"onesky-utils": "^1.2.0"
  },
  "scripts": {
	"start": "node nodeserver.js",
	"test": "node nodeserver.js"
  },
  "repository": {
	"type": "git",
	"url": "git@bitbucket.org:myusername/myrepo.git"
  },
  "keywords": [
	"cool",
	"app"
  ],
  "author": "me.com",
  "private": true,
  "license": "none"
}

githubsaturn · 2018-01-10T05:11:53Z

I've put together a small test project using your package and it works perfectly file:

test.zip

Note that it might be due to some RAM usage, see this: https://github.com/githubsaturn/captainduckduck/wiki/Getting-Started#b-server-with-a-public-ip

My instance has 2GB of RAM.

As for buildpacks, they are nothing more than dockerfile lines. For example, for your PhantomJS, you have two options:

Start PhantomJS as a separate app (container) in your docker instance and connect to it via port xxx. In this case your docker file lines will be just one line: "FROM wernight/phantomjs"
Install PhantomJS in the same container as your app. This is what buildpack does. To do this, you will need a more involved docker file lines:

https://github.com/wernight/docker-phantomjs/blob/master/Dockerfile

Make sure to replace the line (FROM debian:jessie) to FROM node:8.9.3. Also remove last two lines, ENTRYPOINT and CMD

PS: convert dockerfile to captain-definition https://jsbin.com/zehunuwoqe

githubsaturn · 2018-01-10T06:18:02Z

This is the captain-definition file that you should use:

{
  "schemaVersion": 1,
  "dockerfileLines": [
    "FROM node:8.9.3",
    "# Install runtime dependencies",
    "RUN apt-get update \\",
    " && apt-get install -y --no-install-recommends \\",
    "        ca-certificates \\",
    "        bzip2 \\",
    "        libfontconfig \\",
    " && apt-get clean \\",
    " && rm -rf /var/lib/apt/lists/*",
    "# Install official PhantomJS release",
    "# Install dumb-init (to handle PID 1 correctly).",
    "# https://github.com/Yelp/dumb-init",
    "# Runs as non-root user.",
    "# Cleans up.",
    "RUN set -x  \\",
    " && apt-get update \\",
    " && apt-get install -y --no-install-recommends \\",
    "        curl \\",
    " && mkdir /tmp/phantomjs \\",
    " && curl -L https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2 \\",
    "        | tar -xj --strip-components=1 -C /tmp/phantomjs \\",
    " && mv /tmp/phantomjs/bin/phantomjs /usr/local/bin \\",
    " && curl -Lo /tmp/dumb-init.deb https://github.com/Yelp/dumb-init/releases/download/v1.1.3/dumb-init_1.1.3_amd64.deb \\",
    " && dpkg -i /tmp/dumb-init.deb \\",
    " && apt-get purge --auto-remove -y \\",
    "        curl \\",
    " && apt-get clean \\",
    " && rm -rf /tmp/* /var/lib/apt/lists/* \\",
    " && useradd --system --uid 52379 -m --shell /usr/sbin/nologin phantomjs \\",
    " && su phantomjs -s /bin/sh -c \"phantomjs --version\"",
    "USER phantomjs",
    "EXPOSE 8910"
  ]
}

drmrbrewer · 2018-01-10T08:39:18Z

Thanks for the help with phantomjs... I'll try that out.

Concerning the build problem I was having... you're right... I resized the server and ran the build again, and it worked... well the build was successful but when I try to access the node app via the URL I get the following in the browser:

502 Bad Gateway
nginx/1.13.8

I have not messed with nginx at all, nor have I done any clustering of nodes... just one node at present. And (to rule out that resizing the server messed something up) I also tried starting from scratch... completely new server created (on digitalocean) and set up with captain, with a fresh deploy of the node app. Still I get the "bad gateway" error.

githubsaturn · 2018-01-10T16:18:06Z

Ahhh my bad! Sorry! I forgot to add the last lines from #109 (comment) to your dockerfile, see below for the new dockerfile:

{
  "schemaVersion": 1,
  "dockerfileLines": [
		"FROM node:8.9.3",
		"# Install runtime dependencies",
		"RUN apt-get update \\",
		" && apt-get install -y --no-install-recommends \\",
		"        ca-certificates \\",
		"        bzip2 \\",
		"        libfontconfig \\",
		" && apt-get clean \\",
		" && rm -rf /var/lib/apt/lists/*",
		"# Install official PhantomJS release",
		"# Install dumb-init (to handle PID 1 correctly).",
		"# https://github.com/Yelp/dumb-init",
		"# Runs as non-root user.",
		"# Cleans up.",
		"RUN set -x  \\",
		" && apt-get update \\",
		" && apt-get install -y --no-install-recommends \\",
		"        curl \\",
		" && mkdir /tmp/phantomjs \\",
		" && curl -L https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2 \\",
		"        | tar -xj --strip-components=1 -C /tmp/phantomjs \\",
		" && mv /tmp/phantomjs/bin/phantomjs /usr/local/bin \\",
		" && curl -Lo /tmp/dumb-init.deb https://github.com/Yelp/dumb-init/releases/download/v1.1.3/dumb-init_1.1.3_amd64.deb \\",
		" && dpkg -i /tmp/dumb-init.deb \\",
		" && apt-get purge --auto-remove -y \\",
		"        curl \\",
		" && apt-get clean \\",
		" && rm -rf /tmp/* /var/lib/apt/lists/* \\",
		" && useradd --system --uid 52379 -m --shell /usr/sbin/nologin phantomjs \\",
		" && su phantomjs -s /bin/sh -c \"phantomjs --version\"",
		"USER phantomjs",
		"EXPOSE 8910",
		"RUN mkdir -p /usr/src/app",
		"WORKDIR /usr/src/app",
		"COPY ./src/package.json /usr/src/app/",
		"RUN npm install && npm cache clean --force",
		"COPY ./src /usr/src/app",
		"ENV NODE_ENV production",
		"ENV PORT 80",
		"EXPOSE 80",
		"CMD [ \"npm\", \"start\" ]"
  ]
}

drmrbrewer · 2018-01-10T16:45:26Z

@githubsaturn strangely, now I get the following error during deploy:

Step 6/15 : RUN mkdir -p /usr/src/app
---> Running in 0dca35295be6
mkdir: cannot create directory '/usr/src/app': Permission denied

This didn't happen when I wasn't including the phantomjs lines beforehand...

githubsaturn · 2018-01-10T16:48:40Z

It's trial and error game. Remove these lines:

		" && useradd --system --uid 52379 -m --shell /usr/sbin/nologin phantomjs \\",
		" && su phantomjs -s /bin/sh -c \"phantomjs --version\"",
		"USER phantomjs",

drmrbrewer · 2018-01-10T23:13:59Z

Is there any way to cache the phantomjs tar.bz2 file (fetched in the docker script) between deploys? It is incredibly slow fetching that (to a digitalocean droplet), and it is fetched every deploy. Is this a use-case for the persistent data area?

drmrbrewer · 2018-01-10T23:17:25Z

And the only other thing that seems to be missing is font support. Currently, I have a bunch of font files in the .fonts folder of my project, and when this is deployed to heroku, the fonts are picked up and used within phantomjs without any apparent extra configuration required. I will try to figure it out, but you may have some idea!

githubsaturn · 2018-01-11T04:28:26Z

Docker caches the layers if you have use the dockerfile as I shared.

For fonts, you can look up to find where PhantomJS uses fonts, let's say it turns out to be /usr/fonts. Then add this line to your dockerfile: COPY ./src/.font /usr/fonts. So all you need to find out is where PhantomJS loads the fonts.

drmrbrewer · 2018-01-11T09:37:08Z

Regarding caching, I understand a bit more now having read this article. It does indeed use the cached binary file so long as I am not messing around with the layer concerned.

As for fonts, copying the fonts into /usr/share/fonts was sufficient, which is where fontconfig can pick them up.

Thanks for your help, and again thanks for this awesome tool... it is really well put together.

RikusWiehahn · 2021-02-09T02:38:05Z

I got this error because I named my captain-definition file in my root directory: captain-definition.json.
But the .json was not necessary!
Kinda obvious but also kinda not 🤷‍♂️ (for noobs like me)

MrHenryStark · 2021-07-30T17:15:06Z

I need to have the captain-definition file in my repo? i cant just add it locally?

githubsaturn · 2021-07-30T23:30:12Z

Yes - it's needed on your server to build your application.

githubsaturn added the question label Jan 9, 2018

githubsaturn closed this as completed Jan 10, 2018

TylerWanta mentioned this issue May 24, 2024

[Deployment Issue] 'Captain-definition file does not exist' after updating repo structure #2068

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Captain Definition file does not exist (but it does!) #109

Captain Definition file does not exist (but it does!) #109

drmrbrewer commented Jan 9, 2018

gnuns commented Jan 9, 2018 •

edited

githubsaturn commented Jan 9, 2018 •

edited

drmrbrewer commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018 •

edited

drmrbrewer commented Jan 9, 2018

githubsaturn commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018

githubsaturn commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018

githubsaturn commented Jan 10, 2018 •

edited

githubsaturn commented Jan 10, 2018 •

edited

drmrbrewer commented Jan 10, 2018 •

edited

githubsaturn commented Jan 10, 2018 •

edited

drmrbrewer commented Jan 10, 2018

githubsaturn commented Jan 10, 2018

drmrbrewer commented Jan 10, 2018

drmrbrewer commented Jan 10, 2018

githubsaturn commented Jan 11, 2018

drmrbrewer commented Jan 11, 2018

RikusWiehahn commented Feb 9, 2021 •

edited

MrHenryStark commented Jul 30, 2021

githubsaturn commented Jul 30, 2021

Captain Definition file does not exist (but it does!) #109

Captain Definition file does not exist (but it does!) #109

Comments

drmrbrewer commented Jan 9, 2018

gnuns commented Jan 9, 2018 • edited

githubsaturn commented Jan 9, 2018 • edited

drmrbrewer commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018 • edited

drmrbrewer commented Jan 9, 2018

githubsaturn commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018

githubsaturn commented Jan 9, 2018

drmrbrewer commented Jan 9, 2018

githubsaturn commented Jan 10, 2018 • edited

githubsaturn commented Jan 10, 2018 • edited

drmrbrewer commented Jan 10, 2018 • edited

githubsaturn commented Jan 10, 2018 • edited

drmrbrewer commented Jan 10, 2018

githubsaturn commented Jan 10, 2018

drmrbrewer commented Jan 10, 2018

drmrbrewer commented Jan 10, 2018

githubsaturn commented Jan 11, 2018

drmrbrewer commented Jan 11, 2018

RikusWiehahn commented Feb 9, 2021 • edited

MrHenryStark commented Jul 30, 2021

githubsaturn commented Jul 30, 2021

gnuns commented Jan 9, 2018 •

edited

githubsaturn commented Jan 9, 2018 •

edited

drmrbrewer commented Jan 9, 2018 •

edited

githubsaturn commented Jan 10, 2018 •

edited

githubsaturn commented Jan 10, 2018 •

edited

drmrbrewer commented Jan 10, 2018 •

edited

githubsaturn commented Jan 10, 2018 •

edited

RikusWiehahn commented Feb 9, 2021 •

edited